Repository navigation
Description
What happened (please include outputs or screenshots):
Upgrading libraries in one of my images. Last image was built 4 days ago, works fine. We were running kubernetes 33.1.0.
Today's image is pulling 34.1.0. KO
Picking apart each package version that changed as of my previous image, I ended up suspecting kubernetes client itself.
v33.1.0...v34.1.0
Obviously, this:
e3b373f#diff-977bedbaf9339b25749bc96a24266c1fa3e99ab521a835562fe84b2ab87b7001
And then again, the rest client, which is the one throwing in my case, looks OK?
https://github.com/kubernetes-client/python/blob/master/kubernetes/client/rest.py
That "should_bypass_proxy" ... looks correct ...
https://requests.readthedocs.io/en/latest/_modules/requests/utils/
We are using proxies. Ansible would connect public services (eg: aws). Our proxies would only allow for specific names out.
Ansible would connect to our clusters directly/no proxy.
Our environment would include:
HTTP_PROXY: http://my-http-proxy:8080
HTTPS_PROXY: http://my-http-proxy:8080
NO_PROXY: github.corp.com,.github.corp.com,localhost,.local,.svc,10.0.0.0/8,127.0.0.0/8,127.0.0.1,::1,.corp.com
http_proxy: http://my-http-proxy:8080
https_proxy: http://my-http-proxy:8080
no_proxy: github.corp.com,.github.corp.com,localhost,.local,.svc,10.0.0.0/8,127.0.0.0/8,127.0.0.1,::1,.corp.com
My cluster FQDNs would follow format <cluster-name>.<where-it-runs>.corp.com
So far, and testing again with my previous images running 33.1.0, connections to AWS exits through proxies, connections to clusters exits directly.
Testing 34.1.0, connection to clusters are mistakenly sent to my proxy, which denies the request.
Forcing HTTPS_PROXY/https_proxy to empty string, when calling my terraform provider: access to cluster OK, but then I broke access to AWS.
I lost that link, but I found somewhere suggesting that some client may handle subdomains differently, so I tried to add corp.com alongside my previous .corp.com, in no_proxy env var. Still no luck.
Then I tried adding the exact cluster FQDN to no_proxy env var: I can still see ansible connecting through my proxies.
What you expected to happen:
domains/subdomains listed in my NO_PROXY/no_proxy should be connected to directly.
How to reproduce it (as minimally and precisely as possible):
Set proxy variables as shown above. Include Kubernetes cluster domain in no_proxy.
Switch from 33.1.0 to 34.1.0. check access logs in proxy.
Anything else we need to know?:
Environment:
- Kubernetes version (
kubectl version): 1.29.7 - OS (e.g., MacOS 10.13.6): RHEL9
- Python version (
python --version): 3.9 - Python client version (
pip list | grep kubernetes): 34.1.0
Activity
faust64 commentedon Oct 2, 2025
I could work around this re-building my image and pinning kubernetes to 33.1.0
I also found that issue in some controller of mine.
That one connects to kubernetes.default.svc, as well as aws/azure APIs, publishing our serviceaccount public signing keys. Running Python 3.12.
Same as my initial issue (a Tekton task running terraform), that controller has environment variables, configuring proxy/no-proxy, such as kubernetes API accesses would remain private, and everything else goes through proxies. Having
no_proxy=localhost,127.0.0.1,.local,.internal,.svc,.default,.corp.com,,169.254.169.254,10.96.0.1Been working fine, up until I merged that last Renovate pull request, upgrading kubernetes. Now I'm getting 403s as controller connects kube API through proxies that don't allow for it.
mlacko64 commentedon Oct 2, 2025
Got same issue, running automated proxy kubernetes build every week, there was no code change on my side and todays build crashed because proxy denied connection to cluster API which but should not go to proxy but direct to API loadbalancer. After downgrading module to 33.1.0 build passed.
p172913 commentedon Oct 4, 2025
/assign
feuerrot commentedon Oct 6, 2025
There's already a pull request which should fix this bug: #2459
daezaa commentedon Nov 4, 2025
Please check #2459 (comment) for more info..
Currently if
HTTP_PROXY/HTTPS_PROXYandNO_PROXYare set,NO_PROXYproxy wouldn't behave as expect..Downgrading to
v33.1.0wouldn't honourHTTP_PROXYorHTTPS_PROXYat all, so it works as ifNO_PROXYworking correctly..The workaround I have is to manually remove the unnecessary line from #2459
eg.
sed -i -e '173d' /usr/local/lib/<python-version>/site-packages/kubernetes/client/configuration.pyroccotigger commentedon Nov 5, 2025
tried to leave a comment in the MR with a potential fix:
something like the following is needed
faust64 commentedon Nov 6, 2025
Maybe we should point out this behavior is consistent with kubectl/oc clients would work.
Doesn't mean you can't use proxies though.
If you want to use a proxy, connecting a kubernetes API, you would just say so, in your kube configuration:
This has been working great for years, can be done in python code, and is usually easier to deal with than NO_PROXY schenanigans, when some API needs to exit directly and others would need some specific proxies
jplitza commentedon Nov 6, 2025
@faust64 If you are saying that kubectl doesn't honor
$HTTPS_PROXY, you are mistaken. It handles the whole set of variables (http_proxy,HTTP_PROXY,https_proxy,HTTPS_PROXY,no_proxy,NO_PROXY) by virtue of using Go's net/http.askervin commentedon Jan 9, 2026
The problem seems to be in kubernetes/client/configuration.py:
That is, after reading environment variables, the generated code forgets what it has read.
python/kubernetes/client/configuration.py
Line 173 in e752675
jcox10 commentedon Jan 26, 2026
35.0.0has the same problem.daezaa commentedon Jan 26, 2026
#2459
The PR is still under review
jcox10 commentedon Jan 29, 2026
It's been 2 months since any progress was made on #2459, is there any hope of getting this issue fixed?