Skip to content

NO_PROXY/no_proxy env vars not working as expected, as of 34.1.0 #2460

Closed

Description

@faust64

What happened (please include outputs or screenshots):
Upgrading libraries in one of my images. Last image was built 4 days ago, works fine. We were running kubernetes 33.1.0.
Today's image is pulling 34.1.0. KO

Picking apart each package version that changed as of my previous image, I ended up suspecting kubernetes client itself.
v33.1.0...v34.1.0

Obviously, this:
e3b373f#diff-977bedbaf9339b25749bc96a24266c1fa3e99ab521a835562fe84b2ab87b7001

And then again, the rest client, which is the one throwing in my case, looks OK?
https://github.com/kubernetes-client/python/blob/master/kubernetes/client/rest.py

That "should_bypass_proxy" ... looks correct ...
https://requests.readthedocs.io/en/latest/_modules/requests/utils/

We are using proxies. Ansible would connect public services (eg: aws). Our proxies would only allow for specific names out.
Ansible would connect to our clusters directly/no proxy.

Our environment would include:

  HTTP_PROXY: http://my-http-proxy:8080
  HTTPS_PROXY: http://my-http-proxy:8080
  NO_PROXY: github.corp.com,.github.corp.com,localhost,.local,.svc,10.0.0.0/8,127.0.0.0/8,127.0.0.1,::1,.corp.com
  http_proxy: http://my-http-proxy:8080
  https_proxy: http://my-http-proxy:8080
  no_proxy: github.corp.com,.github.corp.com,localhost,.local,.svc,10.0.0.0/8,127.0.0.0/8,127.0.0.1,::1,.corp.com

My cluster FQDNs would follow format <cluster-name>.<where-it-runs>.corp.com
So far, and testing again with my previous images running 33.1.0, connections to AWS exits through proxies, connections to clusters exits directly.
Testing 34.1.0, connection to clusters are mistakenly sent to my proxy, which denies the request.
Forcing HTTPS_PROXY/https_proxy to empty string, when calling my terraform provider: access to cluster OK, but then I broke access to AWS.

I lost that link, but I found somewhere suggesting that some client may handle subdomains differently, so I tried to add corp.com alongside my previous .corp.com, in no_proxy env var. Still no luck.
Then I tried adding the exact cluster FQDN to no_proxy env var: I can still see ansible connecting through my proxies.

What you expected to happen:

domains/subdomains listed in my NO_PROXY/no_proxy should be connected to directly.

How to reproduce it (as minimally and precisely as possible):

Set proxy variables as shown above. Include Kubernetes cluster domain in no_proxy.
Switch from 33.1.0 to 34.1.0. check access logs in proxy.

Anything else we need to know?:

Environment:

  • Kubernetes version (kubectl version): 1.29.7
  • OS (e.g., MacOS 10.13.6): RHEL9
  • Python version (python --version): 3.9
  • Python client version (pip list | grep kubernetes): 34.1.0

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    on Oct 1, 2025
  2. faust64 commented on Oct 2, 2025

    @faust64
    Author

    I could work around this re-building my image and pinning kubernetes to 33.1.0

    I also found that issue in some controller of mine.
    That one connects to kubernetes.default.svc, as well as aws/azure APIs, publishing our serviceaccount public signing keys. Running Python 3.12.

    Same as my initial issue (a Tekton task running terraform), that controller has environment variables, configuring proxy/no-proxy, such as kubernetes API accesses would remain private, and everything else goes through proxies. Having no_proxy=localhost,127.0.0.1,.local,.internal,.svc,.default,.corp.com,,169.254.169.254,10.96.0.1

    Been working fine, up until I merged that last Renovate pull request, upgrading kubernetes. Now I'm getting 403s as controller connects kube API through proxies that don't allow for it.

    Traceback (most recent call last):
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 773, in urlopen
        self._prepare_proxy(conn)
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 1042, in _prepare_proxy
        conn.connect()
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connection.py", line 721, in connect
        self._tunnel()
      File "/usr/lib64/python3.12/http/client.py", line 981, in _tunnel
        raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
    OSError: Tunnel connection failed: 403 Forbidden
    
    The above exception was the direct cause of the following exception:
    
    urllib3.exceptions.ProxyError: ('Unable to connect to proxy', OSError('Tunnel connection failed: 403 Forbidden'))
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/opt/app-root/src/app.py", line 25, in <module>
        dyn_client = dynamic.DynamicClient(k8s_client)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/client.py", line 84, in __init__
        self.__discoverer = discoverer(self, cache_file)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 230, in __init__
        Discoverer.__init__(self, client, cache_file)
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 55, in __init__
        self.__init_cache()
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 71, in __init_cache
        self._load_server_info()
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 146, in _load_server_info
        'kubernetes': self.client.request('get', '/version', serializer=just_json)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/client.py", line 55, in inner
        resp = func(self, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/client.py", line 277, in request
        api_response = self.client.call_api(
                       ^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/api_client.py", line 348, in call_api
        return self.__call_api(resource_path, method,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
        response_data = self.request(
                        ^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/api_client.py", line 373, in request
        return self.rest_client.GET(url,
               ^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/rest.py", line 244, in GET
        return self.request("GET", url,
               ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/rest.py", line 217, in request
        r = self.pool_manager.request(method, url,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/_request_methods.py", line 135, in request
        return self.request_encode_url(
               ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
        return self.urlopen(method, url, **extra_kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/poolmanager.py", line 633, in urlopen
        return super().urlopen(method, url, redirect=redirect, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/poolmanager.py", line 443, in urlopen
        response = conn.urlopen(method, u.request_uri, **kw)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
        return self.urlopen(
               ^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
        return self.urlopen(
               ^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
        return self.urlopen(
               ^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
        retries = retries.increment(
                  ^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
        raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /version (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 403 Forbidden')))
    
  3. mlacko64 commented on Oct 2, 2025

    @mlacko64

    Got same issue, running automated proxy kubernetes build every week, there was no code change on my side and todays build crashed because proxy denied connection to cluster API which but should not go to proxy but direct to API loadbalancer. After downgrading module to 33.1.0 build passed.

  4. p172913 commented on Oct 4, 2025

    @p172913
    Contributor

    /assign

  5. feuerrot commented on Oct 6, 2025

    @feuerrot

    There's already a pull request which should fix this bug: #2459

  6. daezaa commented on Nov 4, 2025

    @daezaa
    Contributor

    Please check #2459 (comment) for more info..

    Currently if HTTP_PROXY/HTTPS_PROXY and NO_PROXY are set, NO_PROXY proxy wouldn't behave as expect..
    Downgrading to v33.1.0 wouldn't honour HTTP_PROXY or HTTPS_PROXY at all, so it works as if NO_PROXY working correctly..

    The workaround I have is to manually remove the unnecessary line from #2459
    eg. sed -i -e '173d' /usr/local/lib/<python-version>/site-packages/kubernetes/client/configuration.py

  7. roccotigger commented on Nov 5, 2025

    @roccotigger

    tried to leave a comment in the MR with a potential fix:
    something like the following is needed

    
        BLOCK=""
        NOPROXY_BLOCK=""
    
        if [ -z "$NO_PROXY_LINE" ]; then
          # self.no_proxy = None is not present → insert full block after self.proxy = None
          BLOCK+="${INDENT}# Load proxy from environment variables (if set)\n"
          BLOCK+="${INDENT}if os.getenv(\"HTTPS_PROXY\"): self.proxy = os.getenv(\"HTTPS_PROXY\")\n"
          BLOCK+="${INDENT}if os.getenv(\"https_proxy\"): self.proxy = os.getenv(\"https_proxy\")\n"
          BLOCK+="${INDENT}if os.getenv(\"HTTP_PROXY\"): self.proxy = os.getenv(\"HTTP_PROXY\")\n"
          BLOCK+="${INDENT}if os.getenv(\"http_proxy\"): self.proxy = os.getenv(\"http_proxy\")\n"
    
          sed -i "${PROXY_LINE}a $BLOCK" "$CONFIG_FILE"
    
          NOPROXY_BLOCK+="${INDENT}# Load no_proxy from environment variables (if set)\n"
          NOPROXY_BLOCK+="${INDENT}if os.getenv(\"NO_PROXY\"): self.no_proxy = os.getenv(\"NO_PROXY\")\n"
          NOPROXY_BLOCK+="${INDENT}if os.getenv(\"no_proxy\"): self.no_proxy = os.getenv(\"no_proxy\")"
    
          sed -i "${NO_PROXY_LINE}a $NOPROXY_BLOCK" "$CONFIG_FILE"
    
          echo "Inserted proxy block after 'self.proxy = None'. and no_proxy block after 'self.no_proxy = None'."
        else
    
  8. faust64 commented on Nov 6, 2025

    @faust64
    Author

    Downgrading to v33.1.0 wouldn't honour HTTP_PROXY or HTTPS_PROXY at all

    Maybe we should point out this behavior is consistent with kubectl/oc clients would work.
    Doesn't mean you can't use proxies though.

    If you want to use a proxy, connecting a kubernetes API, you would just say so, in your kube configuration:

    clusters:
    - cluster:
        proxy-url: http://my-egressproxy:3128
        server: https://api.stuff.com:6443
      name: foo
    

    This has been working great for years, can be done in python code, and is usually easier to deal with than NO_PROXY schenanigans, when some API needs to exit directly and others would need some specific proxies

  9. jplitza commented on Nov 6, 2025

    @jplitza

    @faust64 If you are saying that kubectl doesn't honor $HTTPS_PROXY, you are mistaken. It handles the whole set of variables (http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy, NO_PROXY) by virtue of using Go's net/http.

  10. askervin commented on Jan 9, 2026

    @askervin

    The problem seems to be in kubernetes/client/configuration.py:

            if os.getenv("HTTPS_PROXY"): self.proxy = os.getenv("HTTPS_PROXY")
            if os.getenv("https_proxy"): self.proxy = os.getenv("https_proxy")
            if os.getenv("HTTP_PROXY"): self.proxy = os.getenv("HTTP_PROXY")
            if os.getenv("http_proxy"): self.proxy = os.getenv("http_proxy")
            self.no_proxy = None
            # Load no_proxy from environment variables (if set)
            if os.getenv("NO_PROXY"): self.no_proxy = os.getenv("NO_PROXY")
            if os.getenv("no_proxy"): self.no_proxy = os.getenv("no_proxy")
            """Proxy URL
            """
            self.no_proxy = None

    That is, after reading environment variables, the generated code forgets what it has read.

    self.no_proxy = None

  11. jcox10 commented on Jan 26, 2026

    @jcox10
    Contributor

    35.0.0 has the same problem.

  12. daezaa commented on Jan 26, 2026

    @daezaa
    Contributor

    #2459
    The PR is still under review

  13. jcox10 commented on Jan 29, 2026

    @jcox10
    Contributor

    It's been 2 months since any progress was made on #2459, is there any hope of getting this issue fixed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions