Skip to content

Support HTTP_PROXY #813

Closed
Closed

Description

@marcellodesales

I could successfully use the http json and xml clients using the HTTP_PROXY variable in either the environment or set in the options... Right now, I cannot do that... I'm getting the following:

{ [Error: tunneling socket could not be established, cause=Parse Error] code: 'ECONNRESET' }

Please advise..

Activity

  1. yunong commented on May 29, 2015

    @yunong
    Contributor

    @marcellodesales can you give source code with which you can reproduce this error? Please also include the OS, restify and Node.js versions you're using.

  2. marcellodesales commented on May 29, 2015

    @marcellodesales
    Author

    Any connectivity to the Internet has to go through a HTTP_PROXY in my host... here are some info... I'm using latest restify...

    [mdesales@pppdc9prd8gd ~]$ node --version
    v0.10.36
    [mdesales@pppdc9prd8gd ~]$ uname -a
    Linux pppdc9prd8gd 3.10.0-229.4.2.el7.x86_64 #1 SMP Fri Apr 24 15:26:38 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
    [mdesales@pppdc9prd8gd ~]$ cat node_modules/restify/package.json | grep version
      "version": "2.8.5",
    

    The code was as follows:

    console.log(process.env);
    var restify = require("restify");
    
    var query = "?status=pending";
    var options = {};
    options.url = "http://petstore.swagger.io";
    options.name = options.name || "xxxx-HTTP-Client";
    options.type = options.type || "json";
    options.path = "/v2/pet/findByStatus" + query;
    options.headers = {Accept: "application/json"};
    //options.connectTimeout = 1000 * 5;
    //options.retry = false;
    
    console.log(options);
    var client = restify.createClient(options);
    // HTTP GET request. Taking HTTP_PROXY or https_proxy (or any case)
    client.get(options, function(err, req, res, data) {
      if (err) {
        console.log(err);
        return;
      }
      client.close();
      console.log(JSON.stringify(data));
    });

    The output of this works properly...

    { url: 'http://petstore.swagger.io',
      name: 'xxxx-HTTP-Client',
      type: 'json',
      path: '/v2/pet/findByStatus?status=pending',
      headers: { Accept: 'application/json' } }
    
    [{"id":1432867554,"category":{"id":1432867554,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/ndinge":"tag4"}],"status":"pending"},{"id":1432904252150,"name":"woofwoof jones","photoUrls":[],"tags":[],"status":"pending"}]

    Going through the HTTP_PROXY, the connection just fails with the message above... I had tested this before and it used to work properly...

    [mdesales@pppdc9prd8gd ~]$ time HTTP_PROXY=http://qypprdproxy02.ie.xxxxx.net:80 HTTPS_PROXY=http://qypprdproxy02.ie.xxxxx.net:80 node sample.js 
    

    The first output is the set of environment variables... I removed all the others, but here's only the list of the PROXY ones...

    { HTTP_PROXY: 'http://qypprdproxy02.ie.xxxxx.net:80',
      HTTPS_PROXY: 'http://qypprdproxy02.ie.xxxxx.net:80',
      http_proxy: 'http://qypprdproxy02.ie.xxxxx.net:80',
      FTP_PROXY: 'http://qypprdproxy02.ie.xxxxx.net:80',
      ftp_proxy: 'http://qypprdproxy02.ie.xxxxx.net:80',
      https_proxy: 'http://qypprdproxy02.ie.xxxxx.net:80',
      no_proxy: '.xxxxx.net, .xxxxx.com'}

    The second output is the options used for the client... I'm assuming the HTTP PROXY is automatically used by the client...

    { url: 'http://petstore.swagger.io',
      name: 'xxxxx-HTTP-Client',
      type: 'json',
      path: '/v2/pet/findByStatus?status=pending',
      headers: { Accept: 'application/json' } }
    { [Error: tunneling socket could not be established, cause=Parse Error] code: 'ECONNRESET' }
    
    real    0m15.478s
    user    0m0.405s
    sys 0m0.049s

    From another machine with Internet connection, it works as expected:

    mdesales@ubuntu [05/29/2015 11:28:05] ~/dev/isp (develop) $ node test.js
    { url: 'http://petstore.swagger.io',
      name: 'Intuit-HTTP-Client',
      type: 'json',
      path: '/v2/pet/findByStatus?status=pending',
      headers: { Accept: 'application/json' } }
    [{"id":1432867554,"category":{"id":1432867554,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1432867554,"name":"blank"}],"status":"pending"},{"id":10002,"category":{"id":20002,"name":"category test"},"name":"RUBY UNIT TESTING","photoUrls":[],"tags":[{"id":30002,"name":"tag test"}],"status":"pending"},{"id":1432881297,"category":{"id":1432881297,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1432881297,"name":"blank"}],"status":"pending"},{"id":11020,"category":{"id":56,"name":"sample category name2"},"name":"Csharp test","photoUrls":["sample photoUrls"],"tags":[{"id":11020,"name":"sample tag name2"}],"status":"pending"},{"id":6,"category":{"id":1,"name":"Dogs"},"name":"Dog 3","photoUrls":["url1","url2"],"tags":[{"id":1,"name":"tag3"},{"id":2,"name":"tag4"}],"status":"pending"},{"id":1432904252150,"name":"woofwoof jones","photoUrls":[],"tags":[],"status":"pending"}]

    I have another example with the Request module that works just fine...

    thanks a lot for any help!

  3. BryanDonovan commented on Jun 28, 2015

    @BryanDonovan
    Contributor

    I see this as well. I don't quite get how to use an http proxy with restify (2.8.5 and 3.0.3). I hope I'm just missing something. Here's a gist showing how I'm trying to do this: https://gist.github.com/BryanDonovan/a7c02321b1af1eb216b2

    In that file I can easily use request.js to make a request through a proxy server (also in that file), and request.js uses the same tunnel-agent npm as restify under the hood, so I must be missing a config option somewhere. Is the syntax something like this?

    var restifyClient = restify.createStringClient({
        url: 'http://127.0.0.1:8080',
        proxy: {
            host: '127.0.0.1',
            protocol: 'http:',
            port: 5050
        }
    });

    I also tried omitting the proxy arg and setting my http_proxy env var to http://127.0.0.1:5050 but I get the same result either way:

    { [Error: tunneling socket could not be established, cause=socket hang up] code: 'ECONNRESET' }
    Error: tunneling socket could not be established, cause=socket hang up
        at ClientRequest.onError (/Users/bdonovan/Sites/fuse/proxy-test/node_modules/restify/node_modules/tunnel-agent/index.js:168:17)
        at ClientRequest.g (events.js:199:16)
        at ClientRequest.emit (events.js:107:17)
        at Socket.socketOnEnd (_http_client.js:300:9)
        at Socket.emit (events.js:129:20)
        at _stream_readable.js:908:16
        at process._tickCallback (node.js:355:11)
  4. added this to the Release: 3.1.x milestone on Jun 29, 2015
  5. self-assigned this
    on Jun 29, 2015
  6. micahr commented on Jun 29, 2015

    @micahr
    Contributor

    Thank you for providing an example. I'll take a look and see if I can pinpoint the bug.

  7. marcellodesales commented on Jun 29, 2015

    @marcellodesales
    Author

    @micahr thanks for looking into this... I really need a patch for this! :( I'm glad to verify any patch you may have!

  8. micahr commented on Jul 6, 2015

    @micahr
    Contributor

    After quite a bit of debugging, I believe the issue is from our use of tunnel-agent, which has diverged a lot from the request library which it sprang from, as well as not setting the host and port of the outgoing request appropriately. Until we can get a fix out, you can do these two things:

    When setting up your client do the following, set the agent to false and parse the proxy url:

    var restifyClient = restify.createClient({
        agent: false,
        url: 'http://127.0.0.1:8080',
        proxy: url.parse('http://127.0.0.1:5050')
    });

    You will also need to apply the change from this (aa40eb8) commit, and you should be able to proxy, for now.

    Let me know if those two things combined fixed your issue.

  9. marcellodesales commented on Jul 6, 2015

    @marcellodesales
    Author

    @micahr Unfortunately this did not change... Here's what I tried...

    # root at pppdc9prd5pq in /export/home/mdesales/sp-http-client on git:master x [13:35:41]
    $ cat node_modules/restify/package.json | grep version
      "version": "3.0.3",
    

    Proxy

    # root at pppdc9prd5pq in /export/home/mdesales/sp-http-client on git:master x [13:24:40]
    $ echo $HTTP_PROXY
    http://qypprdproxy02.ie.intuit.net:80
    

    Changes

    I applied the change from aa40eb8 but it seems that it does not make any change...

    $ cat node_modules/restify/lib/clients/http_client.js | grep FIXING
        // CHANGE TO TRY FIXING PROXY as https://github.com/restify/node-restify/issues/813#issuecomment-118744309
        console.log(this);
        if (this.proxy) {
             opts.host = this.proxy.hostname;
             opts.port = this.proxy.port;
        }

    I also change the cline above as follows:

    var restify = require("restify");
    var url = require("url");
    
    var query = "?status=pending";
    var options = {};
    options.url = "http://petstore.swagger.io";
    options.name = options.name || "xxxx-HTTP-Client";
    options.type = options.type || "json";
    options.path = "/v2/pet/findByStatus" + query;
    options.headers = {Accept: "application/json"};
    
    // PROXY SUPPORT
    options.agent = false;
    options.proxy = url.parse(process.env.HTTP_PROXY);
    
    var client = restify.createClient(options);
    // HTTP GET request. Taking HTTP_PROXY or https_proxy (or any case)
    client.get(options, function(err, req, res, data) {
      if (err) {
        console.log(err);
        return;
      }
      client.close();
      console.log(JSON.stringify(data));
    });

    Running the test class

    I also print the this object from the client change

    # root at pppdc9prd5pq in /export/home/mdesales/sp-http-client on git:master x [13:31:22]
    $ HTTP_PROXY=$HTTP_PROXY node test.js
    { url: 'http://petstore.swagger.io',
      name: 'xxxx-HTTP-Client',
      type: 'json',
      path: '/v2/pet/findByStatus?status=pending',
      headers: { Accept: 'application/json' },
      agent: false,
      proxy:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'qypprdproxy02.ie.intuit.net:80',
         port: '80',
         hostname: 'qypprdproxy02.ie.intuit.net',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://qypprdproxy02.ie.intuit.net:80/' } }
    ### LIBRARY ####
    { domain: null,
      _events: {},
      _maxListeners: 10,
      agent: false,
      ca: undefined,
      cert: undefined,
      ciphers: undefined,
      connectTimeout: false,
      requestTimeout: false,
      headers:
       { Accept: 'application/json',
         accept: 'application/json',
         'content-type': 'application/json',
         'user-agent': 'restify/3.0.3 (x64-linux; v8/3.14.5.10; OpenSSL/1.0.1e-fips) node/0.10.26' },
      log:
       { domain: null,
         _events: {},
         _maxListeners: 10,
         _level: 20,
         streams: [ [Object], [Object] ],
         serializers:
          { err: [Function: err],
            req: [Function: req],
            res: [Function: res],
            client_req: [Function: clientReq],
            client_res: [Function: clientRes] },
         src: false,
         fields:
          { name: 'xxxx-HTTP-Client',
            hostname: 'pppdc9prd5pq',
            pid: 19912 } },
      key: undefined,
      name: 'xxxx-HTTP-Client',
      passphrase: undefined,
      pfx: undefined,
      rejectUnauthorized: true,
      proxy:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'qypprdproxy02.ie.intuit.net:80',
         port: '80',
         hostname: 'qypprdproxy02.ie.intuit.net',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://qypprdproxy02.ie.intuit.net:80/' },
      retry: { minTimeout: 1000, maxTimeout: Infinity, retries: 4 },
      signRequest: false,
      socketPath: false,
      url:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'petstore.swagger.io',
         port: null,
         hostname: 'petstore.swagger.io',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://petstore.swagger.io/' },
      gzip: undefined,
      _super:
       { post: [Function: post],
         put: [Function: put],
         patch: [Function: patch],
         read: [Function: read],
         write: [Function: write],
         parse: [Function: parse] } }

    Curl works

    $ time curl "http://petstore.swagger.io/v2/pet/findByStatus?status=pending"
    [{"id":9527,"name":"Kitty","photoUrls":[],"tags":[],"status":"pending"},{"id":1435804807,"category":{"id":1435804807,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1435804807,"name":"blank"}],"status":"pending"},{"id":1435805127,"category":{"id":1435805127,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1435805127,"name":"blank"}],"status":"pending"},{"id":1435805202,"category":{"id":1435805202,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1435805202,"name":"blank"}],"status":"pending"},{"id":1435819364,"category":{"id":1435819364,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1435819364,"name":"blank"}],"status":"pending"},{"id":1435819505,"category":{"id":1435819505,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1435819505,"name":"blank"}],"status":"pending"},{"id":10002,"category":{"id":20002,"name":"category test"},"name":"RUBY UNIT TESTING","photoUrls":[],"tags":[{"id":30002,"name":"tag test"}],"status":"pending"},{"id":1436176466,"category":{"id":1436176466,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1436176466,"name":"blank"}],"status":"pending"},{"id":1436176534,"category":{"id":1436176534,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1436176534,"name":"blank"}],"status":"pending"},{"id":1436176890,"category":{"id":1436176890,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1436176890,"name":"blank"}],"status":"pending"},{"id":1436177116,"category":{"id":1436177116,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1436177116,"name":"blank"}],"status":"pending"},{"id":1436177156,"category":{"id":1436177156,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1436177156,"name":"blank"}],"status":"pending"},{"id":1436177425,"category":{"id":1436177425,"name":"dog"},"name":"hello kity with form updated","photoUrls":["http://foo.bar.com/1","http://foo.bar.com/2"],"tags":[{"id":1436177425,"name":"blank"}],"status":"pending"}]curl "http://petstore.swagger.io/v2/pet/findByStatus?status=pending" 
    
    0.00s user 0.00s system 5% cpu 0.054 total

    Potential improvements

    As you don't check for cases, clients might not get this properly set... Some environments use the environment variable on its upper-case form HTTP_PROXY instead of http_proxy.

    $ cat node_modules/restify/lib/clients/http_client.js | grep proxy
        if (process.env.https_proxy) {
            this.proxy = url.parse(process.env.https_proxy);
        } else if (process.env.http_proxy) {
            this.proxy = url.parse(process.env.http_proxy);
        } else if (options.proxy) {
            this.proxy = options.proxy;
            this.proxy = false;
            if (this.proxy) {
                    if (this.proxy.protocol === 'https:') {
                    if (this.proxy.protocol === 'https:') {
            if (this.proxy) {
                    proxy: self.proxy,
    
        // CHANGE TO TRY FIXING PROXY as https://github.com/restify/node-restify/issues/813#issuecomment-118744309
        if (this.proxy) {
             opts.host = this.proxy.hostname;
             opts.port = this.proxy.port;
  10. micahr commented on Jul 6, 2015

    @micahr
    Contributor

    Looks like your request path isn't being set to the full url you are trying to access.

    Try setting your options.path to http://petstore.swagger.io/v2/pet/findByStatus?status=pending for your example above and see if that helps at all.

  11. marcellodesales commented on Jul 6, 2015

    @marcellodesales
    Author

    @micahr I tried changing both options.path and options.url. None of them works... It looks like the URL is being parsed, so changing "options.path" DOES NOT HAVE ANY AFFECT...

    path change

    # root at pppdc9prd5pq in /export/home/mdesales/sp-http-client on git:master x [14:24:44]
    $ HTTP_PROXY=$HTTP_PROXY node test.js
    { url: 'http://petstore.swagger.io',
      name: 'xxxx-HTTP-Client',
      type: 'json',
      path: 'http://petstore.swagger.io/v2/pet/findByStatus?status=pending',
      headers: { Accept: 'application/json' },
      agent: false,
      proxy:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'qypprdproxy02.ie.intuit.net:80',
         port: '80',
         hostname: 'qypprdproxy02.ie.intuit.net',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://qypprdproxy02.ie.intuit.net:80/' } }
    ### LIBRARY ####
    { domain: null,
      _events: {},
      _maxListeners: 10,
      agent: false,
      ca: undefined,
      cert: undefined,
      ciphers: undefined,
      connectTimeout: false,
      requestTimeout: false,
      headers:
       { Accept: 'application/json',
         accept: 'application/json',
         'content-type': 'application/json',
         'user-agent': 'restify/3.0.3 (x64-linux; v8/3.14.5.10; OpenSSL/1.0.1e-fips) node/0.10.26' },
      log:
       { domain: null,
         _events: {},
         _maxListeners: 10,
         _level: 20,
         streams: [ [Object], [Object] ],
         serializers:
          { err: [Function: err],
            req: [Function: req],
            res: [Function: res],
            client_req: [Function: clientReq],
            client_res: [Function: clientRes] },
         src: false,
         fields:
          { name: 'xxxx-HTTP-Client',
            hostname: 'pppdc9prd5pq',
            pid: 20446 } },
      key: undefined,
      name: 'xxxx-HTTP-Client',
      passphrase: undefined,
      pfx: undefined,
      rejectUnauthorized: true,
      proxy:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'qypprdproxy02.ie.intuit.net:80',
         port: '80',
         hostname: 'qypprdproxy02.ie.intuit.net',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://qypprdproxy02.ie.intuit.net:80/' },
      retry: { minTimeout: 1000, maxTimeout: Infinity, retries: 4 },
      signRequest: false,
      socketPath: false,
      url:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'petstore.swagger.io',
         port: null,
         hostname: 'petstore.swagger.io',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://petstore.swagger.io/' },
      gzip: undefined,
      _super:
       { post: [Function: post],
         put: [Function: put],
         patch: [Function: patch],
         read: [Function: read],
         write: [Function: write],
         parse: [Function: parse] } }
    ^C#

    URL change

    $ HTTP_PROXY=$HTTP_PROXY node test.js
    { url: 'http://petstore.swagger.io/v2/pet/findByStatus?status=pending',
      name: 'xxxx-HTTP-Client',
      type: 'json',
      path: 'http://petstore.swagger.io/v2/pet/findByStatus?status=pending',
      headers: { Accept: 'application/json' },
      agent: false,
      proxy:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'qypprdproxy02.ie.intuit.net:80',
         port: '80',
         hostname: 'qypprdproxy02.ie.intuit.net',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://qypprdproxy02.ie.intuit.net:80/' } }
    ### LIBRARY ####
    { domain: null,
      _events: {},
      _maxListeners: 10,
      agent: false,
      ca: undefined,
      cert: undefined,
      ciphers: undefined,
      connectTimeout: false,
      requestTimeout: false,
      headers:
       { Accept: 'application/json',
         accept: 'application/json',
         'content-type': 'application/json',
         'user-agent': 'restify/3.0.3 (x64-linux; v8/3.14.5.10; OpenSSL/1.0.1e-fips) node/0.10.26' },
      log:
       { domain: null,
         _events: {},
         _maxListeners: 10,
         _level: 20,
         streams: [ [Object], [Object] ],
         serializers:
          { err: [Function: err],
            req: [Function: req],
            res: [Function: res],
            client_req: [Function: clientReq],
            client_res: [Function: clientRes] },
         src: false,
         fields:
          { name: 'xxxx-HTTP-Client',
            hostname: 'pppdc9prd5pq',
            pid: 20407 } },
      key: undefined,
      name: 'xxxx-HTTP-Client',
      passphrase: undefined,
      pfx: undefined,
      rejectUnauthorized: true,
      proxy:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'qypprdproxy02.ie.intuit.net:80',
         port: '80',
         hostname: 'qypprdproxy02.ie.intuit.net',
         hash: null,
         search: null,
         query: null,
         pathname: '/',
         path: '/',
         href: 'http://qypprdproxy02.ie.intuit.net:80/' },
      retry: { minTimeout: 1000, maxTimeout: Infinity, retries: 4 },
      signRequest: false,
      socketPath: false,
      url:
       { protocol: 'http:',
         slashes: true,
         auth: null,
         host: 'petstore.swagger.io',
         port: null,
         hostname: 'petstore.swagger.io',
         hash: null,
         search: '?status=pending',
         query: 'status=pending',
         pathname: '/v2/pet/findByStatus',
         path: '/v2/pet/findByStatus?status=pending',
         href: 'http://petstore.swagger.io/v2/pet/findByStatus?status=pending' },
      gzip: undefined,
      _super:
       { post: [Function: post],
         put: [Function: put],
         patch: [Function: patch],
         read: [Function: read],
         write: [Function: write],
         parse: [Function: parse] } }
    ^C#
  12. marcellodesales commented on Jul 19, 2015

    @marcellodesales
    Author

    @micahr Any news on this?

  13. micahr commented on Jul 20, 2015

    @micahr
    Contributor

    Sorry, nothing yet. If you are blocked, I'd suggest using the request library for the time being.

  14. marcellodesales commented on Jul 20, 2015

    @marcellodesales
    Author

    Sounds good...

  15. changed the title [-]tunneling socket could not be established: ECONNRESET when trying to use HTTP_PROXY[/-] [+]Support HTTP_PROXY[/+] on Apr 29, 2017
  16. added a commit that references this issue on Apr 29, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions