Requests Library Force Use Of Http/1.1 On Https Proxy Connect

I am having a problem with a misbehaving HTTP Proxy server. I have no control over the proxy server, unfortunately -- it's an 'enterprise' product from IBM. The proxy server is par

Solution 1:

httplib (which requests relies upon for HTTP(S) heavy lifting) always uses HTTP/1.0 with CONNECT:

Lib/httplib.py:788:

def _tunnel(self):
    self.send("CONNECT %s:%d HTTP/1.0\r\n"% (self._tunnel_host,
        self._tunnel_port))
    for header, value inself._tunnel_headers.iteritems():
        self.send("%s: %s\r\n"% (header, value))
    self.send("\r\n")
    <...>

So you can't "force" it to use "HTTP/1.1" here other than by editing the subroutine.

---

This MAY be the problem if the proxy doesn't support HTTP/1.0 - in particular, 1.0 does not require a Host: header, and indeed, as you can see by comparing your log output with the code above, httplib does not send it. While, in verity, a proxy may expect it regardless. But if this is the case, you should've gotten an error from the proxy or something in response to CONNECT -- unless the proxy is so borken that it substitutes some default (or garbage) for Host:, returns 200 anyway and tries to connect God-knows-where, at which point you're getting timeouts.

You can make httplib add the Host: header to CONNECT by adding it to _tunnel_headers (indirectly):

s=requests.Session()
proxy_url=os.environ['HTTPS_PROXY']
s.proxies["https"]=proxy_url
# have to specify proxy here because env variable is only detected by httplib code#while we need to trigger requests' proxy logic that acts earlier# "https" means any https host. Since a Session persists cookies,#it's meaningless to make requests to multiple hosts through it anyway.

pm=s.get_adapter("https://").proxy_manager_for(proxy_url)
pm.proxy_headers['Host']="host.com"
del pm,proxy_url
<...>
s.get('https://host.com')

Solution 2:

If you do not depend on the requests library you may find the following snippet useful:

import http.client

conn = http.client.HTTPSConnection("proxy.domain.lu", 8080)
conn.set_tunnel("www.domain.org", 443, headers={'User-Agent': 'curl/7.56.0'})
conn.request("GET", "/api")
response = conn.getresponse()

print( response.read() )