Tuesday 14 January 2020

make any internet-accessing python code work (proxy + custom .crt)

The situation

If the following is not done, all outgoing HTTP or HTTPS requests made with python ends in a WinError 10054 Connection Reset, or a SSL bad handshake error.

  1. set the HTTP_PROXY, HTTPS_PROXY environment variable, or their counterparts
  2. What needs to be verified must be verified with a custom .crt file.

For example, assuming the .crt file is in place, both gets me a 200 OK:

import os
os.environ['HTTP_PROXY'] = #some_appropriate_address
os.environ['HTTPS_PROXY'] = #some appropriate_address
requests.get('http://www.google.com',verify="C:\the_file_nada_nada.crt")
requests.get('http://httpbin.org',verify=False)

Also, the following each gives response, without any errors.

curl http://httpbin.org -x "http://10.114.1.180:8080" -E "C:\the_file.crt
curl http://httpbin.org -x "http://10.114.1.180:8080"

-x is the option for proxy, -E is the option for cert.

curl http://httpbin.org

gives me a curl: (35) schannel: failed to receive handshake, SSL/TLS connection failed.

curl http://httpbin.org -E "C:\the_file.crt"

gives me a curl: (56) Recv failure: Connection was reset. with only the -x, the response is received properly.(And yes, that is where my .crt file is located.)

The Problem

If you want some other piece of code(for instance tf.keras.datasets.mnist.load_data() or code written with urllib3), which works perfectly under no firewall, work in a setting like mine, what should I do?

Is there some global setting(environment variable etc.) you can alter, so that an internet-accessing python code would also work(return a 200 OK upon a GET request to an eligible server) under a "proxy+custom .crt file" setting explained above? (Also, If there is no such way, why is this so?)

What I tried(am trying)

Maybe editing the .condarc file(via conda --config) is a solution. I tried, to no avail: python gives me a "SSL verification failed" error. On the contrary, note that the code snippet above gave me a 200 OK. To my knowledge, this does not fit nicely with many situations that were previousy discussed in stackoverflow.

By the way, setting ssl_verify to false does not solve the problem either; I still get a bad handshake error for some reason.

Update: Win 10, python 3.7.4(Anaconda).

Another update: bounty started, and mended some expressions.



from make any internet-accessing python code work (proxy + custom .crt)

No comments:

Post a Comment