Seriously bad advice about SSL errors in Python and a better fix

This is a post about fixing certificate verify failed error in Python without compromising security of your code. Don’t blindly follow suggestions on the internet because some suggested solutions can have serious unintended consequences.

Yesterday was a casual day in the life of a software developer. Breakfast, coffee, start Jupyter, work on our app, and hit an issue with the most trivial thing - making a simple web request. “certificate verify failed” was a message I did not expect. To be sure we don’t have a problem with the certificate on our server, I tried requesting https://google.com. The error did not go away. I fired up my brain. I put my heart and soul in it. I dug deep into my years of coding experience. All the caffeine and sugar flowing through my veins were now focused on the quest to fix the issue. This is the search query I came up with: “python urllib3 certificate verify failed”. There were hits on StackOverflow with the exact same issue, which is a good thing. It looks like I’m not the only one seeing this error. But the quality of suggested fixes is questionable. Multiple suggested fixes suggested disabling of certificate verification in some way or another. This is really bad advice. It fixes the problem in the sense that the error goes away, but it leaves you exposed to man in the middle attack.

Better way to fix this issue

Most likely, the issue is caused by one of the following:

• misconfigured or missing list of trusted certificate authorities
• the interplay between different Python libraries for issuing HTTPS requests
• self-signed certificate on the server or a signature by a not trusted certificate authority

In my particular situation I discovered that this didn’t work:

import urllib3
import requests
http = urllib3.PoolManager()
r = http.request('GET', 'https://google.com')

but this did:

import urllib3

http = urllib3.PoolManager()
r = http.request('GET', 'https://google.com')

I browsed through the source code of the requests package and found the following lines:

from urllib3.contrib import pyopenssl	 
pyopenssl.inject_into_urllib3()

There was an easy way to reverse this:

import urllib3
import requests
from urllib3.contrib import pyopenssl
pyopenssl.extract_from_urllib3()

Just to make sure, I did the following tests:

import urllib3
import requests
from urllib3.contrib import pyopenssl
pyopenssl.extract_from_urllib3()

r = http.request('GET', 'https://google.com')
print(r.status) #prints '200'

import urllib3
import requests
from urllib3.contrib import pyopenssl
pyopenssl.extract_from_urllib3()

r = requests.get('https://google.com')
print(r.status_code) #prints '200'

The most important checks are the following two, which demonstrate that the certificate validation still works and catches invalid certificates. We expect to see exceptions on both of the following tests:

import urllib3
import requests
from urllib3.contrib import pyopenssl
pyopenssl.extract_from_urllib3()

r = http.request('GET', 'https://incomplete-chain.badssl.com/')
# throws urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='incomplete-chain.badssl.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))

import urllib3
import requests
from urllib3.contrib import pyopenssl
pyopenssl.extract_from_urllib3()

r = requests.get('https://incomplete-chain.badssl.com')
# throws requests.exceptions.SSLError: HTTPSConnectionPool(host='incomplete-chain.badssl.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))

If you don’t see exceptions, something is wrong, and the validation doesn’t work. Check whether your Python is compiled with SSL.

Lesson learned here is that you should understand the implications of solutions to coding problems offered on online forums. Stackoverflow.com has a great feature of up/down-voting responses and commenting on them. But sometimes bad suggestions slip under the radar. You should always check the code you use.