Python Urllib2 SSL error

by

To summarize the comments about the cause of the problem and explain the real problem in more detail:

If you check the trust chain for the OpenSSL client you get the following:

 [0] 54:7D:B3:AC:BF:... /CN=*.s3.amazonaws.com 
 [1] 5D:EB:8F:33:9E:... /CN=VeriSign Class 3 Secure Server CA - G3
 [2] F4:A8:0A:0C:D1:... /CN=VeriSign Class 3 Public Primary Certification Authority - G5
[OT] A1:DB:63:93:91:... /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority

The first certificate [0] is the leaf certificate sent by the server. The following certifcates [1] and [2] are chain certificates sent by the server. The last certificate [OT] is the trusted root certificate, which is not sent by the server but is in the local storage of trusted CA. Each certificate in the chain is signed by the next one and the last certificate [OT] is trusted, so the trust chain is complete.

If you check the trust chain instead by a browser (e.g. Google Chrome using the NSS library) you get the following chain:

 [0] 54:7D:B3:AC:BF:... /CN=*.s3.amazonaws.com 
 [1] 5D:EB:8F:33:9E:... /CN=VeriSign Class 3 Secure Server CA - G3
[NT] 4E:B6:D5:78:49:... /CN=VeriSign Class 3 Public Primary Certification Authority - G5

Here [0] and [1] are again sent by the server, but [NT] is the trusted root certificate. While this looks from the subject exactly like the chain certificate [2] the fingerprint says that the certificates are different. If you would take a closer looks at the certificates [2] and [NT] you would see, that the public key inside the certificate is the same and thus both [2] and [NT] can be used to verify the signature for [1] and thus can be used to build the trust chain.

This means, that while the server sends the same certificate chain in all cases there are multiple ways to verify the chain up to a trusted root certificate. How this is done depends on the SSL library and on the known trusted root certificates:

                          [0] (*.s3.amazonaws.com)
                           |
                          [1] (Verisign G3) --------------------------\
                           |                                          |
      /------------------ [2] (Verisign G5 F4:A8:0A:0C:D1...)         |
      |                                                               |
      |              certificates sent by server                      |
 .....|...............................................................|................
      |              locally trusted root certificates                |
      |                                                               |
     [OT] Public Primary Certification Authority        [NT] Verisign G5 4E:B6:D5:78:49
     OpenSSL library                                    Google Chrome (NSS library)

But the question remains, why your verification was unsuccessful.
What you did was to take the trusted root certificate used by the browser (Verisign G5 4E:B6:D5:78:49) together with OpenSSL. But the verification in browser (NSS) and OpenSSL work slightly different:

  • NSS: build trust chain from certificates send by the server. Stop building the chain when we got a certificate signed by any of the locally trusted root certificates.
  • OpenSSL_ build trust chain from the certificates sent by the server. After this is done check if we have a trusted root certificate signing the latest certificate in the chain.

Because of this subtle difference OpenSSL is not able to verify the chain [0],[1],[2] against root certificate [NT], because this certificate does not sign the latest element in chain [2] but instead [1]. If the server would instead only sent a chain of [0],[1] then the verification would succeed.

This is a long known bug and there exist patches and hopefully the issue if finally addressed in OpenSSL 1.0.2 with the introduction of the X509_V_FLAG_TRUSTED_FIRST option.

More Related Contents:

  1. urllib and “SSL: CERTIFICATE_VERIFY_FAILED” Error
  2. pip install fails with “connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)”
  3. Python Requests throwing SSLError
  4. “ssl module in Python is not available” when installing package with pip3
  5. Import error: No module name urllib2
  6. Proxy with urllib2
  7. Using an HTTP PROXY – Python [duplicate]
  8. using requests with TLS doesn’t give SNI support
  9. Python: download files from google drive using url
  10. Python handling socket.error: [Errno 104] Connection reset by peer
  11. Using MultipartPostHandler to POST form-data with Python
  12. Python-Requests close http connection
  13. Python FTP implicit TLS connection issue
  14. python ignore certificate validation urllib2
  15. Python – requests.exceptions.SSLError – dh key too small
  16. OpenSSL errors in python requests
  17. Requests failing to connect to a TLS server
  18. How to make Python use CA certificates from Mac OS TrustStore?
  19. Making a POST call instead of GET using urllib2
  20. How can I speed up fetching pages with urllib2 in python?
  21. Unescape Python Strings From HTTP
  22. Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”
  23. open url from pythonanywhere
  24. Submitting to a web form using python
  25. urllib2 HTTP Error 400: Bad Request
  26. Limiting number of processes in multiprocessing python
  27. Download and decompress gzipped file in memory?
  28. Connecting to ‘Explicit FTP over TLS’ in Python (??)
  29. urllib2 and json
  30. How can I decode a SSL certificate using python?

Leave a Comment