“SSL: certificate_verify_failed” error when scraping https://www.thenewboston.com/

by

The problem is not in your code but in the web site you are trying to access. When looking at the analysis by SSLLabs you will note:

This server’s certificate chain is incomplete. Grade capped to B.

This means that the server configuration is wrong and that not only python but several others will have problems with this site. Some desktop browsers work around this configuration problem by trying to load the missing certificates from the internet or fill in with cached certificates. But other browsers or applications will fail too, similar to python.

To work around the broken server configuration you might explicitly extract the missing certificates and add them to you trust store. Or you might give the certificate as trust inside the verify argument. From the documentation:

You can pass verify the path to a CA_BUNDLE file or directory with
certificates of trusted CAs:

>>> requests.get('https://github.com', verify='/path/to/certfile')

This list of trusted CAs can also be specified through the
REQUESTS_CA_BUNDLE environment variable.

More Related Contents:

  1. urllib and “SSL: CERTIFICATE_VERIFY_FAILED” Error
  2. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  3. using requests with TLS doesn’t give SNI support
  4. Python requests SSL error – certificate verify failed
  5. Python Requests getting SSLerror
  6. SSL error with Python requests despite up-to-date dependencies
  7. What does “SSLError: [SSL] PEM lib (_ssl.c:2532)” mean using the Python ssl library?
  8. SSL error downloading NLTK data
  9. Scraping Javascript driven web pages with PyQt4 – how to access pages that need authentication?
  10. pip install fails with “connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)”
  11. Python Requests throwing SSLError
  12. How to find elements by class
  13. “ssl module in Python is not available” when installing package with pip3
  14. Using BeautifulSoup to extract text without tags
  15. Scrape multiple urls using QWebPage
  16. Scrapy Very Basic Example
  17. Python: find_element_by_css_selector
  18. Python FTP implicit TLS connection issue
  19. Scraping ajax pages using python
  20. Dynamic dropdown doesn’t populate with auto suggestions on https://www.nseindia.com/ when values are passed using Selenium and Python
  21. OpenSSL errors in python requests
  22. Scraping dynamic content in a website [duplicate]
  23. Scrape multiple pages with BeautifulSoup and Python
  24. Logging in to LinkedIn with python requests sessions
  25. Can bs4 get the dynamic content of a webpage if requests can’t?
  26. Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”
  27. BeautifulSoup: Get the contents of a specific table
  28. BeautifulSoup returns None even though the element exists
  29. NLTK download SSL: Certificate verify failed
  30. How to compile python3 on RHEL with SSL? SSL cannot be imported

Leave a Comment