Error status code 403 even with headers, Python Requests

by

The site is protected by cloudflare which aims to block, among other things, unauthorized data scraping. From What is data scraping?

The process of web scraping is fairly simple, though the
implementation can be complex. Web scraping occurs in 3 steps:

  1. First the piece of code used to pull the information, which we call a scraper bot, sends an HTTP GET request to a specific website.
  2. When the website responds, the scraper parses the HTML document for a specific pattern of data.
  3. Once the data is extracted, it is converted into whatever specific format the scraper bot’s author designed.

You can use urllib instead of requests, it seems to be able to deal with cloudflare

req = urllib.request.Request('https://v2.gcchmc.org/book-appointment/')
req.add_headers('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0')
req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8')
req.add_header('Accept-Language', 'en-US,en;q=0.5')

r = urllib.request.urlopen(req).read().decode('utf-8')
with open("test.html", 'w', encoding="utf-8") as f:
    f.write(r)

More Related Contents:

  1. How to POST JSON data with Python Requests?
  2. How to install packages offline?
  3. How to send a “multipart/form-data” with requests in python?
  4. Sending “User-agent” using Requests library in Python
  5. Using python Requests with javascript pages
  6. How do I disable log messages from the Requests library?
  7. How to send cookies in a post request with the Python Requests library?
  8. How to get Python requests to trust a self signed SSL certificate?
  9. Timeout for python requests.get entire response
  10. How to make python Requests work via socks proxy
  11. Python requests.exceptions.SSLError: EOF occurred in violation of protocol
  12. Passing csrftoken with python Requests
  13. Python requests SSL error – certificate verify failed
  14. Download and save PDF file with Python requests module
  15. Python ‘requests’ library – define specific DNS?
  16. requests: how to disable / bypass proxy
  17. How can I send an xml body using requests library?
  18. Putting a `Cookie` in a `CookieJar`
  19. Requests library: missing SSL handshake certificates file after cx_Freeze
  20. Why doesn’t requests.get() return? What is the default timeout that requests.get() uses?
  21. google search with python requests library
  22. SSL error with Python requests despite up-to-date dependencies
  23. Python Requests with wincertstore
  24. Change the connection pool size for Python’s “requests” module when in Threading
  25. python requests.get always get 404
  26. What is the proper way to make downstream Https requests inside of Uvicorn/FastAPI?
  27. Python requests, how to limit received size, transfer rate, and/or total time?
  28. Difference between using requests.get() and requests.session().get()?
  29. Python 3: using requests does not get the full content of a web page
  30. Python module not found even though “Requirement Already satisfied in Pip”

Leave a Comment