Get html using Python requests?

by

The server in question is giving you a gzipped response. The server is also very broken; it sends the following headers:

$ curl -D - -o /dev/null -s -H 'Accept-Encoding: gzip, deflate' http://www.wrcc.dri.edu/WRCCWrappers.py?sodxtrmts+028815+por+por+pcpn+none+mave+5+01+F
HTTP/1.1 200 OK
Date: Tue, 06 Jan 2015 17:46:49 GMT
Server: Apache
<!DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd"><html xmlns="http: //www.w3.org/1999/xhtml" lang="en-US">
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 3659
Content-Type: text/html

The <!DOCTYPE..> line there is not a valid HTTP header. As such, the remaining headers past Server are ignored. Why the server interjects that is unclear; in all likely hood WRCCWrappers.py is a CGI script that doesn’t output headers but does include a double newline after the doctype line, duping the Apache server into inserting additional headers there.

As such, requests also doesn’t detect that the data is gzip-encoded. The data is all there, you just have to decode it. Or you could if it wasn’t rather incomplete.

The work-around is to tell the server not to bother with compression:

headers = {'Accept-Encoding': 'identity'}
r = requests.get(url, headers=headers)

and an uncompressed response is returned.

Incidentally, on Python 2 the HTTP header parser is not so strict and manages to declare the doctype a header:

>>> pprint(dict(r.headers))
{'<!doctype html public "-//w3c//dtd xhtml 1.0 transitional//en" "dtd/xhtml1-transitional.dtd"><html xmlns="http': '//www.w3.org/1999/xhtml" lang="en-US">',
 'connection': 'Keep-Alive',
 'content-encoding': 'gzip',
 'content-length': '3659',
 'content-type': 'text/html',
 'date': 'Tue, 06 Jan 2015 17:42:06 GMT',
 'keep-alive': 'timeout=5, max=100',
 'server': 'Apache',
 'vary': 'Accept-Encoding'}

and the content-encoding information survives, so there requests decodes the content for you, as expected.

More Related Contents:

  1. How to load all entries in an infinite scroll at once to parse the HTML in python
  2. Using Find_All function returns an unexpected result set
  3. Correct way to try/except using Python requests module?
  4. Max retries exceeded with URL in requests
  5. Python requests. 403 Forbidden
  6. How do I disable the security certificate check in Python requests
  7. Render HTML to PDF in Django site
  8. Checking if element exists with Python Selenium
  9. Python-Requests close http connection
  10. python requests.get() returns improperly decoded text instead of UTF-8?
  11. Parse HTML table to Python list?
  12. Error when loading cookies into a Python request session
  13. Difference between data and json parameters in python requests package
  14. Python check if website exists
  15. Convert HTML entities to Unicode and vice versa
  16. Is there a built in package to parse html into dom?
  17. Python – Requests, Selenium – passing cookies while logging in
  18. Passing a matplotlib figure to HTML (flask)
  19. How to get response SSL certificate from requests in python?
  20. Why is variable1 += variable2 much faster than variable1 = variable1 + variable2?
  21. How can I use cookies in Python Requests?
  22. Can bs4 get the dynamic content of a webpage if requests can’t?
  23. Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”
  24. Error status code 403 even with headers, Python Requests
  25. python requests module and connection reuse
  26. How to Pretty Print HTML to a file, with indentation
  27. Flask css not updating [closed]
  28. HTML to IMAGE using Python
  29. How to store an image in a variable
  30. Python 3: using requests does not get the full content of a web page

Leave a Comment