Does python urllib2 automatically uncompress gzip data fetched from webpage?

by
  1. How can I tell if the data at a URL is gzipped?

This checks if the content is gzipped and decompresses it:

from StringIO import StringIO
import gzip

request = urllib2.Request('http://example.com/')
request.add_header('Accept-encoding', 'gzip')
response = urllib2.urlopen(request)
if response.info().get('Content-Encoding') == 'gzip':
    buf = StringIO(response.read())
    f = gzip.GzipFile(fileobj=buf)
    data = f.read()
  1. Does urllib2 automatically uncompress the data if it is gzipped? Will the data always be a string?

No. The urllib2 doesn’t automatically uncompress the data because the ‘Accept-Encoding’ header is not set by the urllib2 but by you using: request.add_header('Accept-Encoding','gzip, deflate')

More Related Contents:

  1. Download and decompress gzipped file in memory?
  2. How to download image using requests
  3. What are the differences between the urllib, urllib2, urllib3 and requests module?
  4. Downloading a picture via urllib and python
  5. How can I use a SOCKS 4/5 proxy with urllib2?
  6. Python: URLError:
  7. A good way to get the charset/encoding of an HTTP response in Python
  8. How do I prevent Python’s urllib(2) from following a redirect
  9. Python URLLib / URLLib2 POST
  10. Python urllib2 with keep alive
  11. python: urllib2 how to send cookie with urlopen request
  12. Python decompression relative performance?
  13. Login to website using urllib2 – Python 2.7
  14. How to unzip gz file using Python
  15. Python urllib over TOR? [duplicate]
  16. Fetch a Wikipedia article with Python
  17. Python Requests: Post JSON and file in single request
  18. What should I do if socket.setdefaulttimeout() is not working?
  19. Multiple (asynchronous) connections with urllib2 or other http library?
  20. Tell urllib2 to use custom DNS
  21. opening websites using urllib2 from behind corporate firewall – 11004 getaddrinfo failed
  22. Source interface with Python and urllib2
  23. Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str
  24. Python POST binary data
  25. How to get the URL of a redirect with Python
  26. urllib2 file name
  27. Using urllib2 with SOCKS proxy
  28. How do I gzip compress a string in Python?
  29. Multiprocessing useless with urllib2?
  30. Python3 error: initial_value must be str or None, with StringIO

Leave a Comment