How to download any(!) webpage with correct charset in python?

by

When you download a file with urllib or urllib2, you can find out whether a charset header was transmitted:

fp = urllib2.urlopen(request)
charset = fp.headers.getparam('charset')

You can use BeautifulSoup to locate a meta element in the HTML:

soup = BeatifulSoup.BeautifulSoup(data)
meta = soup.findAll('meta', {'http-equiv':lambda v:v.lower()=='content-type'})

If neither is available, browsers typically fall back to user configuration, combined with auto-detection. As rajax proposes, you could use the chardet module. If you have user configuration available telling you that the page should be Chinese (say), you may be able to do better.

More Related Contents:

  1. What are the differences between the urllib, urllib2, urllib3 and requests module?
  2. Downloading a picture via urllib and python
  3. Python: URLError:
  4. A good way to get the charset/encoding of an HTTP response in Python
  5. Python URLLib / URLLib2 POST
  6. Handling urllib2’s timeout? – Python
  7. How to unquote a urlencoded unicode string in python?
  8. Making a POST call instead of GET using urllib2
  9. What should I do if socket.setdefaulttimeout() is not working?
  10. How do I catch a specific HTTP error in Python?
  11. Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str
  12. Submitting to a web form using python
  13. How to save “complete webpage” not just basic html using Python
  14. How to resolve URLError:
  15. Python3 error: initial_value must be str or None, with StringIO
  16. urllib and “SSL: CERTIFICATE_VERIFY_FAILED” Error
  17. Import error: No module name urllib2
  18. Working with UTF-8 encoding in Python source [duplicate]
  19. Writing a connection string when password contains special characters
  20. Get a list of all the encodings Python can encode to
  21. What is the difference between encode/decode?
  22. urllib2 read to Unicode
  23. Python urllib2.urlopen() is slow, need a better way to read several urls
  24. python 3 open() default encoding
  25. Changing User Agent in Python 3 for urrlib.request.urlopen
  26. open url from pythonanywhere
  27. Limiting number of processes in multiprocessing python
  28. no module named urllib.parse (How should I install it?)
  29. Python 3 – urllib, HTTP Error 407: Proxy Authentication Required
  30. urllib2 and json

Leave a Comment