A good way to get the charset/encoding of an HTTP response in Python

by

To parse http header you could use cgi.parse_header():

_, params = cgi.parse_header('text/html; charset=utf-8')
print params['charset'] # -> utf-8

Or using the response object:

response = urllib2.urlopen('http://example.com')
response_encoding = response.headers.getparam('charset')
# or in Python 3: response.headers.get_content_charset(default)

In general the server may lie about the encoding or do not report it at all (the default depends on content-type) or the encoding might be specified inside the response body e.g., <meta> element in html documents or in xml declaration for xml documents. As a last resort the encoding could be guessed from the content itself.

You could use requests to get Unicode text:

import requests # pip install requests

r = requests.get(url)
unicode_str = r.text # may use `chardet` to auto-detect encoding

Or BeautifulSoup to parse html (and convert to Unicode as a side-effect):

from bs4 import BeautifulSoup # pip install beautifulsoup4

soup = BeautifulSoup(urllib2.urlopen(url)) # may use `cchardet` for speed
# ...

Or bs4.UnicodeDammit directly for arbitrary content (not necessarily an html):

from bs4 import UnicodeDammit

dammit = UnicodeDammit(b"Sacr\xc3\xa9 bleu!")
print(dammit.unicode_markup)
# -> Sacré bleu!
print(dammit.original_encoding)
# -> utf-8

More Related Contents:

  1. How to download any(!) webpage with correct charset in python?
  2. Import error: No module name urllib2
  3. Working with UTF-8 encoding in Python source [duplicate]
  4. Writing a connection string when password contains special characters
  5. Python: Converting from ISO-8859-1/latin1 to UTF-8
  6. websocket vs rest API for real time data? [closed]
  7. Write to UTF-8 file in Python
  8. Python: download files from google drive using url
  9. Python handling socket.error: [Errno 104] Connection reset by peer
  10. Using MultipartPostHandler to POST form-data with Python
  11. Python-Requests close http connection
  12. python ignore certificate validation urllib2
  13. How can I get all the request headers in Django?
  14. Making a POST call instead of GET using urllib2
  15. How can I speed up fetching pages with urllib2 in python?
  16. Unescape Python Strings From HTTP
  17. python 3 open() default encoding
  18. What encoding does open() use by default?
  19. open url from pythonanywhere
  20. Submitting to a web form using python
  21. urllib2 HTTP Error 400: Bad Request
  22. Limiting number of processes in multiprocessing python
  23. Checking if a website is up via Python
  24. Download and decompress gzipped file in memory?
  25. Ideal method for sending multiple HTTP requests over Python? [duplicate]
  26. Why do I get “‘str’ object has no attribute ‘read'” when trying to use `json.load` on a string? [duplicate]
  27. How to limit download rate of HTTP requests in requests python library?
  28. Multiprocessing useless with urllib2?
  29. Python3 error: initial_value must be str or None, with StringIO
  30. Bytes in a unicode Python string

Leave a Comment