python requests.get() returns improperly decoded text instead of UTF-8?

by

Educated guesses (mentioned above) are probably just a check for Content-Type header as being sent by server (quite misleading use of educated imho).

For response header Content-Type: text/html the result is ISO-8859-1 (default for HTML4), regardless any content analysis (ie. default for HTML5 is UTF-8).

For response header Content-Type: text/html; charset=utf-8 the result is UTF-8.

Luckily for us, requests uses chardet library and that usually works quite well (attribute requests.Response.apparent_encoding), so you usually want to do:

r = requests.get("https://martin.slouf.name/")
# override encoding by real educated guess as provided by chardet
r.encoding = r.apparent_encoding
# access the data
r.text

More Related Contents:

  1. How to POST JSON data with Python Requests?
  2. How to install packages offline?
  3. How to send a “multipart/form-data” with requests in python?
  4. Changing default encoding of Python?
  5. Sending “User-agent” using Requests library in Python
  6. Using python Requests with javascript pages
  7. Convert Unicode to ASCII without errors in Python
  8. How do I disable log messages from the Requests library?
  9. Python & MySql: Unicode and Encoding
  10. How to send cookies in a post request with the Python Requests library?
  11. Timeout for python requests.get entire response
  12. Writing UTF-8 String to MySQL with Python
  13. Passing csrftoken with python Requests
  14. Python requests SSL error – certificate verify failed
  15. How to receive json data using HTTP POST request in Django 1.6?
  16. Why doesn’t requests.get() return? What is the default timeout that requests.get() uses?
  17. Python ascii utf unicode
  18. Python Requests with wincertstore
  19. Python requests speed up using keep-alive
  20. Change the connection pool size for Python’s “requests” module when in Threading
  21. How to send utf-8 e-mail?
  22. Python requests, how to limit received size, transfer rate, and/or total time?
  23. UnicodeDecodeError in Python when reading a file, how to ignore the error and jump to the next line?
  24. Saving response from Requests to file
  25. SQLite, python, unicode, and non-utf data
  26. Python Requests – SSL error for client side cert
  27. Getting HEAD content with Python Requests
  28. XHR request URL says does not exist when attempting to parse it’s content
  29. Return a requests.Response object from Flask
  30. How to measure download speed and progress using requests?

Leave a Comment