How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]

by

As justhalf points out above, my question here is essentially a duplicate of this question.

The HTML content reported itself as UTF-8 encoded and, for the most part it was, except for one or two rogue invalid UTF-8 characters.

This apparently confuses BeautifulSoup about which encoding is in use, and when trying to first decode as UTF-8 when passing the content to BeautifulSoup like
this:

soup = BeautifulSoup(response.read().decode('utf-8'))

I would get the error:

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 186812-186813: 
                    invalid continuation byte

Looking more closely at the output, there was an instance of the character Ü which was wrongly encoded as the invalid byte sequence 0xe3 0x9c, rather than the correct 0xc3 0x9c.

As the currently highest-rated answer on that question suggests, the invalid UTF-8 characters can be removed while parsing, so that only valid data is passed to BeautifulSoup:

soup = BeautifulSoup(response.read().decode('utf-8', 'ignore'))

More Related Contents:

  1. How to remove \xa0 from string in Python?
  2. Python and BeautifulSoup encoding issues [duplicate]
  3. UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xa0′ in position 20: ordinal not in range(128)
  4. Saving utf-8 texts with json.dumps as UTF8, not as \u escape sequence
  5. u’\ufeff’ in Python string
  6. Unicode (UTF-8) reading and writing to files in Python
  7. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1
  8. Convert Unicode to ASCII without errors in Python
  9. How to fetch a non-ascii url with urlopen?
  10. Python & MySql: Unicode and Encoding
  11. How to convert a string to utf-8 in Python
  12. How do I check if a string is unicode or ascii?
  13. Writing UTF-8 String to MySQL with Python
  14. General Unicode/UTF-8 support for csv files in Python 2.6
  15. MySQL “incorrect string value” error when save unicode string in Django
  16. urllib2 read to Unicode
  17. Why doesn’t Python recognize my utf-8 encoded source file?
  18. UnicodeEncodeError: ‘ascii’ codec can’t encode character at special name [duplicate]
  19. Python 3 print() function with Farsi/Arabic characters [duplicate]
  20. Extracting text from script tag using BeautifulSoup in Python
  21. Python ascii utf unicode
  22. Python DictWriter writing UTF-8 encoded CSV files
  23. ElementTree and unicode
  24. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  25. Convert UTF-16 to UTF-8 and remove BOM?
  26. Convert “\x” escaped string into readable string in python
  27. python encoding utf-8
  28. Python – dealing with mixed-encoding files
  29. SQLite, python, unicode, and non-utf data
  30. UnicodeDecodeError when performing os.walk

Leave a Comment