Python and BeautifulSoup encoding issues [duplicate]

by

In your case this page has wrong utf-8 data which confuses BeautifulSoup and makes it think that your page uses windows-1252, you can do this trick:

soup = BeautifulSoup.BeautifulSoup(content.decode('utf-8','ignore'))

by doing this you will discard any wrong symbols from the page source and BeautifulSoup will guess the encoding correctly.

You can replace ‘ignore’ by ‘replace’ and check text for ‘?’ symbols to see what has been discarded.

Actually it’s a very hard task to write crawler which can guess page encoding every time with 100% chance(Browsers are very good at this nowadays), you can use modules like ‘chardet’ but, for example, in your case it will guess encoding as ISO-8859-2, which is not correct too.

If you really need to be able to get encoding for any page user can possibly supply – you should either build a multi-level(try utf-8, try latin1, try etc…) detection function(like we did in our project) or use some detection code from firefox or chromium as C module.

More Related Contents:

  1. How to remove \xa0 from string in Python?
  2. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  3. UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xa0′ in position 20: ordinal not in range(128)
  4. Saving utf-8 texts with json.dumps as UTF8, not as \u escape sequence
  5. u’\ufeff’ in Python string
  6. Unicode (UTF-8) reading and writing to files in Python
  7. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1
  8. Convert Unicode to ASCII without errors in Python
  9. Python & MySql: Unicode and Encoding
  10. How to convert a string to utf-8 in Python
  11. How do I check if a string is unicode or ascii?
  12. Writing UTF-8 String to MySQL with Python
  13. General Unicode/UTF-8 support for csv files in Python 2.6
  14. MySQL “incorrect string value” error when save unicode string in Django
  15. Why doesn’t Python recognize my utf-8 encoded source file?
  16. UnicodeEncodeError: ‘ascii’ codec can’t encode character at special name [duplicate]
  17. Python 3 print() function with Farsi/Arabic characters [duplicate]
  18. Python ascii utf unicode
  19. Python DictWriter writing UTF-8 encoded CSV files
  20. ElementTree and unicode
  21. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  22. Convert UTF-16 to UTF-8 and remove BOM?
  23. Convert “\x” escaped string into readable string in python
  24. python encoding utf-8
  25. Python – dealing with mixed-encoding files
  26. SQLite, python, unicode, and non-utf data
  27. UnicodeDecodeError when performing os.walk
  28. Bytes in a unicode Python string
  29. Changing default encoding of Python?
  30. BeautifulSoup: just get inside of a tag, no matter how many enclosing tags there are

Leave a Comment