Python unicode equal comparison failed

by

You may use the == operator to compare unicode objects for equality.

>>> s1 = u'Hello'
>>> s2 = unicode("Hello")
>>> type(s1), type(s2)
(<type 'unicode'>, <type 'unicode'>)
>>> s1==s2
True
>>> 
>>> s3='Hello'.decode('utf-8')
>>> type(s3)
<type 'unicode'>
>>> s1==s3
True
>>>

But, your error message indicates that you aren’t comparing unicode objects. You are probably comparing a unicode object to a str object, like so:

>>> u'Hello' == 'Hello'
True
>>> u'Hello' == '\x81\x01'
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
False

See how I have attempted to compare a unicode object against a string which does not represent a valid UTF8 encoding.

Your program, I suppose, is comparing unicode objects with str objects, and the contents of a str object is not a valid UTF8 encoding. This seems likely the result of you (the programmer) not knowing which variable holds unicide, which variable holds UTF8 and which variable holds the bytes read in from a file.

I recommend http://nedbatchelder.com/text/unipain.html, especially the advice to create a “Unicode Sandwich.”

More Related Contents:

  1. What exactly do “u” and “r” string flags do, and what are raw string literals?
  2. UnicodeDecodeError: ‘charmap’ codec can’t decode byte X in position Y: character maps to
  3. How to remove \xa0 from string in Python?
  4. How to get string objects instead of Unicode from JSON?
  5. Python string prints as [u’String’]
  6. Read and Write CSV files including unicode with Python 2.7
  7. Remove punctuation from Unicode formatted strings
  8. Why does Python print unicode characters when the default encoding is ASCII?
  9. text with unicode escape sequences to unicode in python [duplicate]
  10. Get a list of all the encodings Python can encode to
  11. What is the difference between encode/decode?
  12. Conversion of strings like \\uXXXX in python
  13. How to read Unicode input and compare Unicode strings in Python?
  14. Python 3 print() function with Farsi/Arabic characters [duplicate]
  15. Python – Unicode to ASCII conversion
  16. UnicodeDecodeError: ‘utf8’ codec can’t decode bytes in position 3-6: invalid data
  17. What does ‘u’ mean in a list?
  18. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  19. How is unicode represented internally in Python?
  20. Unicode identifiers in Python?
  21. Printing a string prints ‘u’ before the string in Python?
  22. Matching only a unicode letter in Python re
  23. How does unicodedata.normalize(form, unistr) work?
  24. Python – dealing with mixed-encoding files
  25. Python the same char not equals
  26. string.translate() with unicode data in python
  27. Unicode encoding for filesystem in Mac OS X not correct in Python?
  28. Chinese unicode fonts in PyGame
  29. Printing Unicode in eclipse Pydev console and in Idle
  30. UnicodeDecodeError when reading CSV file in Pandas

Leave a Comment