‘UCS-2’ codec can’t encode characters in position 1050-1050

by

Your data contains characters outside of the Basic Multilingual Plane. Emoji’s for example, are outside the BMP, and the window system used by IDLE, Tk, cannot handle such characters.

You could use a translation table to map everything outside of the BMP to the replacement character:

import sys
non_bmp_map = dict.fromkeys(range(0x10000, sys.maxunicode + 1), 0xfffd)
print(x.translate(non_bmp_map))

The non_bmp_map maps all codepoints outside the BMP (any codepoint higher than 0xFFFF, all the way up to the highest Unicode codepoint your Python version can handle) to U+FFFD REPLACEMENT CHARACTER:

>>> print('This works outside IDLE! \U0001F44D')
This works outside IDLE! 👍
>>> print('This works in IDLE too! \U0001F44D'.translate(non_bmp_map))
This works in IDLE too! �

More Related Contents:

  1. Replace non-ASCII characters with a single space
  2. Windows cmd encoding change causes Python crash
  3. Why does Python print unicode characters when the default encoding is ASCII?
  4. Character reading from file in Python
  5. Get a list of all the encodings Python can encode to
  6. How to find out if Python is compiled with UCS-2 or UCS-4?
  7. How do I check if a string is unicode or ascii?
  8. Why doesn’t Python recognize my utf-8 encoded source file?
  9. UnicodeEncodeError: ‘ascii’ codec can’t encode character at special name [duplicate]
  10. Python 3: How to specify stdin encoding
  11. Python ascii utf unicode
  12. Python – Unicode to ASCII conversion
  13. ElementTree and unicode
  14. How to detect string byte encoding?
  15. Convert “\x” escaped string into readable string in python
  16. python encoding utf-8
  17. How does unicodedata.normalize(form, unistr) work?
  18. Python – dealing with mixed-encoding files
  19. UnicodeDecodeError when performing os.walk
  20. Changing default encoding of Python?
  21. Python regex matching Unicode properties
  22. How can I percent-encode URL parameters in Python?
  23. Suppress the u’prefix indicating unicode’ in python strings
  24. Python & MySql: Unicode and Encoding
  25. Python __str__ versus __unicode__
  26. remove unicode emoji using re in python
  27. latin-1 to ascii
  28. Read a base 64 encoded image from memory using OpenCv python library
  29. Can’t open Unicode URL with Python
  30. Python 3 UnicodeDecodeError: ‘charmap’ codec can’t decode byte 0x9d

Leave a Comment