How can I convert surrogate pairs to normal string in Python?

by

You’ve mixed a literal string \ud83d in a json file on disk (six characters: \ u d 8 3 d) and a single character u'\ud83d' (specified using a string literal in Python source code) in memory. It is the difference between len(r'\ud83d') == 6 and len('\ud83d') == 1 on Python 3.

If you see '\ud83d\ude4f' Python string (2 characters) then there is a bug upstream. Normally, you shouldn’t get such string. If you get one and you can’t fix upstream that generates it; you could fix it using surrogatepass error handler:

>>> "\ud83d\ude4f".encode('utf-16', 'surrogatepass').decode('utf-16')
'🙏'

Python 2 was more permissive.

Note: even if your json file contains literal \ud83d\ude4f (12 characters); you shouldn’t get the surrogate pair:

>>> print(ascii(json.loads(r'"\ud83d\ude4f"')))
'\U0001f64f'

Notice: the result is 1 character ( '\U0001f64f'), not the surrogate pair ('\ud83d\ude4f').

More Related Contents:

  1. What is the best way to remove accents (normalize) in a Python unicode string?
  2. UnicodeDecodeError: ‘charmap’ codec can’t decode byte X in position Y: character maps to
  3. How to set sys.stdout encoding in Python 3?
  4. “Unicode Error “unicodeescape” codec can’t decode bytes… Cannot open text files in Python 3 [duplicate]
  5. How to correct TypeError: Unicode-objects must be encoded before hashing?
  6. Normalizing Unicode
  7. Facebook JSON badly encoded
  8. NameError: global name ‘unicode’ is not defined – in Python 3
  9. Why doesn’t Python recognize my utf-8 encoded source file?
  10. Python 3 print() function with Farsi/Arabic characters [duplicate]
  11. How to make unicode string with python3
  12. Python 3: How to specify stdin encoding
  13. Identifier normalization: Why is the micro sign converted into the Greek letter mu?
  14. requests.get returns 403 while the same url works in browser
  15. How to encode Python 3 string using \u escape code?
  16. What is internal representation of string in Python 3.x
  17. Set encoding in Python 3 CGI scripts
  18. How to deal with unicode string in URL in python3?
  19. Python 3 chokes on CP-1252/ANSI reading
  20. How to make print() output UTF-8 in Python 3.0?
  21. Python script to ping linux server
  22. SyntaxError: Non-ASCII character ‘\xa3’ in file when function returns ‘£’
  23. except-clause deletes local variable
  24. Is it possible to make the keyboard module work with pygame and threading
  25. Meaning of end=” in the statement print(“\t”,end=”)? [duplicate]
  26. Sending a password over SSH or SCP with subprocess.Popen
  27. How to Open a file through python
  28. pandas.io.json.json_normalize with very nested json
  29. multiprocessing vs multithreading vs asyncio
  30. How to expire Django session in 5minutes?

Leave a Comment