Warning raised by inserting 4-byte unicode to mysql

by

If MySQL cannot handle UTF-8 codes of 4 bytes or more then you’ll have to filter out all unicode characters over codepoint \U00010000; UTF-8 encodes codepoints below that threshold in 3 bytes or fewer.

You could use a regular expression for that:

>>> import re
>>> highpoints = re.compile(u'[\U00010000-\U0010ffff]')
>>> example = u'Some example text with a sleepy face: \U0001f62a'
>>> highpoints.sub(u'', example)
u'Some example text with a sleepy face: '

Alternatively, you could use the .translate() function with a mapping table that only contains None values:

>>> nohigh = { i: None for i in xrange(0x10000, 0x110000) }
>>> example.translate(nohigh)
u'Some example text with a sleepy face: '

However, creating the translation table will eat a lot of memory and take some time to generate; it is probably not worth your effort as the regular expression approach is more efficient.

This all presumes you are using a UCS-4 compiled python. If your python was compiled with UCS-2 support then you can only use codepoints up to '\U0000ffff' in regular expressions and you’ll never run into this problem in the first place.

I note that as of MySQL 5.5.3 the newly-added utf8mb4 codec does supports the full Unicode range.

More Related Contents:

  1. Take first some character of string but if particular character will come consider that character also
  2. MySQL parameterized queries
  3. PyInstaller, spec file, ImportError: No module named ‘blah’
  4. Extract part of a regex match
  5. What is the difference between re.search and re.match?
  6. How can I do multiple substitutions using regex?
  7. regex error – nothing to repeat
  8. Load CSV data into MySQL in Python
  9. Split string based on regex
  10. Python Mysql, “commands out of sync; you can’t run this command now”
  11. How do I re.search or re.match on a whole file without reading it all into memory?
  12. ImportError: No module named MySQLdb
  13. Making letters uppercase using re.sub in python?
  14. How can I split this comma-delimited string in Python? [duplicate]
  15. Can’t install mysql-python (newer versions) in Windows
  16. Using Django database layer outside of Django?
  17. TypeError: expected string or buffer
  18. How to move a model between two Django apps (Django 1.7)
  19. Empty string instead of unmatched group error
  20. python re.sub, only replace part of match [duplicate]
  21. Python: Elegant way to check if at least one regex in list matches a string
  22. How do I get time from a datetime.timedelta object?
  23. Regex matching – why does this not match and return None?
  24. How to get the first word in the string
  25. Generate a String that matches a RegEx in Python [duplicate]
  26. Print results in MySQL format with Python
  27. Split string at commas except when in bracket environment
  28. Using ^ to match beginning of line in Python regex
  29. How to split long regular expression rules to multiple lines in Python
  30. Python regular expression not matching

Leave a Comment