How to filter (or replace) unicode characters that would take more than 3 bytes in UTF-8?

by

Unicode characters in the ranges \u0000-\uD7FF and \uE000-\uFFFF will have 3 byte (or less) encodings in UTF8. The \uD800-\uDFFF range is for multibyte UTF16. I do not know python, but you should be able to set up a regular expression to match outside those ranges.

pattern = re.compile("[\uD800-\uDFFF].", re.UNICODE)
pattern = re.compile("[^\u0000-\uFFFF]", re.UNICODE)

Edit adding Python from Denilson Sá’s script in the question body:

re_pattern = re.compile(u'[^\u0000-\uD7FF\uE000-\uFFFF]', re.UNICODE)
filtered_string = re_pattern.sub(u'\uFFFD', unicode_string)

More Related Contents:

  1. MySQL “incorrect string value” error when save unicode string in Django
  2. Python & MySql: Unicode and Encoding
  3. Mac OS X – EnvironmentError: mysql_config not found
  4. Setting Django up to use MySQL
  5. UnicodeEncodeError: ‘latin-1’ codec can’t encode character
  6. How to replace Django’s primary key with a different integer that is unique for that table
  7. Install mysql-python (Windows)
  8. How to use MySQLdb with Python and Django in OSX 10.6?
  9. Django : Table doesn’t exist
  10. Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock
  11. Error loading MySQLdb Module ‘Did you install mysqlclient or MySQL-python?’
  12. How to make Django slugify work properly with Unicode strings?
  13. How to run Django’s test database only in memory?
  14. How can I check if a Python unicode string contains non-Western letters?
  15. MySQL Improperly Configured Reason: unsafe use of relative path
  16. Using Django database layer outside of Django?
  17. How to move a model between two Django apps (Django 1.7)
  18. Can one use the Django database layer outside of Django?
  19. InterfaceError (0, ”)
  20. How do i cast char to integer while querying in django ORM?
  21. How to make an Inner Join in django?
  22. Simple Subquery with OuterRef
  23. Using .aggregate() on a value introduced using .extra(select={…}) in a Django Query?
  24. django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named MySQLdb
  25. Install mysqlclient for Django Python on Mac OS X Sierra
  26. Remove non-ASCII characters from a string using python / django
  27. Unable to encode/decode pprint output
  28. Django + MySQL on Mac OS 10.6.2 Snow Leopard
  29. Single Django model, multiple tables?
  30. What is PyMySQL and how does it differ from MySQLdb? Can it affect Django deployment?

Leave a Comment