Regex to detect invalid UTF-8 string

by

You can use this PCRE regular expression to check for a valid UTF-8 in a string. If the regex matches, the string contains invalid byte sequences. It’s 100% portable because it doesn’t rely on PCRE_UTF8 to be compiled in.

$regex = '/(
    [\xC0-\xC1] # Invalid UTF-8 Bytes
    | [\xF5-\xFF] # Invalid UTF-8 Bytes
    | \xE0[\x80-\x9F] # Overlong encoding of prior code point
    | \xF0[\x80-\x8F] # Overlong encoding of prior code point
    | [\xC2-\xDF](?![\x80-\xBF]) # Invalid UTF-8 Sequence Start
    | [\xE0-\xEF](?![\x80-\xBF]{2}) # Invalid UTF-8 Sequence Start
    | [\xF0-\xF4](?![\x80-\xBF]{3}) # Invalid UTF-8 Sequence Start
    | (?<=[\x00-\x7F\xF5-\xFF])[\x80-\xBF] # Invalid UTF-8 Sequence Middle
    | (?<![\xC2-\xDF]|[\xE0-\xEF]|[\xE0-\xEF][\x80-\xBF]|[\xF0-\xF4]|[\xF0-\xF4][\x80-\xBF]|[\xF0-\xF4][\x80-\xBF]{2})[\x80-\xBF] # Overlong Sequence
    | (?<=[\xE0-\xEF])[\x80-\xBF](?![\x80-\xBF]) # Short 3 byte sequence
    | (?<=[\xF0-\xF4])[\x80-\xBF](?![\x80-\xBF]{2}) # Short 4 byte sequence
    | (?<=[\xF0-\xF4][\x80-\xBF])[\x80-\xBF](?![\x80-\xBF]) # Short 4 byte sequence (2)
)/x';

We can test it by creating a few variations of text:

// Overlong encoding of code point 0
$text = chr(0xC0) . chr(0x80);
var_dump(preg_match($regex, $text)); // int(1)
// Overlong encoding of 5 byte encoding
$text = chr(0xF8) . chr(0x80) . chr(0x80) . chr(0x80) . chr(0x80);
var_dump(preg_match($regex, $text)); // int(1)
// Overlong encoding of 6 byte encoding
$text = chr(0xFC) . chr(0x80) . chr(0x80) . chr(0x80) . chr(0x80) . chr(0x80);        
var_dump(preg_match($regex, $text)); // int(1)
// High code-point without trailing characters
$text = chr(0xD0) . chr(0x01);
var_dump(preg_match($regex, $text)); // int(1)

etc…

In fact, since this matches invalid bytes, you could then use it in preg_replace to replace them away:

preg_replace($regex, '', $text); // Remove all invalid UTF-8 code-points

More Related Contents:

  1. Regex For a Simple Validation [closed]
  2. PHP validation/regex for URL
  3. How to validate an Email in PHP?
  4. How to “Validate” Human Names in CakePHP?
  5. PHP regular expression for strong password validation [duplicate]
  6. Validating Crontab Entries with PHP
  7. Validating US phone number with php/regex
  8. php regex word boundary matching in utf-8
  9. Regex for password PHP [duplicate]
  10. PHP regex for validating a URL
  11. UTF-8 in PHP regular expressions [duplicate]
  12. How to validate that a string only contain lowercase letters?
  13. Validate date format in php
  14. deny direct access to a folder and file by htaccess
  15. PHP: Replace umlauts with closest 7-bit ASCII equivalent in an UTF-8 string
  16. Convert plain text URLs into HTML hyperlinks in PHP
  17. Convert latin1 characters on a UTF8 table into UTF8
  18. PHP DOMDocument failing to handle utf-8 characters (☆)
  19. “vertical” regex matching in an ASCII “image”
  20. Regular expression pattern to match URL with or without http://www
  21. Easiest Form validation library for PHP? [closed]
  22. Parsing CSS by regex
  23. get_headers Inconsistency [closed]
  24. Converting Javascript Regex to PHP
  25. Extract floating point numbers from a delimited string in PHP
  26. Best way to check for positive integer (PHP)?
  27. How to return custom error message from controller method validation
  28. Regular expression preg_quote symbols are not detected
  29. Trim unicode whitespace in PHP
  30. Detecting a url using preg_match? without http:// in the string

Leave a Comment