Ensuring valid UTF-8 in PHP

by

UTF-8 can store any Unicode character. If your encoding is anything else at all, including ISO-8859-1 or Windows-1252, UTF-8 can store every character in it. So you don’t have to worry about losing any characters when you convert a string from any other encoding to UTF-8.

Further, both ISO-8859-1 and Windows-1252 are single-byte encodings where any byte is valid. It is not technically possible to distinguish between them. I would chose Windows-1252 as your default match for non-UTF-8 sequences, as the only bytes that decode differently are the range 0x80-0x9F. These decode to various characters like smart quotes and the Euro in Windows-1252, whereas in ISO-8859-1 they are invisible control characters which are almost never used. Web browsers may sometimes say they are using ISO-8859-1, but often they will really be using Windows-1252.

would this code ensure that a string is safe to insert into a UTF-8 encoded document

You would certainly want to set the optional ‘strict’ parameter to TRUE for this purpose. But I’m not sure this actually covers all invalid UTF-8 sequences. The function does not claim to check a byte sequence for UTF-8 validity explicitly. There have been known cases where mb_detect_encoding would guess UTF-8 incorrectly before, though I don’t know if that can still happen in strict mode.

If you want to be sure, do it yourself using the W3-recommended regex:

if (preg_match('%^(?:
      [\x09\x0A\x0D\x20-\x7E]            # ASCII
    | [\xC2-\xDF][\x80-\xBF]             # non-overlong 2-byte
    | \xE0[\xA0-\xBF][\x80-\xBF]         # excluding overlongs
    | [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}  # straight 3-byte
    | \xED[\x80-\x9F][\x80-\xBF]         # excluding surrogates
    | \xF0[\x90-\xBF][\x80-\xBF]{2}      # planes 1-3
    | [\xF1-\xF3][\x80-\xBF]{3}          # planes 4-15
    | \xF4[\x80-\x8F][\x80-\xBF]{2}      # plane 16
)*$%xs', $string))
    return $string;
else
    return iconv('CP1252', 'UTF-8', $string);

More Related Contents:

  1. Detect encoding and make everything UTF-8
  2. How can I write a file in UTF-8 format?
  3. Convert utf8-characters to iso-88591 and back in PHP
  4. PHP json encode – Malformed UTF-8 characters, possibly incorrectly encoded [duplicate]
  5. How to handle user input of invalid UTF-8 characters
  6. json_encode() non utf-8 strings?
  7. How can I detect a malformed UTF-8 string in PHP?
  8. PHP messing with HTML Charset Encoding
  9. utf-8 special characters not displaying [duplicate]
  10. UTF8 Filenames in PHP and Different Unicode Encodings
  11. How do I detect if have to apply UTF-8 decode or encode on a string?
  12. Reference: Why are my “special” Unicode characters encoded weird using json_encode?
  13. How can I get a hex dump of a string in PHP?
  14. PHP DOMDocument loadHTML not encoding UTF-8 correctly
  15. SET NAMES utf8 in MySQL?
  16. file_get_contents() Breaks Up UTF-8 Characters
  17. PHP output showing little black diamonds with a question mark
  18. How to best configure PHP to handle a UTF-8 website [duplicate]
  19. Decode gzipped web page retrieved via cURL in PHP
  20. Error: “Input is not proper UTF-8, indicate encoding !” using PHP’s simplexml_load_string
  21. How to sort an array of UTF-8 strings?
  22. Regex to detect invalid UTF-8 string
  23. php header excel and utf-8
  24. Fix incorrectly displayed encoding on an html document with php
  25. UTF-8 encoding for subject in contact form email
  26. How to iterate UTF-8 string in PHP?
  27. URL Decoding in PHP
  28. php substr() function with utf-8 leaves � marks at the end
  29. How to check if letter is upper or lower in PHP?
  30. Is a BLOB converted using the current/default charset in MySQL?

Leave a Comment