How do I properly use std::string on UTF-8 in C++?

by

Unicode Glossary

Unicode is a vast and complex topic. I do not wish to wade too deep there, however a quick glossary is necessary:

  1. Code Points: Code Points are the basic building blocks of Unicode, a code point is just an integer mapped to a meaning. The integer portion fits into 32 bits (well, 24 bits really), and the meaning can be a letter, a diacritic, a white space, a sign, a smiley, half a flag, … and it can even be “the next portion reads right to left”.
  2. Grapheme Clusters: Grapheme Clusters are groups of semantically related Code Points, for example a flag in unicode is represented by associating two Code Points; each of those two, in isolation, has no meaning, but associated together in a Grapheme Cluster they represent a flag. Grapheme Clusters are also used to pair a letter with a diacritic in some scripts.

This is the basic of Unicode. The distinction between Code Point and Grapheme Cluster can be mostly glossed over because for most modern languages each “character” is mapped to a single Code Point (there are dedicated accented forms for commonly used letter+diacritic combinations). Still, if you venture in smileys, flags, etc… then you may have to pay attention to the distinction.

UTF Primer

Then, a serie of Unicode Code Points has to be encoded; the common encodings are UTF-8, UTF-16 and UTF-32, the latter two existing in both Little-Endian and Big-Endian forms, for a total of 5 common encodings.

In UTF-X, X is the size in bits of the Code Unit, each Code Point is represented as one or several Code Units, depending on its magnitude:

  • UTF-8: 1 to 4 Code Units,
  • UTF-16: 1 or 2 Code Units,
  • UTF-32: 1 Code Unit.

std::string and std::wstring.

  1. Do not use std::wstring if you care about portability (wchar_t is only 16 bits on Windows); use std::u32string instead (aka std::basic_string<char32_t>).
  2. The in-memory representation (std::string or std::wstring) is independent of the on-disk representation (UTF-8, UTF-16 or UTF-32), so prepare yourself for having to convert at the boundary (reading and writing).
  3. While a 32-bits wchar_t ensures that a Code Unit represents a full Code Point, it still does not represent a complete Grapheme Cluster.

If you are only reading or composing strings, you should have no to little issues with std::string or std::wstring.

Troubles start when you start slicing and dicing, then you have to pay attention to (1) Code Point boundaries (in UTF-8 or UTF-16) and (2) Grapheme Clusters boundaries. The former can be handled easily enough on your own, the latter requires using a Unicode aware library.

Picking std::string or std::u32string?

If performance is a concern, it is likely that std::string will perform better due to its smaller memory footprint; though heavy use of Chinese may change the deal. As always, profile.

If Grapheme Clusters are not a problem, then std::u32string has the advantage of simplifying things: 1 Code Unit -> 1 Code Point means that you cannot accidentally split Code Points, and all the functions of std::basic_string work out of the box.

If you interface with software taking std::string or char*/char const*, then stick to std::string to avoid back-and-forth conversions. It’ll be a pain otherwise.

UTF-8 in std::string.

UTF-8 actually works quite well in std::string.

Most operations work out of the box because the UTF-8 encoding is self-synchronizing and backward compatible with ASCII.

Due the way Code Points are encoded, looking for a Code Point cannot accidentally match the middle of another Code Point:

  • str.find('\n') works,
  • str.find("...") works for matching byte by byte1,
  • str.find_first_of("\r\n") works if searching for ASCII characters.

Similarly, regex should mostly works out of the box. As a sequence of characters ("haha") is just a sequence of bytes ("哈"), basic search patterns should work out of the box.

Be wary, however, of character classes (such as [:alphanum:]), as depending on the regex flavor and implementation it may or may not match Unicode characters.

Similarly, be wary of applying repeaters to non-ASCII “characters”, "哈?" may only consider the last byte to be optional; use parentheses to clearly delineate the repeated sequence of bytes in such cases: "(哈)?".

1 The key concepts to look-up are normalization and collation; this affects all comparison operations. std::string will always compare (and thus sort) byte by byte, without regard for comparison rules specific to a language or a usage. If you need to handle full normalization/collation, you need a complete Unicode library, such as ICU.

More Related Contents:

  1. Legality of COW std::string implementation in C++11
  2. Converting std::__cxx11::string to std::string
  3. Conveniently Declaring Compile-Time Strings in C++
  4. Will std::string always be null-terminated in C++11?
  5. Why is conversion from string constant to ‘char*’ valid in C but invalid in C++
  6. How to return a std::string.c_str()
  7. Is string::c_str() no longer null terminated in C++11?
  8. Split a string using C++11
  9. Set precision of std::to_string when converting floating point values [duplicate]
  10. Why is modifying a string through a retrieved pointer to its data not allowed?
  11. C++: Can a macro expand “abc” into ‘a’, ‘b’, ‘c’?
  12. Convert between string, u16string & u32string
  13. Is writing to &str[0] buffer (of a std:string) well-defined behaviour in C++11?
  14. “to_string” isn’t a member of “std”?
  15. C++ Passing std::string by reference to function in dll
  16. Concatenate compile-time strings in a template at compile time?
  17. Sizeof string literal
  18. What is a subnormal floating point number?
  19. Pointers in c++ after delete
  20. Fast String Hashing Algorithm with low collision rates with 32 bit integer [closed]
  21. why is `std::initializer_list` often passed by value?
  22. Difference between std::result_of and decltype
  23. How to enable experimental C++11 concurrency features in MinGW?
  24. Is int arr[ ] valid C++?
  25. Need iterator when using ranged-based for loops
  26. variable or field declared void
  27. Uniform initialization fails to copy when object has no data members
  28. std::this_thread::sleep_for() and GCC
  29. Why is reinterpret_cast not constexpr?
  30. How to initialize all tuple elements by the same arguments?

Leave a Comment