Unicode encoding for string literals in C++11

by

Are the \x/\u/\U character references freely combinable with all string types?

No. \x can be used in anything, but \u and \U can only be used in strings that are specifically UTF-encoded. However, for any UTF-encoded string, \u and \U can be used as you see fit.

Are all the string types fixed-width, i.e. the arrays contain precisely as many elements as appear in the literal, or to \x/\u/\U references get expanded into a variable number of bytes?

Not in the way you mean. \x, \u, and \U are converted based on the string encoding. The number of those “code units” (using Unicode terms. A char16_t is a UTF-16 code unit) values depends on the encoding of the containing string. The literal u8"\u1024" would create a string containing 2 chars plus a null terminator. The literal u"\u1024" would create a string containing 1 char16_t plus a null terminator.

The number of code units used is based on the Unicode encoding.

Do u”” and u8″” strings have encoding semantics, e.g. can I say char16_t x[] = u”\U0010FFFF”, and the non-BMP codepoint gets encoded into a two-unit UTF16 sequence?

u"" creates a UTF-16 encoded string. u8"" creates a UTF-8 encoded string. They will be encoded per the Unicode specification.

In (1), can I write lone surrogates with \u?

Absolutely not. The specification expressly forbids using the UTF-16 surrogate pairs (0xD800-0xDFFF) as codepoints for \u or \U.

Finally, are any of the string functions encoding aware (i.e. they are character-aware and can detect invalid byte sequences)?

Absolutely not. Well, allow me to rephrase that.

std::basic_string doesn’t deal with Unicode encodings. They certainly can store UTF-encoded strings. But they can only think of them as sequences of char, char16_t, or char32_t; they can’t think of them as a sequence of Unicode codepoints that are encoded with a particular mechanism. basic_string::length() will return the number of code units, not code points. And obviously, the C standard library string functions are totally useless

It should be noted however that “length” for a Unicode string does not mean the number of codepoints. Some code points are combining “characters” (an unfortunate name), which combine with the previous codepoint. So multiple codepoints can map to a single visual character.

Iostreams can in fact read/write Unicode-encoded values. To do so, you will have to use a locale to specify the encoding and properly imbue it into the various places. This is easier said than done, and I don’t have any code on me to show you how.

More Related Contents:

  1. How well is Unicode supported in C++11?
  2. Computing length of a C string at compile time. Is this really a constexpr?
  3. 😃 (and other Unicode characters) in identifiers not allowed by g++
  4. What does “ provide, and where is it documented?
  5. Convert between string, u16string & u32string
  6. Concat two `const char` string literals
  7. Do C++11 regular expressions work with UTF-8 strings?
  8. what is the concept of evaluation of expression z = (y = 30) + (y = 10) + (y = 20); [closed]
  9. Is cout synchronized/thread-safe?
  10. Conveniently Declaring Compile-Time Strings in C++
  11. What is the correct way of using C++11’s range-based for?
  12. How can I efficiently select a Standard Library container in C++11?
  13. Meaning of = delete after function declaration
  14. Delegate Constructor C++
  15. Selecting a member function using different enable_if conditions
  16. How to make generic computations over heterogeneous argument packs of a variadic template function?
  17. When to use std::begin and std::end instead of container specific versions [duplicate]
  18. How does the range-based for work for plain arrays?
  19. How to set the stacksize with C++11 std::thread
  20. In C++11, does `i += ++i + 1` exhibit undefined behavior?
  21. What exactly is an R-Value in C++?
  22. std::unique_ptr, deleters and the Win32 API
  23. Which headers in the C++ standard library are guaranteed to include another header?
  24. Default template argument when using std::enable_if as templ. param.: why OK with two template functions that differ only in the enable_if parameter?
  25. How can I use C++ 11 features in Clang?
  26. The new keyword “auto”; When should it be used to declare a variable type? [duplicate]
  27. Equivalent of “using namespace X” for scoped enumerations?
  28. The implementation of random_device in VS2010?
  29. Why aren’t there compiler-generated swap() methods in C++0x?
  30. Why must an enumeration’s size be provided when it is forward declared?

Leave a Comment