Is a wide character string literal starting with L like L”Hello World” guaranteed to be encoded in Unicode?

by

The L symbol in front of a string literal simply means that each character in the string will be stored as a wchar_t. But this doesn’t necessarily imply Unicode. For example, you could use a wide character string to encode GB 18030, a character set used in China which is similar to Unicode. The C++03 standard doesn’t have anything to say about Unicode, (however C++11 defines Unicode char types and string literals) so it’s up to you to properly represent Unicode strings in C++03.

Regarding string literals, Chapter 2 (Lexical Conventions) of the C++ standard mentions a “basic source character set”, which is basically equivalent to ASCII. So this essentially guarantees that "abc" will be represented as a 3-byte string (not counting the null), and L"abc" will be represented as a 3 * sizeof(wchar_t)-byte string of wide-characters.

The standard also mentions “universal-character-names” which allow you to refer to non-ASCII characters using the \uXXXX hexadecimal notation. These “universal-character-names” usually map directly to Unicode values, but the standard doesn’t guarantee that they have to. However, you can at least guarantee that your string will be represented as a certain sequence of bytes by using universal-character-names. This will guarantee Unicode output provided the runtime environment supports Unicode, has the appropriate fonts installed, etc.

As for string literals in C++03 source files, again there is no guarantee. If you have a Unicode string literal in your code which contains characters outside of the ASCII range, it is up to your compiler to decide how to interpret these characters. If you want to explicitly guarantee that the compiler will “do the right thing”, you’d need to use \uXXXX notation in your string literals.

More Related Contents:

  1. std::wstring VS std::string
  2. UTF8 to/from wide char conversion in STL
  3. WChars, Encodings, Standards and Portability
  4. How to convert wstring into string?
  5. How to print Unicode character in C++?
  6. How well is Unicode supported in C++11?
  7. What is the Windows equivalent for en_US.UTF-8 locale?
  8. What’s “wrong” with C++ wchar_t and wstrings? What are some alternatives to wide characters?
  9. Output Unicode to console Using C++, in Windows
  10. Using Unicode in C++ source code
  11. How do you properly use WideCharToMultiByte
  12. Convert CString to const char*
  13. Unicode Processing in C++
  14. How to convert char* to wchar_t*?
  15. 😃 (and other Unicode characters) in identifiers not allowed by g++
  16. Writing utf16 to file in binary mode
  17. What does “ provide, and where is it documented?
  18. Correctly reading a utf-16 text file into a string without external libraries?
  19. C++ & Boost: encode/decode UTF-8
  20. does (w)ifstream support different encodings
  21. Windows Unicode C++ Stream Output Failure
  22. Should I use wchar_t when using UTF-8?
  23. Convert between string, u16string & u32string
  24. Is wchar_t needed for unicode support?
  25. What does _T stands for in a CString
  26. Specification of source charset encoding in MSVC++, like gcc “-finput-charset=CharSet”
  27. Inheriting and overriding functions of a std::string?
  28. converting narrow string to wide string
  29. Output Unicode strings in Windows console
  30. Why does wide file-stream in C++ narrow written data by default?

Leave a Comment