Using Unicode in C++ source code

by

Encoding in C++ is quite a bit complicated. Here is my understanding of it.

Every implementation has to support characters from the basic source character set. These include common characters listed in §2.2/1 (§2.3/1 in C++11). These characters should all fit into one char. In addition implementations have to support a way to name other characters using a way called universal-character-names and look like \uffff or \Uffffffff and can be used to refer to Unicode characters. A subset of them are usable in identifiers (listed in Annex E).

This is all nice, but the mapping from characters in the file, to source characters (used at compile time) is implementation defined. This constitutes the encoding used. Here is what it says literally (C++98 version):

Physical source file characters are
mapped, in an implementation-defined
manner, to the basic source character
set (introducing new-line characters
for end-of-line indicators) if
necessary. Trigraph sequences (2.3)
are replaced by corresponding
single-character internal
representations. Any source file
character not in the basic source
character set (2.2) is replaced by the
universal-character-name that des-
ignates that character. (An
implementation may use any internal
encoding, so long as an actual
extended character encountered in the
source file, and the same extended
character expressed in the source file
as a universal-character-name (i.e.
using the \uXXXX notation), are
handled equivalently.)

For gcc, you can change it using the option -finput-charset=charset. Additionally, you can change the execution character used to represet values at runtime. The proper option for this is -fexec-charset=charset for char (it defaults to utf-8) and -fwide-exec-charset=charset (which defaults to either utf-16 or utf-32 depending on the size of wchar_t).

More Related Contents:

  1. UTF8 to/from wide char conversion in STL
  2. How do you properly use WideCharToMultiByte
  3. does (w)ifstream support different encodings
  4. Specification of source charset encoding in MSVC++, like gcc “-finput-charset=CharSet”
  5. Inheriting and overriding functions of a std::string?
  6. What is the proper declaration of main in C++?
  7. Order of evaluation in C++ function parameters
  8. Unicode encoding for string literals in C++11
  9. How to print Unicode character in C++?
  10. Array placement-new requires unspecified overhead in the buffer?
  11. Is TCHAR still relevant?
  12. Is “argv[0] = name-of-executable” an accepted standard or just a common convention?
  13. What’s the difference between __PRETTY_FUNCTION__, __FUNCTION__, __func__?
  14. How to open an std::fstream (ofstream or ifstream) with a unicode filename?
  15. Output Unicode to console Using C++, in Windows
  16. A most vexing parse error: constructor with no arguments
  17. Why does C++ output negative numbers when using modulo?
  18. C/C++ int[] vs int* (pointers vs. array notation). What is the difference?
  19. Read Unicode UTF-8 file into wstring
  20. Is the order of iterating through std::map known (and guaranteed by the standard)?
  21. What are declarations and declarators and how are their types interpreted by the standard?
  22. Convert a unicode String In C++ To Upper Case
  23. C++ Visual Studio character encoding issues
  24. Is wchar_t needed for unicode support?
  25. What could C/C++ “lose” if they defined a standard ABI?
  26. Is it legal/well-defined C++ to call a non-static method that doesn’t access members through a null pointer?
  27. What does _T stands for in a CString
  28. Does the C++ standard specify anything on the representation of floating point numbers?
  29. Do pointers to string literals remain valid after a function returns?
  30. Why does wide file-stream in C++ narrow written data by default?

Leave a Comment