does (w)ifstream support different encodings

by

C++ supports character encodings by means of std::locale and the facet std::codecvt. The general idea is that a locale object describes the aspects of the system that might vary from culture to culture, (human) language to language. These aspects are broken down into facets, which are template arguments that define how localization-dependent objects (include I/O streams) are constructed. When you read from an istream or write to a ostream, the actual writing of each character is filtered through the locale’s facets. The facets cover not only encoding of Unicode types but such varied features as how large numbers are written (e.g. with commas or periods), currency, time, capitalization, and a slew of other details.

However just because the facilities exist to do encodings doesn’t mean the standard library actually handles all encodings, nor does it make such code simple to do right. Even such basic things as the size of character you should be reading into (let alone the encoding part) is difficult, as wchar_t can be too small (mangling your data), or too large (wasting space), and the most common compilers (e.g. Visual C++ and Gnu C++) do differ on how big their implementation is. So you generally need to find external libraries to do the actual encoding.

  • iconv is generally acknowledge to be correct, but examples of how to bind it to the C++ mechanism are hard to find.
  • jla3ep mentions libICU, which is very thorough but the C++ API does not try to play nicely with the standard (As far as I can tell: you can scan the examples to see if you can do better.)

The most straightforward example I can find that covers all the bases, is from Boost’s UTF-8 codecvt facet, with an example that specifically tries to encode UTF-8 (UCS4) for use by IO streams. It looks like this, though I don’t suggest just copying it verbatim. It takes a little more digging in the source to understand it (and I don’t claim to):

typedef wchar_t ucs4_t;

std::locale old_locale;
std::locale utf8_locale(old_locale,new utf8_codecvt_facet<ucs4_t>);

...

std::wifstream input_file("data.utf8");
input_file.imbue(utf8_locale);
ucs4_t item = 0;
while (ifs >> item) { ... }

To understand more about locales, and how they use facets (including codecvt), take a look at the following:

More Related Contents:

  1. UTF8 to/from wide char conversion in STL
  2. How to convert wstring into string?
  3. Using Unicode in C++ source code
  4. How do you properly use WideCharToMultiByte
  5. Windows Unicode C++ Stream Output Failure
  6. Specification of source charset encoding in MSVC++, like gcc “-finput-charset=CharSet”
  7. Inheriting and overriding functions of a std::string?
  8. What’s the difference between “STL” and “C++ Standard Library”?
  9. Is std::vector so much slower than plain arrays?
  10. Why use iterators instead of array indices?
  11. How do I erase an element from std::vector by index?
  12. What requirements must std::map key classes meet to be valid keys?
  13. How to get std::vector pointer to the raw data?
  14. Is list::size() really O(n)?
  15. Heterogeneous containers in C++
  16. Unicode Processing in C++
  17. Why use functors over functions?
  18. How to iterate over a priority_queue?
  19. How do I store arrays in an STL list?
  20. Performance of qsort vs std::sort?
  21. How does C++ STL unordered_map resolve collisions?
  22. How do I erase elements from STL containers?
  23. Are const_iterators faster?
  24. STL analogue in Fortran
  25. Why the libc++ std::vector internally keeps three pointers instead of one pointer and two sizes?
  26. Visual c++ “for each” portability
  27. Is stl vector concurrent read thread-safe?
  28. Why can’t I replace std::map with std::unordered_map
  29. error for hash function of pair of ints
  30. Why does (i|o)fstream take a const char* parameter for a file name?

Leave a Comment