Difference between MBCS and UTF-8 on Windows

by

I noticed that there are two compiler
flags in Visual Studio compiler (for
C++) called MBCS and UNICODE. What is
the difference between them ?

Many functions in the Windows API come in two versions: One that takes char parameters (in a locale-specific code page) and one that takes wchar_t parameters (in UTF-16).

int MessageBoxA(HWND hWnd, const char* lpText, const char* lpCaption, unsigned int uType);
int MessageBoxW(HWND hWnd, const wchar_t* lpText, const wchar_t* lpCaption, unsigned int uType);

Each of these function pairs also has a macro without the suffix, that depends on whether the UNICODE macro is defined.

#ifdef UNICODE
   #define MessageBox MessageBoxW
#else
   #define MessageBox MessageBoxA
#endif

In order to make this work, the TCHAR type is defined to abstract away the character type used by the API functions.

#ifdef UNICODE
    typedef wchar_t TCHAR;
#else
    typedef char TCHAR;
#endif

This, however, was a bad idea. You should always explicitly specify the character type.

What I am not getting is how UTF-8 is
conceptually different from a MBCS
encoding ?

MBCS stands for “multi-byte character set”. For the literal minded, it seems that UTF-8 would qualify.

But in Windows, “MBCS” only refers to character encodings that can be used with the “A” versions of the Windows API functions. This includes code pages 932 (Shift_JIS), 936 (GBK), 949 (KS_C_5601-1987), and 950 (Big5), but NOT UTF-8.

To use UTF-8, you have to convert the string to UTF-16 using MultiByteToWideChar, call the “W” version of the function, and call WideCharToMultiByte on the output. This is essentially what the “A” functions actually do, which makes me wonder why Windows doesn’t just support UTF-8.

This inability to support the most common character encoding makes the “A” version of the Windows API useless. Therefore, you should always use the “W” functions.

Update: As of Windows 10 build 1903 (May 2019 update), UTF-8 is now supported as an “ANSI” code page. Thus, my original (2010) recommendation to always use “W” functions no longer applies, unless you need to support old versions of Windows. See UTF-8 Everywhere for text-handling advice.

Unicode is a 16-bit character encoding

This negates whatever I read about the
Unicode.

MSDN is wrong. Unicode is a 21-bit coded character set that has several encodings, the most common being UTF-8, UTF-16, and UTF-32. (There are other Unicode encodings as well, such as GB18030, UTF-7, and UTF-EBCDIC.)

Whenever Microsoft refers to “Unicode”, they really mean UTF-16 (or UCS-2). This is for historical reasons. Windows NT was an early adopter of Unicode, back when 16 bits was thought to be enough for everyone, and UTF-8 was only used on Plan 9. So UCS-2 was Unicode.

More Related Contents:

  1. How to make Unicode charset in cmd.exe by default?
  2. How to have multiple colors in a Windows batch file?
  3. What encoding are filenames in NTFS stored as?
  4. python3 print unicode to windows xp console encode cp437
  5. How to Output Unicode Strings on the Windows Console
  6. Decode PowerShell output possibly containing non-ASCII Unicode characters into a Python string
  7. UTF-16 on cmd.exe
  8. Unicode (utf-8) with git-bash
  9. UTF-8 in Windows 7 CMD [duplicate]
  10. Why isn’t UTF-8 allowed as the “ANSI” code page?
  11. Unicode console I/O in Haskell on Windows
  12. What does “@” mean in Windows batch scripts
  13. Bat file to run a .exe at the command prompt
  14. Difference between .dll and .exe?
  15. Ping with timestamp on Windows CLI
  16. How to add more than one machine to the trusted hosts list using winrm
  17. Powershell Add System Variable
  18. creating batch script to unzip a file without additional zip tools
  19. Can we see the source code for PowerShell cmdlets?
  20. Can’t access 127.0.0.1
  21. Get current Windows user name within Silverlight
  22. Specification of source charset encoding in MSVC++, like gcc “-finput-charset=CharSet”
  23. How to make my program DEP-compatible?
  24. Can I set an environment variable for an application using a shortcut in Windows?
  25. Windows Batch: Set Variables from Text File
  26. How can I make all line endings (EOLs) in all files in Visual Studio Code, Unix-like?
  27. Emacs – Error when calling (server-start)
  28. How to send a simple email from a Windows batch file?
  29. npm global install does not add packages to PATH on Windows 8.1
  30. How to correctly compute the length of a String in Java?

Leave a Comment