git, msysgit, accents, utf-8, the definitive answers

by

Update Oct. 2021: With Git 2.34 (Q4 2021), the unicode character width table (used for output alignment) has been updated.

See commit 187fc8b (17 Sep 2021) by Carlo Marcelo Arenas Belón (carenas).
(Merged by Junio C Hamano — gitster — in commit 3d875f9, 28 Sep 2021)

unicode: update the width tables to Unicode 14

Update Feb. 2017 (Git 2.12): The character width table has been updated to match Unicode 9.0.
The update_unicode.sh is moved it into contrib/update-unicode: see its README.

Update August 2014 (git 2.1): commit a67c821 (Torsten Bögershausen (tboegi)) adds support for Unicode 7.0.

Update April 2014: commit d813ab9 (Torsten Bögershausen (tboegi)) adds support for Unicode 6.3
(git 1.9.2):

Unicode 6.3 defines more code points as combining or accents.
For example, the character “ö” could be expressed as an “o” followed by U+0308 COMBINING DIARESIS (aka umlaut, double-dot-above).
We should consider that such a sequence of two codepoints occupies one display column for the alignment purposes, and for that, git_wcwidth() should return 0 for them.

Affected codepoints are:

U+0358..U+035C
U+0487
U+05A2, U+05BA, U+05C5, U+05C7
U+0604, U+0616..U+061A, U+0659..U+065F

Earlier unicode standards had defined these as “reserved”.

Only the range 0..U+07FF has been checked to see which codepoints need to be marked as 0-width while preparing for this commit; more updates may be needed.

Update April 2012: Unicode support is released in version 1.7.10. See this page for notes and settings you should set.

Namely:

git config [--global] core.quotepath off
git config [--global] i18n.logoutputencoding utf8
git config [--global] i18n.commitencoding utf8
git config [--global] --unset svn.pathnameencoding

The recodetree check command scans the entire history of a git repository and prints all non-ASCII file names. If the output is empty, no migration is necessary.

Update February 2012: patches for UTF-8 supports are comming in branch ‘devel’ of msysgit repo on GitHub, including Update less settings for UTF-8 .

The Git for Windows Google+ page mentions:

Karsten Blees’ UTF-8 patches for Git for Windows has now been merged to ‘devel‘.
This means the upcoming release will support Unicode filenames!

May 2011

I believe the msysgit issue 80 has the latest on that bug.
Also described in issue 376.

For example:

This is what happens:

  1. git on Windows operates on file names and treats them essentially as byte streams.
    In your case, the streams happen to be UTF8 encoded text.

  2. git on Windows asks the runtime to create a file, and passes it the byte stream.

  3. Since internally on Windows everything is Unicode, the runtime converts the byte
    stream to UTF16 using the currently set locale (aka “codepage”).
    That is, it effectively interprets the byte stream as CP949 (Korean) encoded text.
    Apparently, some of the UTF8 byte sequences are invalid CP949 sequences, and the conversion fails (“Invalid argument”); or if the UTF8 sequences happen to be correct CP949 sequences, the result is (most likely) a different character.

The true fix should be on MingW though:

It occurs to me that one solution would be this: solve it at the GCC C run-time
library level.
That is, for the mingw GCC run-time library on Windows, make it possible via build-time options to be in a mode where the command-line parameters (passed to main()) and file I/O functions use the underlying Windows Unicode API calls, and translate to/from UTF-8 encoding in C’s standard function APIs that use byte-strings.
That would “just work” for git perhaps, and could be useful for other Linux-originated open source projects running the Windows environment.

ak2 comments that MingW isn’t the right place for this fix:

“MinGW compilers provide access to the functionality of the Microsoft C runtime and some language-specific runtimes.
MinGW, being Minimalist, does not, and never will, attempt to provide a POSIX runtime environment for POSIX application deployment on MS-Windows.
If you want POSIX application deployment on this platform, please consider Cygwin instead.”

There is some work in progress on a msysgit variant to support unicode.

More Related Contents:

  1. Trying to fix line-endings with git filter-branch, but having no luck
  2. How to change line-ending settings
  3. Where do the settings in my Git configuration come from?
  4. Difference between msysgit and Cygwin + git?
  5. How do I force git to use LF instead of CR+LF under windows?
  6. Escape @ character in git proxy password
  7. fatal: early EOF fatal: index-pack failed
  8. Git 1.6.4 beta on Windows (msysgit) – Unix or DOS line termination
  9. Git Bash is extremely slow on Windows 7 x64
  10. Git under windows: MSYS or Cygwin?
  11. How do I use Notepad++ (or other) with msysgit?
  12. git: ‘credential-cache’ is not a git command
  13. use Winmerge inside of Git to file diff
  14. Msysgit bash is horrendously slow in Windows 7
  15. Fix msysGit Portable $HOME location
  16. github: No supported authentication methods available
  17. Which is the smarter git protocol, ssh or git(over ssh) or https protocol?
  18. gitolite push error -> remote: ENV GL_RC not set
  19. Clone works, remote push doesn’t. Remote repository over copssh
  20. Change the location of the ~ directory in a Windows install of Git Bash
  21. Can Git treat ZIP files as directories and files inside the ZIP as blobs?
  22. List of all commands that cause git gc –auto
  23. How do I write a batch file which opens the GitBash shell and runs a command in the shell?
  24. Make git diff show UTF8 encoded characters properly
  25. How to retrieve the hash for the current commit in Git?
  26. What determines default branch after “git clone”?
  27. Dynamically Fill Jenkins Choice Parameter With Git Branches In a Specified Repo
  28. Git: How to diff two different files in different branches?
  29. git refusing to fetch into current branch
  30. What is git doing when it says it is “resolving deltas”?

Leave a Comment