How to handle a large git repository?

by

Update April 2015: Git Large File Storage (LFS) (by GitHub).

It uses git-lfs (see git-lfs.github.com) and tested with a server supporting it: lfs-test-server:
You can store metadata only in the git repo, and the large file elsewhere.

https://cloud.githubusercontent.com/assets/1319791/7051226/c4570828-ddf4-11e4-87eb-8fc165e5ece4.gif

Original answer (2012)

One solution, for large binary files that don’t change much, is to store them in a different referential (like a Nexus repository), and version only a text file which declares which version you need.
Using an “artifact repository” is easier than storing binary elements in a source repo (made for comparing versions and merging between branches, which isn’t of much use for said binaries).

The other solution, more git-centric, is git-annex:

git-annex allows managing files with git, without checking the file contents into git.
While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space.

It is however not compatible with Windows.

A more generic solution could be git-media, which also allows you to use Git with large media files without storing the media in Git itself.

Finally, the easiest solution is to isolate those binaries in their own git submodule as you mention in your question: it isn’t very satisfactory, and the initial clone will still take times, but the next updates for the parent repo will be short.

More Related Contents:

  1. How to remove file from Git history?
  2. What’s the purpose of git-mv?
  3. master branch and ‘origin/master’ have diverged, how to ‘undiverge’ branches’?
  4. How to make git mark a deleted and a new file as a file move?
  5. How do I reset or revert a file to a specific revision?
  6. Why does Git treat this text file as a binary file?
  7. Show git ahead and behind info for all branches, including remotes
  8. There is no tracking information for the current branch
  9. How to tell which commit a tag points to in Git?
  10. How do I change the default location for Git Bash on Windows?
  11. How to replace local branch with remote branch entirely in Git?
  12. How do I avoid the specification of the username and password at every git push?
  13. What to do with commit made in a detached head
  14. Cannot push to Git repository on Bitbucket
  15. Recursively add files by pattern
  16. Git push error: “origin does not appear to be a git repository”
  17. Merge git repository in subdirectory
  18. How to permanently delete a file stored in GIT? [duplicate]
  19. Switch branch names in git
  20. How to sync with a remote Git repository?
  21. git reset asks ‘more?’
  22. Git push failed, “Non-fast forward updates were rejected”
  23. GIT commit as different user without email / or only email
  24. Changing the Git user inside Visual Studio Code
  25. Clone only the .git directory of a git repo
  26. Git – head (lowercase) vs HEAD (uppercase)
  27. Why has git-filter-branch not rewritten tags?
  28. PowerShell Capture Git Output
  29. git – remote add origin vs remote set-url origin
  30. git clone with HTTPS or SSH remote?

Leave a Comment