EMZETT.
Login

Emzett Wiki

Git LFS and large files in repositories

Properties1
tagsglossar, git, werkzeuge

2 min read

Git LFS and large files in repositories

In short: Git is made for source code; large binary files such as videos or models bloat repositories, and Git LFS moves them out and stores only pointers in the repository.

In more detail: Git stores every version of every file permanently. A 100 MB file changed ten times quickly occupies a gigabyte, and every clone downloads the entire history. Platforms limit file sizes (for example GitHub: warning from 50 MB, block from 100 MB per file).

In Depth

The problem

Large binary files (high-resolution images, videos, audio, game assets, databases, AI models, design files) are hard to diff and compress. The repository becomes large and slow (cloning, checkout), CI runs take longer. Even accidentally committed large files remain in history even if deleted later.

Git LFS (Large File Storage)

git lfs install
git lfs track "*.psd" "*.mp4" "models/*.bin"   # set patterns
git add .gitattributes
git add model.bin
git commit -m "Add model"
git push

In the repository lies only a small pointer with hash and size; the real file lies on an LFS server and is downloaded on checkout. Advantages: lean history, faster cloning, files only when needed (git lfs pull, GIT_LFS_SKIP_SMUDGE=1). Disadvantages: additional tool, quotas/costs with providers, harder clean-up.

Alternatives

  • Not putting large files into Git at all: storage in object storage (S3, cloud storage) or release artifacts and only a reference/download script.
  • DVC (Data Version Control) and Hugging Face Hub for data and AI models.
  • Git-Annex for very large data sets.
  • Perforce/Plastic SCM for game studios with huge assets.
  • Partial clones and sparse checkout (git clone --filter=blob:none) reduce data when cloning.

Cleaning up and preventing

.gitignore for build outputs, dependencies (node_modules), large test data. For already checked-in giant files: clean history with git filter-repo or BFG Repo-Cleaner (history is rewritten: inform the team, everyone clones anew). Check size: git count-objects -vH, git rev-list --objects --all | …. Pre-commit hook against large files (check-added-large-files).

See also: gitignore and submodules, Git hooks, Git