Git Performance Tuning for Very Large Monorepos

1. Where Git hits limits on very large repositories

Git was originally designed for the Linux kernel and scales remarkably well out of the box, but a monorepo with several million files, tens of thousands of branches, and years of history places different demands on it than a classic open source project. Three symptoms typically show up first: an initial clone that takes hours instead of seconds, a working directory that consumes far more disk space than any single developer actually needs, and a git status that noticeably takes longer than a single keystroke would suggest.

All three symptoms share a common cause: Git's fundamental model assumes every clone holds the complete history and every checkout holds the complete file tree. In a monorepo where a single team actually needs only a small slice of the code, that model turns into a performance problem that cannot be solved by faster hardware alone and requires deliberate configuration instead.

2. Partial clone: not downloading every file up front

Partial clone lets you skip certain objects when cloning and fetch them from the server only when actually needed. With the blob:none filter, only commit and tree objects are downloaded initially, while the actual file contents, the blobs, are fetched only once a checkout, diff, or blame actually requires them. That drastically reduces the initial clone size, since years of history are usually many times larger than the current working state.

The blob:limit=1m filter is a variant that only skips blobs above a certain size, which is useful when a handful of large binary files are the main problem while the rest of the source stays manageable. The server, typically via uploadpack.allowFilter, has to explicitly support partial clone, which is the default on modern hosting solutions like GitHub, GitLab, and Gerrit.

# Load only commits and trees up front, blobs on demand
git clone --filter=blob:none https://example.com/monorepo.git

# Load only blobs under 1 MB up front
git clone --filter=blob:limit=1m https://example.com/monorepo.git

# Missing objects are fetched automatically as needed
git checkout feature/payment-service

3. Sparse checkout: materializing only the relevant subtree

While partial clone reduces the history, sparse checkout ensures that only a specific subtree of the file system is actually materialized in the working directory. In the modern cone mode, you specify a list of directories, and Git checks out only those paths plus the files at the repository root, while everything else is tracked in the internal index but never written to disk.

For a team working on just one microservice inside a huge monorepo, this reduces the number of files in the working directory from several million down to a few thousand, which directly affects the speed of git status, git add, and file indexing in the editor. Cone mode is noticeably faster than the older, pattern-based sparse checkout mode, since Git works with simple directory prefixes instead of arbitrary glob patterns.

git sparse-checkout init --cone
git sparse-checkout set services/payment services/shared-libs

# The working directory now contains only these paths plus root files
ls

4. Commit-graph and bloom filters for faster history queries

Commands such as git log, git merge-base, or git blame need to trace the relationships between commits, which means with millions of commit objects Git has to read many individual objects from disk for every query. The commit-graph file stores these relationships in a compact, precomputed format, so Git can traverse the history without deserializing every single commit object.

Bloom filters extend the commit-graph with a fast answer to whether a given path even changed in a commit, before Git computes the actual diff. That massively speeds up a path filtered git log query in a monorepo in particular, since without bloom filters Git would have to check every single commit across the entire history to see whether the path was affected.

git commit-graph write --reachable --changed-paths

# Keep it up to date automatically on every fetch
git config fetch.writeCommitGraph true

5. File system monitor: status checks without a full scan

Without further help, git status has to check every file in the working directory individually for changes, which is noticeably slow with millions of files even on a fast SSD. The built-in file system monitor, fsmonitor, instead listens continuously to file system events from the operating system and hands Git an already pre-filtered list of changed files, eliminating the expensive full scan.

Since Git 2.37, a cross-platform fsmonitor has been available as a built-in daemon feature that works without an external dependency on macOS, Linux, and Windows and can be enabled per repository with a single configuration line. Combined with sparse checkout, which already reduces the number of files to watch, the effect is largest.

git config core.fsmonitor true

# Check daemon status
git fsmonitor--daemon status

6. Multi-pack-index and automated maintenance

An active repository accumulates many individual pack files over time, since Git creates a new pack on every fetch. Without maintenance, Git might have to search each of these pack files individually during an object lookup. The multi-pack-index, MIDX for short, combines the indexes of multiple pack files into one compact structure, so object lookups stay fast regardless of how many pack files exist.

The git maintenance command bundles commit-graph updates, multi-pack-index upkeep, object compression, and other cleanup tasks into a schedulable background service, set up via git maintenance start as a scheduled task in the operating system. Instead of an occasional manual git gc, which can block for minutes or hours on a large repository, maintenance tasks then run in small, unobtrusive intervals in the background.

# Set up maintenance as a background service (cron/launchd/Task Scheduler)
git maintenance start

# Tasks that get scheduled automatically
git maintenance run --task=commit-graph
git maintenance run --task=pack-refs
git maintenance run --task=incremental-repack

7. Protocol v2 and server side factors

Beyond client configuration, the Git protocol in use also matters. Protocol v2 negotiates references more efficiently than the older protocol v0, because the server no longer has to enumerate every reference in the repository on a fetch and can instead filter for the requested references directly. In a monorepo with tens of thousands of branches and tags, that noticeably reduces the overhead of every single fetch operation.

On the server side, it is also worth monitoring bandwidth and CPU load for uploadpack operations, since a monorepo with many concurrent CI clones hits server limits much faster than a small repository. A dedicated cache server or a geo replica, as offered by GitHub Enterprise and GitLab Geo, further offloads the primary server for globally distributed teams.

git config protocol.version 2

8. Shallow clones in CI: weighing the trade-offs deliberately

In CI pipelines that only need to build and test the current state of a branch, a shallow clone with --depth=1 is often the fastest option, since no history is transferred. The downside shows up as soon as a job actually needs history, for example for git describe, a comparison against the last successful build, or a blame step in an analysis job, since a shallow clone either refuses these operations or produces incorrect results.

A pragmatic middle ground combines a shallow clone with a targeted git fetch --deepen=50 or --unshallow exactly when a particular job needs more history, instead of transferring full depth for every job by default. For monorepos with many parallel CI jobs, this distinction makes a measurable difference in overall pipeline runtime.

9. Measuring improvements instead of just assuming them

Every measure described here should be backed by concrete before and after timing, since the benefit varies significantly depending on repository structure. Git's built-in Trace2 mechanism, enabled via GIT_TRACE2_PERF=1, provides detailed timing for individual internal operations and shows exactly which step of a command actually consumes the time.

In practice a small set of reference commands is worth keeping around, for example a fresh clone, a git status after a larger merge, and a path filtered git log query against a frequently changed directory, measured regularly with the same method. Only that way can you objectively judge whether a new Git version or a changed configuration actually brings an improvement or just feels faster subjectively.

GIT_TRACE2_PERF=1 git status 2> trace-status.log
tail -n 40 trace-status.log

10. Summary

11. FAQ: Monorepo Performance