Partial Clone and Shallow Clone: Faster Cloning of Large Histories

1. Why large histories become a problem when cloning

A repository with ten years of history, thousands of commits, and large binary files can easily accumulate several gigabytes of object data, even when the current state of the code is comparatively small. A standard git clone, however, always downloads every commit, every tree object, and every blob object for every file ever committed, regardless of whether it is relevant to the current work.

For CI pipelines that clone a fresh repository on every run, this unnecessary data volume quickly adds up to noticeable time and cost. Developers who only work on the current state and rarely dig into history also benefit whenever Git does not have to download every version of every file that ever existed up front.

2. Shallow clone: limiting history to a handful of commits

Shallow clone limits the number of transferred commits via the --depth flag. Instead of the complete history, the client only receives the last n commits of each cloned branch, represented by special boundary commits that Git marks as shallow and whose parent relationship is not resolved any further locally.

The big advantage is simplicity: a --depth=1 clone effectively only downloads the current snapshot, which is ideal for CI builds that only need to compile or test the current state. The downside shows up as soon as history is needed, for example with git blame, git log across older commits, or a git bisect, all of which only work in a limited fashion inside a shallow history.

# Only download the last commit of each branch
git clone --depth=1 https://git.example.com/repo.git

# Fetch additional history afterward
git fetch --deepen=50

# Restore the full history later on
git fetch --unshallow

3. Partial clone: filtering objects instead of commits

Partial clone takes a different approach than shallow clone: instead of limiting the number of commits, the full commit history and tree structure are kept intact, but certain object types, usually large blob objects, are only fetched on demand. That means commands like git log work completely from the start, while the actual file content for older commits is only requested from the server when accessed.

This is made possible by the --filter flag, available since Git 2.19, which tells the server which object types may be omitted from the initial clone. The server has to support partial clone for this to work, which modern hosting providers such as GitHub and GitLab, as well as up to date self hosted Git servers, support by default.

# Fetch all blob objects on demand only, history stays complete
git clone --filter=blob:none https://git.example.com/repo.git

# Also fetch tree objects beyond the root on demand
git clone --filter=tree:0 https://git.example.com/repo.git

4. Filter variants in detail: blob:none, blob:limit, and tree:0

The blob:none filter is the most common one: it fetches no file content up front, only commits and tree structure, and downloads each blob exactly when a checkout or git show actually needs it. The blob:limit=1m filter is a middle ground that transfers small files immediately, but defers large blobs above the given size, which suits repositories with a handful of very large assets.

The tree:0 filter goes one step further and also fetches tree objects for directories beyond the root on demand. Combined with sparse checkout, this results in the smallest possible data transfer overall, since neither tree nor blob objects get loaded for directories that are not visible in the sparse profile anyway.

# Transfer small files right away, defer large blobs
git clone --filter=blob:limit=1m https://git.example.com/repo.git

# Combine partial clone and sparse checkout for the smallest possible transfer
git clone --filter=blob:none --sparse https://git.example.com/monorepo.git

5. How missing objects get fetched in the background

As soon as a command such as git checkout, git log -p, or git blame encounters an object that is not yet available locally, Git automatically requests it from the so called promisor remote, the server registered at clone time as the source for objects needed later. This happens transparently, but depends on network access and can noticeably slow down individual commands on a slow connection.

For operations that request many historical objects at once, for example a full git log -p across the entire history, this on demand fetching can trigger many individual server requests. In such cases, a targeted git fetch --filter=blob:none --unshallow or deliberate fetching with git backfill is worth considering, provided the Git version already supports that command, available since Git 2.30.

# Check which objects are missing locally and marked as promisor objects
git rev-list --objects --all --missing=print | head

# Deliberately backfill missing blob objects for the current branch
git backfill

6. Partial clone and shallow clone in CI pipelines

In CI pipelines, the right choice depends heavily on the job type: a job that only builds and tests the current state benefits most from --depth=1, because it simply does not need any history. A job that has to compute changed files against a target branch, on the other hand, needs at least the shared history of both branches and fails on a too shallow clone due to a missing merge base commit.

For such cases, partial clone often offers the better tradeoff, because the full commit history is preserved and only the expensive blob objects are skipped. Many CI systems now support GIT_CLONE_FILTER environment variables or equivalent pipeline options to set partial clone as the default across a project.

# Example CI configuration using partial clone
variables:
  GIT_STRATEGY: clone
  GIT_CLONE_FILTER: "blob:none"
  GIT_DEPTH: "0"

7. Combining partial clone with submodules

Partial clone initially only affects the main repository, while linked submodules get cloned in full by default as soon as they are initialized. Anyone cloning a repository with many large submodules quickly loses the speed advantage partial clone was supposed to bring to the main repository, since every submodule checkout downloads the full object set of that submodule's repository all over again.

Since Git 2.36, the flag --also-filter-submodules automatically applies the filter to recursively initialized submodules as well, so blob objects inside submodules get fetched on demand just like in the main repository. On older Git versions, the filter has to be applied manually per submodule with git submodule update --init --filter=blob:none, which should be accounted for explicitly in scripts that set up a fresh workstation.

# Automatically apply the partial clone filter to submodules too
git clone --filter=blob:none --also-filter-submodules --recurse-submodules https://git.example.com/repo.git

# Older Git version: apply the filter manually per submodule
git submodule update --init --filter=blob:none

8. Best practices for choosing the right strategy

As a rule of thumb: --depth=1 for isolated build and test jobs with no history requirement, --filter=blob:none for anything that needs history but not every file in every version. Developer workstations should generally use partial clone over shallow clone, since developers do end up needing git blame or older commits occasionally, and a later --unshallow is more expensive than transparently fetching individual blobs.

Large binary files belong in a dedicated system such as Git LFS regardless of partial clone, because partial clone only defers blob downloads, but still transfers them completely into the local object database on actual access and stores them there permanently.

9. Common pitfalls with shallow and partial clones

A frequent mistake is trying to push a branch in a shallow clone that is based on a boundary commit that does not exist on the remote, which Git rejects with an error about missing ancestors. Similarly, many CI tools that rely on merge base calculations for diff views fail against clones that are too shallow, because the common ancestor of two branches lies beyond the fetched depth.

With partial clone, the trap is more subtle: working offline, or losing access to the promisor remote, can suddenly surface an error the moment an object that is not yet available locally is requested. Before longer offline periods, a deliberate git backfill or a full --unshallow respectively a filter reset is worth doing ahead of time.

10. Summary

11. FAQ: Partial and Shallow Clone