Cleaning Git History with BFG Repo-Cleaner

1. When Rewriting Git History Is Actually Necessary

A real case for history cleanup exists when one of two categories of data has made its way into the commit history: leaked credentials such as API keys, database passwords, or private SSH keys, and accidentally committed binaries such as a database dump, a full node_modules folder, or a compiled build artifact. Both cases share one property: a plain git rm followed by a new commit removes the file from the current working state, but the old blob remains permanently in the object store and stays retrievable through git log -p, git show <hash>:path, or any existing clone.

For a leaked secret, that means a standing security risk regardless of how far back the commit dates. For a large binary, it means a repository that stays permanently bloated even after the file has long disappeared from the current HEAD, because git clone still downloads the entire history including every old blob. BFG Repo-Cleaner was built for exactly these two scenarios: secret removal and size reduction.

2. When a New Commit or .gitignore Is Enough

Not every unwanted file in history justifies a history cleanup. If, for example, an .env.example file with no real values was committed, a debug log with no sensitive content, or a configuration file that is meanwhile harmless, a regular commit with git rm --cached file plus an added entry in .gitignore is entirely sufficient. The file disappears from future commits, and the history stays untouched, with every referenced commit hash, tag, and pull request reference intact.

The decisive trade-off is risk versus benefit: a history rewrite changes every downstream commit hash, forces a coordinated force-push, and can create more chaos than the original problem in a careless team. In a small, private repository with theoretical rather than real risk, such as an internal tool with no outside access, the effort often outweighs the actual security gain. The rule of thumb: if the file is still relevant to an attacker or to disk usage today, that justifies the rewrite. If it is not, prevention is enough.

3. What Is BFG Repo-Cleaner

The BFG Repo-Cleaner is a Java-based command line tool by Roberto Tyley, purpose-built for exactly two tasks: removing large or specifically named files from the entire Git history, and replacing text patterns such as passwords or tokens across every historical commit. Unlike the built-in git filter-branch, BFG is not a generic tool for arbitrary history transformations, it is deliberately scoped to these two use cases, which directly translates into a much simpler command line interface.

Technically, BFG does not check out every single commit the way filter-branch does. It reads the object database directly and rewrites only the affected blobs and commit objects, while unchanged trees are left untouched. The result is a run that typically takes seconds to a few minutes on a mid-sized repository, while the same operation with filter-branch can take hours depending on history length. BFG ships as a single .jar file and only needs an installed Java runtime.

# Install BFG Repo-Cleaner (macOS via Homebrew)
$ brew install bfg
$ java -version
openjdk version "17.0.9" 2023-10-17

# Alternative: download the jar directly (Linux/Windows/macOS)
$ curl -LO https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar
$ java -jar bfg-1.14.0.jar --version
BFG 1.14.0

# Optional: shell alias for daily use
$ echo 'alias bfg="java -jar ~/bin/bfg-1.14.0.jar"' >> ~/.bashrc

4. Installing BFG Repo-Cleaner

Installing BFG Repo-Cleaner is straightforward on every common platform. On macOS, brew install bfg installs the tool along with its Java dependencies through Homebrew. On Linux and Windows, download the current .jar file directly from the official project page and invoke it with java -jar bfg.jar. In both cases, an installed Java Runtime Environment version 8 or newer is required, which can be checked with java -version.

For daily use, a shell alias that shortens the full java -jar call and makes BFG feel like a native command is worthwhile. One thing matters before the first production run: BFG never operates on a regular working clone, it requires a fresh, mirrored clone of the repository via git clone --mirror. This mirror clone contains every branch, tag, and ref in its internal Git representation and is the only safe working basis for a history cleanup.

# BFG never works on a regular working clone: create a fresh mirror clone first
$ git clone --mirror git@github.com:mironsoft/example-repo.git
$ cd example-repo.git

# Remove a specific file by name from the entire history
$ bfg --delete-files dump.sql
...
Deleted files
-------------
    dump.sql (12 commits)

# Remove all blobs larger than 10 MB, regardless of file name
$ bfg --strip-blobs-bigger-than 10M

# Remove a whole directory, e.g. an accidentally committed node_modules
$ bfg --delete-folders node_modules --no-blob-protection

5. Removing Large Files with bfg --delete-files

The bfg --delete-files command removes files by name or glob pattern from every commit in the history, regardless of when or where they were created. A typical call such as bfg --delete-files dump.sql scans the entire history for files with exactly that name and removes every occurrence, without requiring the path or commit order to be specified manually. Patterns such as *.zip or node_modules work with the same syntax using wildcards or directory names.

A complementary option is --strip-blobs-bigger-than 10M, which removes every blob exceeding a set size regardless of file name, useful for repositories where large files were repeatedly committed under changing names. BFG logs in detail after every run which commits were changed, and writes a mapping of old to new commit hashes to .git/bfg-report. These reports are the basis for verifying, before the final push, that only the intended objects were actually removed.

6. Purging Secrets with bfg --replace-text

For removing secrets, use bfg --replace-text patterns.txt, where the patterns.txt file contains, one per line, text patterns or regular expressions that BFG replaces in every affected file across every historical commit with a placeholder such as ***REMOVED***. Unlike deleting an entire file, the file itself and its structure remain intact, only the sensitive text content is obscured, which is particularly useful for configuration files that also contain legitimate entries still needed alongside the secret.

The patterns file supports both plain strings and regular expressions using the ==regex suffix, which allows replacing every value matching a certain environment variable pattern in a single pass. Important: BFG only replaces text in files that are not detected as binary, and it works line by line, which means multi-line secrets are not reliably caught by a single pattern. For such edge cases, an additional manual review of the generated report before the push is essential.

# replace-patterns.txt: one pattern per line for bfg --replace-text
# Plain strings are matched literally
sk_live_51H8x9k2mN3pQ7rS

# Suffix ==regex enables full regular expression matching
DB_PASSWORD=.*==>DB_PASSWORD=***REMOVED***==regex
AWS_SECRET_ACCESS_KEY=[A-Za-z0-9/+=]{40}==regex

# A trailing ==> value overrides the default ***REMOVED*** placeholder
ghp_[a-zA-Z0-9]{36}==>***TOKEN_REMOVED***==regex

7. Comparison to git filter-branch

git filter-branch was for years the only history rewrite tool built into Git, and it is correspondingly generic: it checks out every commit individually, runs an arbitrary filter command, and re-commits the result, which works for virtually any transformation but is exactly why it is exceptionally slow. On a repository with several thousand commits, a single filter-branch run with a --tree-filter can take several hours, because a full checkout into a temporary directory happens for every single commit.

The Git documentation itself has explicitly warned against filter-branch for several years now, labeling it "deprecated" in the man page with a clear pointer to use git filter-repo instead. Beyond speed, error proneness is also a problem: the combination of shell filters, implicit environment variables, and the many edge cases around merges and empty commits regularly leads, in practice, to silent, hard-to-detect errors in the rewritten history. BFG and filter-repo avoid this problem by manipulating the object database natively and directly instead of repeatedly checking commits out.

8. Comparison to git filter-repo

git filter-repo is the official successor to filter-branch, developed by the Git maintainers themselves and officially recommended. As a Python tool, it operates directly on the object database much like BFG and achieves comparable speeds, but it brings a much wider feature set: renaming paths, transforming commit messages via callbacks, globally rewriting authors, or defining complex conditional filter rules that go far beyond the file deletion and text replacement that BFG deliberately limits itself to as its only two use cases.

That feature set comes at a cost: the command line syntax of filter-repo is considerably more complex than the two BFG flags, its documentation assumes Python knowledge for advanced callbacks, and filter-repo also strictly requires a fresh clone rather than an existing working copy. For the two most common cases, secret removal and size reduction, BFG remains the more pragmatic choice thanks to its minimal learning curve. Once more complex restructuring is needed, however, such as splitting a monorepo or rewriting author identities after a company merger, filter-repo is the more powerful, officially supported tool.

# git filter-repo: official replacement for filter-branch (pip install)
$ pip install git-filter-repo

# filter-repo also requires a fresh clone, never an existing working copy
$ git clone git@github.com:mironsoft/example-repo.git fresh-clone
$ cd fresh-clone

# Equivalent of bfg --delete-files, with glob support
$ git filter-repo --path dump.sql --invert-paths

# Equivalent of bfg --strip-blobs-bigger-than
$ git filter-repo --strip-blobs-bigger-than 10M

# filter-repo can do things BFG cannot: rewrite author identity globally
$ git filter-repo --mailmap .mailmap

9. Force-Push, Team Coordination, and Post-Cleanup Steps

By far the most critical step of any history cleanup is not the tool, it is the coordination: a rewrite changes the SHA hash of every affected commit and every commit downstream of it, which makes the new history incompatible with every existing clone. A force-push is then mandatory, typically with git push --force origin refs/heads/* refs/tags/* from the cleaned mirror clone, and it must be coordinated with the entire team, any running CI pipelines, and open pull requests, ideally inside an announced maintenance window.

After the force-push, no team member may update their old clone with git pull or git rebase, because both commands can, through local refs and reflogs, silently push the supposedly removed objects right back to the server and effectively undo the cleanup. The only safe path is to delete the old clone entirely and re-clone the repository from scratch. On the server itself, git reflog expire --expire=now --all and git gc --prune=now --aggressive must also run to actually evict the old objects from the pack files and reclaim disk space.

Even after a technically successful cleanup, one rule does not move: every leaked credential still has to be rotated. Old forks, other developers' local clones, CI artifact caches, and cached commit views on platforms like GitHub itself may still contain the secret, no matter how thoroughly your own history was cleaned. The table below compares the three tools side by side.

# Run BFG against the mirror clone using the patterns file
$ bfg --replace-text replace-patterns.txt

# Inspect the report before touching the remote
$ cat .git/bfg-report/*/cache-stats.txt
$ git log --all --oneline | head -5   # verify commit hashes changed

# Clean up dangling refs BFG leaves behind, then push everything
$ cd example-repo.git
$ git reflog expire --expire=now --all
$ git gc --prune=now --aggressive

# Coordinated force-push: rewrite every branch and tag on the remote
$ git push --force origin refs/heads/* refs/tags/*

For the vast majority of cases, secret removal and size reduction, BFG is the fastest and most straightforward choice. Only for noticeably more complex restructuring does switching to filter-repo pay off, and by now even the Git maintainers actively advise against filter-branch.

10. Summary

A Git history cleanup is only justified when leaked credentials or permanently bloating binaries are genuinely stuck in the history, not for every unwanted but harmless file. BFG Repo-Cleaner solves exactly these two cases with the --delete-files and --replace-text commands, noticeably faster and simpler than the deprecated git filter-branch, without needing the flexibility of the officially recommended git filter-repo.

The technical rewrite is only half the work. Because every commit hash changes after the rewrite, it takes a coordinated force-push, a clearly communicated maintenance window, a complete re-clone by every team member instead of a pull, plus reflog expire and git gc --prune=now --aggressive on the server to actually reclaim disk space. And regardless of how successful the cleanup was: leaked credentials always need to be rotated.

11. FAQ: Cleaning Git History with BFG Repo-Cleaner