Vault7: CIA Hacking Tools Revealed
Navigation: » Directory » Git Distributed Version Control » Git Distributed Version Control Home » How-to articles
Owner: User #524297
Remove Binary Files from a Git Repository
Beforehand: Avoid committing binary files to a Git repository in the first place
Git is a great distributed version control system. It's fantastic for easily storing changes to text files wherever you are, and then easily copying them up to a server or servers or sharing them across the office.
Git functions at its best while computing differences and changes in text files. Textual changes are easy to read and understand. But this difference function is useless for binary file data, especially if those files change across multiple commits. Data about changes in binary files comingled with changes about text files pollutes the commits and makes them difficult to read and understand. Git history should be small, incremental, and above all clear and easy to understand.
There is another very good reason for keeping binary files out of your repository: The files are usually much bigger. Images, videos, documentation, and compiled binaries are all much bigger than text files. If you commit them to your repository, the size of your repository will become much larger. This matters not because storage is expensive - it's not. It matters because the point of using a distributed VCS is that it makes it cheap and easy to clone and navigate. You want to be able to spin up a new machine and copy the repository as quickly as possible. You want to be able to switch branches as quickly as possible. If you commit any significant number of binary files you will see all of these tasks slow down considerably, because Git is storing the ENTIRE binary file in your repository for EVERY change made to it (not just the differences over time, like it would with a text file). Git repositories store the history of each file, so Git decides that naturally you would need every possible copy of that one binary file.
The other big downside to committing binary files: once you've committed them they are in the repository history and are very annoying to remove. You can delete the files from the current version of the project -- but they'll remain in the repository history, meaning that the overall repository size will still be large. Git still considers those files as part of the ongoing history, so it still stores those copies just in case you want to reminisce. This is also why you see security issues with some open source projects, because they leave private credentials or password data in their repositories and don't remove the files well when discovered.
So all of that being said, here's a short list of things you should NOT be including your Git repository:
- Any file that can be dynamically generated by your build system (binaries, configurations, archive files, build artifacts)
- Any binary deliveries, SDKs, pre-compiled libraries or programs
- Open source/external codebases (see Add an Open Source Library to a Stash Project)
Afterward: How you can fix it
Ok, so you messed up. We all make mistakes. Let's chalk it up as a learning experience and move on to greater things.
The sad fact is, however, you can only completely remove large files from the repository by rewriting history.
 |
Rewriting history is extremely dangerous! It will overwrite all commits since files were added, producing a completely different version of the revision history. Different hashes, different HEADs, different everything. You are overwriting your one Source of Truth in Stash. "With great power comes great responsibility." Caveat emptor. Consider yourself warned. |
|---|
If you have shared your repository with anyone, or stored it anywhere else (like a VMVirtual Machine or another server), you must make sure that all versions of these repositories get updated to your new version before anyone tries to add any new work on top. This can be a huge headache.
However, if you understand the risks you can rewrite history using a script I found online (I suggest reading it and understanding it first however).
It first uses git filter-branch to remove the files from the commits, and then deletes the relevant caches of the files.
Make sure you've committed all your work and have a backup copy of your up-to-date repository somewhere. Then do the following:
# Get hold of the script and install it
$ git clone ssh://git@stash.devlan.net:7999/~User #524297/git-prune-files.git
$ sudo cp git-prune-files/git-prune-files /usr/bin
$ sudo chmod +x /usr/bin/git-prune-files
# Change to your project directory
$ cd ~/projects/my-git-project
# Remove the relevant files
$ git-prune-files static/images
This might take a while and should give you an output something like this:
...
rm 'static/images/9419.jpg'
rm 'static/images/9420.jpg'
rm 'static/images/9421.jpg'
Rewrite 325bfc6a34e33a9d4ef4b19ec88b52dfdc3f1e74 (9421/9421)
Ref 'refs/heads/master' was rewritten
Counting objects: 22041, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (21761/21761), done.
Writing objects: 100% (22041/22041), done.
Total 22041 (delta 15312), reused 5817 (delta 0)
Now make absolutely sure this version of the repository replaces all copies of your repository (yours and anyone else's) immediately.
# Overwrite the repository on Stash with your shiny clean local copy.
# In other words, your local copy becomes the new "truth" on the Stash server.
# In other other words, this is a command that can ruin your day (and the day of your team).
$ git push --force origin masterAlternative Solutions
I can hear the uproar already. I understand that there might be some things out there that you want to track. I know that some folks like to track their binary deliveries in Git for whatever reasons.
I wholeheartedly suggest that you keep these binaries out of your main source repository, for all of the reasons outlined above. If files are not necessary to build your source, then they should not be required when checking out your code. Use other repositories within your Stash project (see Stash Project DOs and DON'Ts). If you are trying to save something that is required for a build process (an APIApplication Programming Interface or library), look at Use Git Submodules to Manage Libraries .
Again, all of this applies to documentation files too!
Word documents, Pages, Open Office files, and PDFs. Seen them all in Git repositories. They are ALL binary files. Investigate writing your documentation in a plain text format (e.g. Markdown) and converting it to a final presentation format (e.g. PDFPortable Document Format) during your build process. Not only can it be automated, but then you can easily track changes in your documentation using Git as well!
Bottom line: No one should have to clone a 20+ GB repository in order to edit scripts. 
(I'm looking at you, Spottsroide!)
Related articles
('contentbylabel' missing)
('details' missing)
Comments:
-
2016-02-09 15:42 [User #20251227]:
This. Page. Is. Handy! <whew>
-
2015-12-30 10:54 [User #524297]:
Yep, never liked LaTeX myself for that exact reason. Markdown -> PDFPortable Document Format conversion via pandoc is my preferred experimental method at the moment, especially since Markdown is just so darn easy. Some other folks in AEDApplied Engineering Devision are also looking at Confluence -> PDFPortable Document Format as well, and while I like the "living documentation" aspect, you lose the ability to track the changes alongside your code.
-
2015-12-30 10:09 [User #20251227]:
Version controlling docs is ugly. The markdown -> PDFPortable Document Format idea is something I haven't explicitly seen before.
I have seen LaTeX-based docs before & tried it myself. The positives are that the doc is text and is treated as such, and that making the final form (e.g., PDFPortable Document Format) is like compiling code (e.g., can set up a make file, ...). The negatives are that the learning curve is a bit steep, crafting all the markup & editing can be a pain, and the fact the compilation steps can get a bit unwieldy. Additionally one still runs into the problem of what to do with version controlling the supporting binary bits (e.g., a JPG of a graph which gets embedded as "Figure foo").
-
2015-12-30 03:54 [User #524297]:
Can't take full credit for authorship, it's a modified version of a blog post I found online. I tweaked it to emphasize the more important points, and provide some solutions to counterpoints that I have heard before. Just added a new bit about documentation files. Also another big offender.
-
2015-12-29 15:45 [User #20251227]:
Good stuff. Amazing how many folks out in the wide world don't understand the ramifications of checking binaries into Distributed Version Control Systems (e.g., Mercurial, Git). One can make an argument that checking binaries into a centralized VCS such as Subversion isn't **AS** bad (but still not optimal). The section on backing out such things via Git is useful info. I don't even know how one would approach the problem in Mercurial as it takes more of a "No, history is, in fact, actually, for realZ immutable" tack than Git.