Vault7: CIA Hacking Tools Revealed
Navigation: » Latest version
Owner: User #524297
Remove Binary Files from a Repository
Beforehand: Avoid committing binary files to a Git repository in the first place
Git is a great distributed version control system. It's fantastic for easily storing changes to text files wherever you are, and then easily copying them up to a server or servers or sharing them across the office.
Git functions at its best while computing differences and changes in text files. Textual changes are easy to read and understand. But this difference function is useless for binary file data, especially if those files change across multiple commits. Data about changes in binary files comingled with changes about text files pollutes the commits and makes them difficult to read and understand. Git history should be small, incremental, and above all clear and easy to understand.
There is another very good reason for keeping binary files out of your repository: The files are usually much bigger. Images, videos, and compiled binaries are all much bigger than text files. If you commit them to your repository, the size of your repository will become much larger. This matters not because storage is expensive - it's not. It matters because the point of using a distributed VCS is that it makes it cheap and easy to clone and navigate. You want to be able to spin up a new machine and copy the repository as quickly as possible. You want to be able to switch branches as quickly as possible. If you commit any significant number of binary files you will see all of these tasks slow down considerably, because Git is storing the ENTIRE binary file in your repository for EVERY change made to it (not just the differences over time, like it would with a text file). Git repositories store the history of each file, so Git decides that naturally you would need every possible copy of that one binary file.
The other big downside to committing binary files: once you've committed them they are in the repository history and are very annoying to remove. You can delete the files from the current version of the project -- but they'll remain in the repository history, meaning that the overall repository size will still be large. Git still considers those files as part of the ongoing history, so it still stores those copies just in case you want to reminisce. This is also why you see security issues with some open source projects, because they leave private credentials or password data in their repositories and don't remove the files well when discovered.
So all of that being said, here's a short list of things you should NOT be including your Git repository:
- Any file that can be dynamically generated by your build system (binaries, configurations, archive files, build artifacts)
- Any binary deliveries, SDKs, pre-compiled libraries or programs
- Open source/external codebases (see Add an Open Source Library to a Stash Project)
Afterward: How you can fix it
Ok, so you messed up. We all make mistakes. Let's chalk it up as a learning experience and move on to greater things.
The sad fact is, however, you can only completely remove large files from the repository by rewriting history.
 |
Rewriting history is extremely dangerous! It will overwrite all commits since files were added, producing a completely different version of the revision history. Different hashes, different HEADs, different everything. You are overwriting your one Source of Truth in Stash. "With great power comes great responsibility." Caveat emptor. Consider yourself warned. |
|---|
If you have shared your repository with anyone, or stored it anywhere else (like a VMVirtual Machine or another server), you must make sure that all versions of these repositories get updated to your new version before anyone tries to add any new work on top. This can be a huge headache.
However, if you understand the risks you can rewrite history using a script I found online (I suggest reading it and understanding it first however).
It first uses git filter-branch to remove the files from the commits, and then deletes the relevant caches of the files.
Make sure you've committed all your work and have a backup copy of your up-to-date repository somewhere. Then do the following:
# Get hold of the script and install it
$ git clone ssh://git@stash.devlan.net:7999/~User #524297/git-prune-files.git
$ sudo cp git-prune-files/git-prune-files /usr/bin
$ sudo chmod +x /usr/bin/git-prune-files
# Change to your project directory
$ cd ~/projects/my-git-project
# Remove the relevant files
$ git-prune-files static/images
This might take a while and should give you an output something like this:
...
rm 'static/images/9419.jpg'
rm 'static/images/9420.jpg'
rm 'static/images/9421.jpg'
Rewrite 325bfc6a34e33a9d4ef4b19ec88b52dfdc3f1e74 (9421/9421)
Ref 'refs/heads/master' was rewritten
Counting objects: 22041, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (21761/21761), done.
Writing objects: 100% (22041/22041), done.
Total 22041 (delta 15312), reused 5817 (delta 0)
Now make absolutely sure this version of the repository replaces all copies of your repository (yours and anyone else's) immediately.
# Overwrite the repository on Stash with your shiny clean local copy.
# In other words, your local copy becomes the new "truth" on the Stash server.
# In other other words, this is a command that can ruin your day (and the day of your team).
$ git push --force origin masterAlternative Solutions
I can hear the uproar already. I understand that there might be some things out there that you want to track. I know that some folks like to track their binary deliveries in Git for whatever reasons.
I wholeheartedly suggest that you keep these binaries out of your main source repository, for all of the reasons outlined above. If files are not necessary to build your source, then they should not be required when checking out your code. Use other repositories within your Stash project (see Stash Project DOs and DON'Ts). If you are trying to save something that is required for a build process (an APIApplication Programming Interface or library), look at Use Git Submodules to Manage Libraries .
Bottom line: No one should have to clone a 20+ GB repository in order to edit scripts. 
(I'm looking at you, Spottsroide!)
Related articles
('contentbylabel' missing)
('details' missing)