error: object file is empty

A coworker could not push a branch of his development to our company Stash server this afternoon. He tried once in the morning, but the server went down, so he stayed late until the server was online again and then tried again to push his commits, only to be hit with a very cryptic error.

I stayed late debugging it with him, and here’s what we found.

When he tried to push his branch, he got an error saying that an object file is empty and that the repo is corrupt. We figured that was that, and spent a lot of time exporting his patches, and reimporting them into a freshly cloned repo. When we got his commit history back into shape, we hit `git push` again expecting smooth sailing, but got the same error message:

error: object file .git/objects/20/8529317841b2f25e1213f7e4f6ed3a665b7311 is empty
fatal: loose object 208529317841b2f25e1213f7e4f6ed3a665b7311 (stored in .git/objects/20/8529317841b2f25e1213f7e4f6ed3a665b7311) is corrupt

So the problem likely wasn’t corruption of his local repo. It took me some time to make the mental leap here, though, that the issue was likely in the only other repo involved in the transaction. When that occurred to me, it also occurred to me how we might verify that was the case.

$ cat .git/objects/20/8529317841b2f25e1213f7e4f6ed3a665b7311
f6ed3a665b7311
x+)JMU06`01??d???~???
??f???ih??l
?

Gobbledygook, but the file isn’t empty.

$ git show abcd
tree 208529317841b2f25e1213f7e4f6ed3a665b7311

src/

Its a tree object. Tree objects are immutable binary files in the git repo representing the source structure at some point in the past. Trees point to other trees (directories) or blobs (files). I am not familiar with git’s algorithms, but I am familiar with its data structures, and at this point I think I can mentally fill in the details of what has gone wrong. On the server, the .git/objects files were created, but for whatever reason never had their full data written to them. I imagine some process during pushing walks the pushed commit’s tree object, and when that walk reaches the corrupt tree object it barfs.

The interesting thing to me was that because he created the exact files as the original commit, the tree and blob objects that git stores had the same sha1 hashes, and that prevented him from pushing his branch even though the commit hash differed (since it was made at a different time) and despite it being pushed from a freshly cloned repo.

Wrapping Up
By trivially modifying the files in the commit, mostly adding comments or whitespace, the hashes of the blobs at the leaves were changed. Trees are hashed based on their contents, so those changed as well, everything was rehashed up to the commit’s root tree object. The push walked a path of objects that weren’t corrupt and things completed successfully.

I’m not sure how our IT department can fix the corruption on the server. I tried corrupting a local repo by overwriting a tree object file, and it prevented me from running `git status`, `git gc` or `git fsck`, failing with the same error. Its possible that they could delete the exact .git/objects files mentioned in the error messages we encountered. Possibly there’s a flag so that a push can overwrite the object files on a remote, which would also fix the issue.

I’m not sure I would have figured out what was going wrong or even thought to inspect git’s lower-level structures if I hadn’t read the excellent “Git from the bottom up”. I would highly recommend it, and will try and post back here when our server gets sorted.