On git clone failing and tweaking object repack

I had been experiencing errors when cloning the OpenEZX kernel with git and I was finally able to solve the issue on the server, I am putting some notes here to save some time to someone else.

Here's how the story started:

ao2@jcn:~$ git clone git://git.openezx.org/openezx.git
Cloning into openezx...
remote: Counting objects: 1954118, done.
remote: fatal: unable to create thread: Resource temporarily unavailable
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed

Looking at the server statistics with htop confirmed this was happening because git was running out of memory on the server; it is worth noticing that the issue was not present when cloning other projects from the same server (Linux kernel projects as well, hence with a comparable size in principle).

With the following points in mind:

  • The server hosting the remote repository has about 1.6 GiB of RAM, but no swap space;
  • In OpenEZX we are using TopGit to track topic branches, which generates tons of merge commits and makes the git repository heavier;

my thoughts went like:

  1. during a clone operation some compression of git objects occurs;
  2. because of the lack of swap space the compression operation must fit in memory;
  3. because of TopGit the compression operation on this repository was more onerous than on the other projects, causing the memory outage.

So some optimization on the repository about objects compression was needed.

The first thing that came to mind was git-gc but that was failing as well:

openezx:~# cd /home/git/repositories/openezx.git
openezx:/home/git/repositories/openezx.git# git gc --aggressive
Counting objects: 1954118, done.
Delta compression using up to 2 threads.
warning: suboptimal pack - out of memory02)   
fatal: Out of memory, malloc failed (tried to allocate 602234 bytes)
error: failed to run repack

The repack operation is performed by git-pack-objects, so after Reading The Fine Manual I was able to make git-gc run successfully using these steps:

openezx:/home/git/repositories/openezx.git# git config pack.threads 1
openezx:/home/git/repositories/openezx.git# git gc --aggressive
Counting objects: 1954118, done.
Compressing objects: 100% (1936802/1936802), done.
Writing objects: 100% (1954118/1954118), done.
Total 1954118 (delta 1618716), reused 0 (delta 0)

note that using only one thread makes the process quite slow.

In some other cases it could be useful to change also the pack.windowMemory config option, but a relation involving the address space, the available memory and the number of threads has to be found, so since just using one thread worked fine for me, I took that as the minimal fix for my problem.

Bottom line: cloning the repository is now back working:

ao2@jcn:~$ git clone git://git.openezx.org/openezx.git
Cloning into openezx...
remote: Counting objects: 1954118, done.
remote: Compressing objects: 100% (318086/318086), done.
remote: Total 1954118 (delta 1618716), reused 1954118 (delta 1618716)
Receiving objects: 100% (1954118/1954118), 377.38 MiB | 738 KiB/s, done.
Resolving deltas: 100% (1618716/1618716), done.

Side note: the more attentive reader would have noticed a cosmetic defect in the output when git-gc fails (more precisely from git-pack-objects: warning: suboptimal pack - out of memory02)), well don't loose any sleep over it :) it has been reported and there is a workaround for that already.


CommentsSyndicate content

I was having the same issue

Francis Lewis's picture

I was having the same issue running git gc, got the threads changed to 1 and where it was taking over 4 hours, then failing before, it took just 4 minutes and succeeded.

Also, be sure to use "pack.threads", not "pack.thread" - pack.thread is not a valid config variable.

Thanks Francis, I fixed the

ao2's picture

Thanks Francis, I fixed the typo in the article.

Thank you very much for your

Antonio's picture

Thank you very much for your tip. I was having almost the same problem, just changed the number of threads (two in my case), and I was able to pack and clone.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account associated with the e-mail address you provide, it will be used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
M
i
D
Z
V
H
Enter the code without spaces.