ao2.it
Antonio Ospite, theorist attacks

I had been experiencing errors when cloning the OpenEZX kernel with git and I was finally able to solve the issue on the server, I am putting some notes here to save some time to someone else.

Here's how the story started:

ao2@jcn:~$ git clone git://git.openezx.org/openezx.git
Cloning into openezx...
remote: Counting objects: 1954118, done.
remote: fatal: unable to create thread: Resource temporarily unavailable
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed

Looking at the server statistics with htop confirmed this was happening because git was running out of memory on the server; it is worth noticing that the issue was not present when cloning other projects from the same server (Linux kernel projects as well, hence with a comparable size in principle).

With the following points in mind:

• The server hosting the remote repository has about 1.6 GiB of RAM, but no swap space;
• In OpenEZX we are using TopGit to track topic branches, which generates tons of merge commits and makes the git repository heavier;

my thoughts went like:

1. during a clone operation some compression of git objects occurs;
2. because of the lack of swap space the compression operation must fit in memory;
3. because of TopGit the compression operation on this repository was more onerous than on the other projects, causing the memory outage.

So some optimization on the repository about objects compression was needed.

The first thing that came to mind was git-gc but that was failing as well:

openezx:~# cd /home/git/repositories/openezx.git
openezx:/home/git/repositories/openezx.git# git gc --aggressive
Counting objects: 1954118, done.
Delta compression using up to 2 threads.
warning: suboptimal pack - out of memory02)   
fatal: Out of memory, malloc failed (tried to allocate 602234 bytes)
error: failed to run repack

The repack operation is performed by git-pack-objects, so after Reading The Fine Manual I was able to make git-gc run successfully using these steps:

openezx:/home/git/repositories/openezx.git# git config pack.threads 1
openezx:/home/git/repositories/openezx.git# git gc --aggressive
Counting objects: 1954118, done.
Compressing objects: 100% (1936802/1936802), done.
Writing objects: 100% (1954118/1954118), done.
Total 1954118 (delta 1618716), reused 0 (delta 0)

note that using only one thread makes the process quite slow.

In some other cases it could be useful to change also the pack.windowMemory config option, but a relation involving the address space, the available memory and the number of threads has to be found, so since just using one thread worked fine for me, I took that as the minimal fix for my problem.

Bottom line: cloning the repository is now back working:

ao2@jcn:~$ git clone git://git.openezx.org/openezx.git
Cloning into openezx...
remote: Counting objects: 1954118, done.
remote: Compressing objects: 100% (318086/318086), done.
remote: Total 1954118 (delta 1618716), reused 1954118 (delta 1618716)
Receiving objects: 100% (1954118/1954118), 377.38 MiB | 738 KiB/s, done.
Resolving deltas: 100% (1618716/1618716), done.

Side note: the more attentive reader would have noticed a cosmetic defect in the output when git-gc fails (more precisely from git-pack-objects: warning: suboptimal pack - out of memory02)), well don't loose any sleep over it :) it has been reported and there is a workaround for that already.