gitindex: use --unordered for git cat-file

[keegancsmith]

UNTESTED! Will follow up next week.

We disabled cat-file due to memory issues. After some research I suspect that passing --unordered may work even though we don't do --batch-all-objects.

So in theory this will speed cat-file up. But the other theory is this makes better use of the delta base cache.

[keegancsmith]

Full details below from deep search

Details

---

title: In github.com/sourcegraph/zoekt we use "git cat-file --batch --buffer" on all blob objects for a commit. However, in production we are now seeing huge memory usage leading too OOMs. In particular it is the git process taking up so much memory. Can you inspect github.com/git/git to see why that may be the case. Could it be due to --buffer?
author: keegan
share_url: https://sourcegraph.sourcegraph.com/deepsearch/eb46bf6f-4cf5-433f-a576-116b7cb9e336
created: 2026-04-02T19:20:41Z
updated: 2026-04-02T19:28:48Z

In github.com/sourcegraph/zoekt we use "git cat-file --batch --buffer" on all blob objects for a commit. However, in production we are now seeing huge memory usage leading too OOMs. In particular it is the git process taking up so much memory. Can you inspect github.com/git/git to see why that may be the case. Could it be due to --buffer?

Based on my investigation of the git/git repository, I've found the key issue. Let me provide you with my findings:

The Issue with --buffer

When using git cat-file --batch --buffer, the implementation uses fwrite() instead of write() for output. This is the critical difference:

Code Analysis

In builtin/cat-file.c, the batch_write() function shows:

static void batch_write(struct batch_options *opt, const void *data, int len)
{
    if (opt->buffer_output) {
        if (fwrite(data, 1, len, stdout) != len)
            die_errno("unable to write to stdout");
    } else
        write_or_die(1, data, len);
}

The Memory Problem

The --buffer flag was added in commit fc4937c to improve performance by using stdio buffering instead of making individual write() syscalls. However, this introduces a memory issue:

fwrite() uses stdio's internal buffers maintained by libc (typically glibc). When you write large amounts of data through fwrite():

1. Stdio allocates internal buffers - glibc maintains its own buffer for the FILE* stream (stdout)
2. Buffer growth - These buffers can grow dynamically as more data is written
3. fflush() doesn't free memory - While fflush() is called before streaming blobs, it only writes buffered data to the OS - it doesn't deallocate the stdio buffer
4. Memory persists - The stdio buffers remain allocated until fclose() (or process exit)

Why Git Process Uses Memory

The problem isn't Git's object handling code - Git streams blobs efficiently using a 16KB buffer.

The issue is that libc's stdio maintains large internal buffers when using fwrite(). When processing many blob objects in batch mode with --buffer:

• Each fwrite() call for object metadata uses the stdio buffer
• The stdio buffer can grow to accommodate large writes
• Even with fflush(), this memory is never freed until the process exits
• With thousands/millions of objects, this accumulated buffer space causes high memory usage

Solution

Remove --buffer from your command. Without it:

• Git uses write() syscalls directly (via write_or_die())
• No stdio buffering overhead
• Slightly more syscall overhead, but vastly better memory behavior

The documentation in Documentation/git-cat-file.adoc states that --buffer uses "normal stdio buffering" for efficiency with large numbers of objects, but it doesn't mention the memory cost of this approach.

For your use case at Sourcegraph (zoekt), you should avoid --buffer to prevent the OOM issues you're experiencing.

Sources

• repo:^github.com/git/git$ cat-file batch buffer
• repo:^github.com/git/git$ file:builtin/cat-file.c --buffer
• repo:^github.com/git/git$ file:builtin/cat-file.c batch_options
• github.com/git/git/builtin/cat-file.c@2565546
• repo:^github.com/git/git$ file:builtin/cat-file.c fwrite fflush stdout
• repo:^github.com/git/git$ stdout buffer setvbuf setbuffer
• type:commit (message:"cat-file --buffer" OR message:"cat-file buffer memory") repo:github.com/git/git count:20 timeout:60s
• type:commit content:buffer_output repo:github.com/git/git count:30 timeout:60s
• repo:^github.com/git/git$ fwrite write_or_die batch_write
• type:commit (message:--buffer OR message:"cat-file buffer") repo:github.com/git/git count:30 timeout:60s
• repo:^github.com/git/git$ file:Documentation/git-cat-file.adoc --buffer
• github.com/git/git/Documentation/git-cat-file.adoc@2565546?L188-200
• type:commit message:"cat-file: add --buffer option" repo:github.com/git/git count:5 timeout:60s
• github.com/git/git (diff fc4937c37219347f4e2c25a271577b333942453f~1...fc4937c37219347f4e2c25a271577b333942453f)
• repo:^github.com/git/git$ stdio buffer fwrite stream blob
• repo:^github.com/git/git$ file:builtin/cat-file.c stream_blob fflush
• repo:^github.com/git/git$ odb_stream_blob_to_fd
• github.com/git/git/odb/streaming.c@2565546?L230-270
• repo:^github.com/git/git$ file:builtin/cat-file.c print_object_or_die batch_write
• type:commit (message:fwrite OR message:"stdio buffer") repo:github.com/git/git count:20 timeout:60s
• repo:^github.com/git/git$ file:builtin/cat-file.c fflush fwrite

---

I find it surprising that glibc would be so poor here. Is it not more likely the issue is related to opening so many objects from a packfile?

Perfect! Now I can give you the real answer:

The Actual Problem: Delta Base Cache Thrashing

You're absolutely right to be skeptical about glibc's stdio. The real issue is not stdio buffering - it's the delta base cache in Git.

How the Delta Base Cache Works

When Git unpacks deltified objects from a packfile, it caches the base objects in memory to speed up delta resolution. See packfile.c:1971-1973:

add_delta_base_cache(p, base_obj_offset, base, base_size,
                     p->repo->settings.delta_base_cache_limit,
                     type);

The default limit is 96 MB (DEFAULT_DELTA_BASE_CACHE_LIMIT).

The Problem with Hash Order

Looking at zoekt's code, you're using:

args := []string{"cat-file", "--batch", "--buffer"}

Without --unordered, Git iterates objects in hash (OID) order, which is effectively random with respect to pack file layout. This is terrible for the delta base cache because:

1. Deltified objects in packfiles are stored next to their bases
2. When you access in hash order, you're jumping around the packfile randomly
3. Each deltified object needs its base unpacked and cached
4. The cache limit (96MB) is hit constantly, causing cache thrashing
5. Bases are evicted and then needed again shortly after
6. The 96MB cache fills up and Git keeps the process memory high

Why --buffer Makes it Worse

The --buffer flag doesn't directly cause the problem, but it may correlate because:

1. Without buffering, each write() syscall provides a natural backpressure point
2. With --buffer, the pipeline flows faster, potentially causing more delta resolutions to happen in a burst
3. More concurrent delta unpacking = more cache pressure

The Solution

Add --unordered to your cat-file command:

args := []string{"cat-file", "--batch", "--buffer", "--unordered"}

From commit 0750bb5:

If you're going to access the contents of every object in a packfile, it's generally much more efficient to do so in pack order, rather than in hash order. That increases the locality of access within the packfile, which in turn is friendlier to the delta base cache, since the packfile puts related deltas next to each other.

With --unordered, Git iterates in pack order (ODB_FOR_EACH_OBJECT_PACK_ORDER), accessing objects sequentially as they appear in the packfile. This means:

• Bases are unpacked right before their deltas need them
• Cache locality is maximized
• Much less cache thrashing
• Significantly lower memory usage

You can keep --buffer for performance - the real fix is --unordered.

Sources

• repo:^github.com/git/git$ delta base cache memory
• repo:^github.com/git/git$ delta_base_cache packed_git window
• github.com/git/git/packfile.c@2565546?L1424-1550
• repo:^github.com/git/git$ file:packfile.c add_delta_base_cache delta_base_cached
• github.com/git/git/packfile.c@2565546?L1543-1600
• repo:^github.com/git/git$ file:pack-objects.h DEFAULT_DELTA_BASE_CACHE_LIMIT
• repo:^github.com/git/git$ file:packfile.c unpack_entry add_delta_base_cache
• github.com/git/git/packfile.c@2565546?L1700-1850
• github.com/git/git/packfile.c@2565546?L1850-1950
• github.com/git/git/packfile.c@2565546?L1950-2010
• type:commit (message:"delta base cache" OR message:"clear cache") content:cat-file repo:github.com/git/git count:20 timeout:60s
• github.com/git/git (diff 0750bb5b51f021ecad6f33b7ec88cdfc2a8cdff4~1...0750bb5b51f021ecad6f33b7ec88cdfc2a8cdff4)
• repo:^github.com/git/git$ FOR_EACH_OBJECT_PACK_ORDER
• github.com/git/git/packfile.h@2565546?L1-60
• github.com/git/git/packfile.c@2565546?L2300-2350
• repo:^github.com/sourcegraph/zoekt$ cat-file --batch

[keegancsmith]

cc @clemlesne

[keegancsmith]

This made no difference to actual RSS usage in git.

❯ ZOEKT_BENCH_REPO=../../../kubernetes/kubernetes/ go test -bench=. -run='^$'
goos: darwin
goarch: arm64
pkg: github.com/sourcegraph/zoekt/gitindex
cpu: Apple M2 Max
BenchmarkBlobRead_GoGit/files=1000-12       2030            501196 ns/op           8918205 content-bytes/op           1000 files/op       148037 B/op       2020 allocs/op
BenchmarkBlobRead_GoGit/files=5000-12        242           4181960 ns/op          46009709 content-bytes/op           5000 files/op      1381668 B/op      10519 allocs/op
BenchmarkBlobRead_GoGit/files=29226-12                 1        2576134000 ns/op         262191758 content-bytes/op          29226 files/op     920401208 B/op    767920 allocs/op
BenchmarkBlobRead_CatfileReader/files=1000/mode=ordered-12                    27          43134375 ns/op           8829187 content-bytes/op           1000 files/op       72822025 git-maxrss-bytes/op   9922541 B/op       2083 allocs/op
BenchmarkBlobRead_CatfileReader/files=1000/mode=unordered-12                  26          43051803 ns/op           8829187 content-bytes/op           1000 files/op       72796633 git-maxrss-bytes/op   9922816 B/op       2085 allocs/op
BenchmarkBlobRead_CatfileReader/files=5000/mode=ordered-12                     8         139615344 ns/op          49518018 content-bytes/op           5000 files/op      151400448 git-maxrss-bytes/op  52493006 B/op      10080 allocs/op
BenchmarkBlobRead_CatfileReader/files=5000/mode=unordered-12                   8         136448391 ns/op          49518018 content-bytes/op           5000 files/op      151230464 git-maxrss-bytes/op  52494963 B/op      10081 allocs/op
BenchmarkBlobRead_CatfileReader/files=29226/mode=ordered-12                    2         598792584 ns/op         262191758 content-bytes/op          29226 files/op      245325824 git-maxrss-bytes/op  276515704 B/op     58495 allocs/op
BenchmarkBlobRead_CatfileReader/files=29226/mode=unordered-12                  2         600316521 ns/op         262191758 content-bytes/op          29226 files/op      245415936 git-maxrss-bytes/op  276516104 B/op     58497 allocs/op