Anthropic Says Naive Git Costs 600 PB a Day. Stock Git Cut a 1.1 GB Clone to 3.9 MB

Correction, 2026-10-14 (same day). The first version of this post reported the working directory as 154.5 MiB and 11,395 files. That was wrong. On Windows, my clones stopped checking out at 17% with a “Filename too long” error, because this repository has paths past the 260-character limit, and my script did not notice. The real working directory is 350.4 MiB and 63,662 files. I reran everything with core.longpaths enabled and added a check that fails loudly. The object store sizes did not change (for example 50.4 MiB for the shallow clone in both runs), so the size comparisons stand, but the times and every number about the working directory are new, and the repository had moved to a newer commit (343,502 commits).

On 13 October 2026, Anthropic published an engineering post about leaving git. Merges to its monorepo had grown more than tenfold, from around 2,500 code changes a day in May to over 30,000 a day by September, and source control had become the biggest bottleneck. They built an in-house system called Scion and moved the whole repository over in five months.¹

One sentence in the post makes the case in numbers. A compressed working copy of the repository is over 25 GB, CI performs about 25 million checkouts a day, and “fetched naively, that would be over 600 PB of git traffic per day.”¹ The arithmetic holds: 25 million times 25 GB is 625 PB. What I wanted to know is what “naively” leaves out. The same post says git already has tools for working with a slice of a repository, namely shallow clones, sparse checkouts and promisor remotes, and says Anthropic used “all of them,” internal mirrors and repository snapshots included, before building Scion.¹ So how far do those tools actually get you? I measured it on a big public repository.

A few git terms, quickly

A repository stores three kinds of things. Commits are snapshots with a message and a parent. Trees are directory listings. Blobs are the contents of files. A normal clone downloads all of it, for every commit in history.

Three features cut that down, and they cut different things:

• A shallow clone (--depth 1) keeps only the latest commit and drops history.²
• A partial clone (--filter=blob:none) keeps the full history but leaves out file contents until something needs them. The server that can supply the missing pieces later is called a promisor remote.² A stricter filter, tree:0, leaves out the trees as well.³
• A sparse checkout limits which files appear in your working directory.⁴

The documentation describes sparse checkout in terms of the working tree only. It does not say whether it reduces what you download. I measured that too.

flowchart LR
  A["Full clone<br/>1,108.9 MiB<br/>343,502 commits"] -->|"drop history<br/>--depth 1"| B["Shallow<br/>50.4 MiB"]
  B -->|"limit working dir<br/>sparse-checkout"| C["Shallow + sparse<br/>50.4 MiB downloaded<br/>8.4 MiB on disk"]
  B -->|"also skip blobs<br/>--filter=blob:none"| D["Shallow + blobless + sparse<br/>3.9 MiB"]

The experiment

I used rust-lang/rust,⁵ a repository with 343,502 commits and, at the tip, 63,662 files taking 350.4 MiB of working directory. It is nowhere near Anthropic’s scale, and I come back to that below. For each strategy I cloned into a fresh directory, did a checkout, and then measured the size of the object store (.git/objects), the working directory, and the wall time.⁶ The last three strategies add a sparse checkout of one directory, library/core, which stands in for a CI job that builds one component.

What the numbers say

History is most of the size. A shallow clone is 50.4 MiB against 1,108.9 MiB for the full clone, about 22 times smaller, and it still delivers all 63,662 files of the tip. The working directory is 350.4 MiB uncompressed, so the shallow clone is about a seventh of that, which is what you would expect from a compressed copy of mostly text. That is the quantity Anthropic describes when it talks about a compressed working copy.¹ I am inferring that their “naive” figure means each checkout downloads about that much, not a full clone, because 25 million full clones of a history close to 100 GB would come to more than 2,000 PB.

Sparse checkout alone does not reduce the download. Shallow plus sparse downloaded 50.4 MiB, exactly what plain shallow did. It reduced the working directory from 350.4 MiB and 63,662 files to 8.4 MiB and 356 files, and that is all. If you hope a sparse checkout will save bandwidth, it will not, unless a partial clone is also asking the server to skip the files you do not need.

The combination is where it gets small. Shallow, blobless and sparse together downloaded 3.9 MiB in 4 seconds. That is about 284 times less than the full clone and about 13 times less than the shallow clone. The checkout fetched only the file contents inside library/core, which left 356 files, 0.6% of the 63,662 at the tip.

Most of a shallow clone’s time is writing files. The shallow clone took 113 seconds and the shallow clone with sparse checkout took 15, though both downloaded 50.4 MiB. The difference is creating 63,662 files instead of 356. That is a Windows measurement, where antivirus scanning can make file creation slow, so treat the size of the gap as specific to this machine.

Keeping history is expensive in a different way. Blobless with full history is 525.1 MiB, and with sparse it is still 473.8 MiB. The repository has 343,502 commits, and git still has to send the commits and the trees that describe every one of them. Treeless drops that to 219.1 MiB, at a cost GitHub’s own guidance spells out: commands that need history then have to fetch trees on demand, and it recommends treeless clones for automated builds, not for daily developer work.⁷

Staying up to date

A fresh clone is one cost. Most CI jobs also update an existing copy. I started from the tag 1.85.0, then fetched the current tip.

In both cases the update cost about as much as starting over: 45.1 MiB against 50.4 MiB for a fresh shallow clone, and 4.1 MiB against 3.9 MiB for a fresh minimal one. GitHub’s guide makes the same warning about fetching into a shallow clone, which can pull an almost-full history.⁷ One caveat matters here. Tag 1.85.0 is far behind the tip, so this is closer to a worst case than to a nightly refresh. I did not measure a daily update.

How far does this go at Anthropic’s scale?

As a back-of-envelope illustration, not a prediction: if a CI job at Anthropic needed the same fraction of its tree that my library/core job needs (3.9 MiB out of the 50.4 MiB a shallow clone costs, 7.7%), 625 PB a day would become roughly 48 PB a day. Their jobs may need much more of the tree, and a repository of 25 GB compressed and almost 100 GB of history may not scale like mine.

But even the optimistic number is large, and it exposes the part my measurements cannot reach. Stock git sends every one of those bytes from a server. Anthropic’s post describes tiered caches in front of that server, each with a hit rate over 99%, plus a design in which each unit of work is a server-side changeset and clients fetch only what they need on demand.¹ That is a claim about CPU, memory and caching, and it is outside what a single git clone can show. They also report that their new infrastructure is 200 times smaller than their previous mirroring setup and that CI checkouts are two orders of magnitude faster, without publishing methods or raw data.¹ I could not check those, and I am not suggesting they are wrong. The measurement here only says that stock git narrows the transfer a lot, and that the remaining problem is serving it at that volume.

Where this can mislead

One repository, far smaller than theirs. rust-lang/rust has about 1.1 GB of git data. Anthropic’s history alone is almost 100 GB. Behaviors like negotiation, pack generation and server load may not scale linearly, and I did not test that.

One server. Everything came from GitHub’s servers, which have their own caching and tuning. A self-hosted git server may behave differently, and partial clone has to be supported by the server in the first place.²

Sizes are on disk, not on the wire. I measured the object store, which includes index files git builds locally. For the full clone, the progress output reported about 1,002 MiB received while the object store was 1,108.9 MiB, so the table overstates bytes on the wire by roughly ten percent for large clones. The ratios are what to trust.

Times are single runs over my home connection. They show the order of magnitude, not a benchmark. The sizes are deterministic apart from the repository moving between runs.

The sparse job is my choice. I picked library/core. A job that needs a third of the tree would see a much smaller gain.

I did not test partial-clone side effects. After a blobless clone, later commands such as git blame can trigger more downloads, which the table does not include.⁷

Takeaways

• The 600 PB figure describes naive checkouts, and git’s own slicing features change the picture a lot. In my test, shallow, partial and sparse together cut one checkout from 1,108.9 MiB to 3.9 MiB, and from 327 seconds to 4.
• Sparse checkout is not a bandwidth feature. It shrank the working directory by 98% and the download by 0%. Pair it with --filter=blob:none if you want the savings.
• Shallow clones are good for throwaway CI, poor for staying current. An update into a shallow clone cost almost as much as a fresh one.
• The hard part moves to the server. If hundreds of thousands of jobs ask for the small checkout, the remaining cost is serving and caching, which is the problem Anthropic’s post says Scion addresses. I did not measure that part.
• The experiment is cheap to repeat on your own repository. The script takes a repository URL and a directory, so you can see your own numbers before you decide anything.

References