git-upload-pack means "I want to fetch.", git-receive-pack means "I want to push".`,
+ },
+ {
+ k: "001e# service=…",
+ d: `A smart-HTTP-only preamble that lets the client confirm it reached a smart server (not a dumb file server), followed by a 0000 flush.`,
+ },
+ {
+ k: "HEAD\\0…caps",
+ d: `Here you can see the capabilities list is just tucked behind a NUL byte after the first ref. This allowed the addition of capabilities without technically changing the protocol, but it's why the capability list was size-limited and awkward to grow.`,
+ },
+ {
+ k: "symref=HEAD:HEAD is symbolic, because there was nowhere else to put this information.`,
+ },
+ {
+ k: "refs/pull/1/head",
+ d: `Finally, notice that it's advertising every reference. On a large repo this can result in a huge amount of data over the wire before anything else.`,
+ },
+ ]}
+/>
+
+---
+
+The second request is a POST body that says what heads we want data for and the response is the packfile and sideband information to allow the client to show progress as it's downloading the streamed body.
+
+---
+
+
+
+9cb163a… (sc-branch-1). The capabilities it actually intends to use ride on the first want line.`,
+ },
+ {
+ k: "0000 then done",
+ d: `A flush ends the want list; done tells the server to stop negotiating and just send a pack. On a fresh clone there are no have lines — the client has nothing yet. On a fetch we would also send the stuff we already have so the server can determine the smallest packfile needed.`,
+ },
+ {
+ k: "NAK",
+ d: `"No common commit found". Because this is a clone. In an incremental fetch this is where ACK <oid> lines would appear as the two sides find shared history.`,
+ },
+ {
+ k: "\\x01 / \\x02",
+ d: `The packfile is multiplexed on a side-band: a leading byte per chunk marks channel 1 (pack bytes), 2 (progress), or 3 (error). This is so the client can do the "Receiving objects…" meter that you see on a clone.`,
+ },
+ ]}
+/>
+
+---
+
++ You may notice that there are a lot of 4-char hexadecimal leaders in these + protocols. This is called a `pkt-line`. It specifies the length of the data + on that line, including the four bytes itself. +
+
+ Because the length is only four hex digits, a single pkt-line tops out at{" "}
+ 65520 bytes (65516 of payload). Anything larger —
+ a whole packfile, say — is simply sent as many pkt-lines in a row.
+
+ v2 adds two special packets — 0001 (delimiter) and 0002 (response-end) — on + top of the existing 0000 flush." +
+fetch here supports shallow, wait-for-done, and filter (partial clone).`,
+ },
+ ]}
+/>
+
+---
+
+OK, so now the client knows what the server can do, including what object format the repository is in (`sha1` or `sha256`).
+
+Now that we know that, we can then start POSTing commands to run. For a clone, we start with `ls-refs`, which is similar to the first GET we did in the previous protocol, but crucially we can filter the returned values.
+
+This is a little funny, because now in v2 the GET to `/info/refs` does not return refs, it returns capabilities. Then a POST to `/git-upload-pack` with a `ls-refs` command is used to actually return the refs we want to inspect (as upload-pack is now just a service endpoint used to run any of several commands).
+
+---
+
+
+
+refs/heads/, refs/tags/, and HEAD. It could also only ask for a single ref.`,
+ },
+ {
+ k: "symref-target:",
+ d: `The symbolic-ref info now has a proper place in the response, instead of being smuggled into a capability string as it was in v0.`,
+ },
+ {
+ k: "0002",
+ d: `The response-end-pkt, so the client can tell the response is complete over a stateless HTTP POST.`,
+ },
+ ]}
+/>
+
+---
+
+Now we have a list of the objects we want and can ask for the packfile with the `fetch` command (basically the only thing `git-upload-pack` did before v2).
+
+---
+
+
+
+acknowledgments section and jumps straight to the pack data`,
+ },
+ {
+ k: "packfile",
+ d: `A named response section. v2 responses are structured into labelled sections (acknowledgments, shallow-info, wanted-refs, packfile); here only the pack is needed.`,
+ },
+ {
+ k: "\\x01 … 0002",
+ d: `The pack is side-band-multiplexed just like the old protocol, then 0002 ends the response.`,
+ },
+ ]}
+/>
+
+---
+
+In this simple example, in practice, there is actually a lot more going over the wire in v2 than in the first version of the protocol. There are 3 round trips consisting of 456 bytes, instead of 2 and 234 bytes.
+
+However, in practical terms, v2 normally saves a lot of traffic because it can filter the references it cares about. Some larger repositories have many megabytes of reference data, so that first refs advertisement is killer overhead.
+
+## What all can a Git server do nowadays?
+
+So this is really mostly what I didn't know. What can a git server _do_ these days outside of listing references and negotiating packfiles? What all commands can you tell it to advertise and allow clients to invoke?
+
+The reality is that out of the box, it's pretty much nothing. Outside of `ls-refs` and `fetch` the vanilla server only advertises `server-option` and `object-format`, which aren't actually _commands_, they're just simple ways to pass other data (to send arbitrary values to custom servers like Gerrit, and tell the client the hash-format).
+
+There _are_ other commands (like the `bundle-uri` and `packfile-uri` I mentioned at the start of the article), but they're all gated behind server (and sometimes also client-side) config options.
+
+Here is a full list of the current potential capabilities of a v2 Git server _(green rows are enabled by default, gray must be configured on)_ and when they were introduced.
+
+--server-option) inside a request, so custom deployments can act on hints without a protocol change. Advertised out of the box, but only useful when both the client passes options and the server understands them.",
+ },
+ {
+ cap: "fetch.shallow",
+ ver: "2.18",
+ yr: "'18",
+ does: "Shallow / --depth clones & deepening",
+ enabled: true,
+ purpose: "Shallow clone",
+ tone: "on",
+ author: "Brandon Williams",
+ sha: "f7e205010542dc9b712473d260058e43ca2b26f7",
+ subject: "fetch-pack: support shallow requests",
+ date: "Mar 15, 2018",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_fetch",
+ thread:
+ "https://lore.kernel.org/git/20180314183213.223440-23-bmwill@google.com/",
+ email: "bmwill@google.com",
+ desc: "Brings shallow clones and history deepening to v2's fetch — --depth, --shallow-since, --deepen — negotiated through the request's shallow arguments and the shallow-info response section. A base fetch feature, advertised by default and exercised by every shallow clone.",
+ },
+ {
+ cap: "fetch.filter",
+ ver: "2.18",
+ yr: "'18",
+ does: "Partial clone (blobless/treeless)",
+ enabled: false,
+ purpose: "Partial clone",
+ tone: "new",
+ author: "Jonathan Tan",
+ sha: "ba95710a3bdcb2a80495b1d93a0e482dd69905e1",
+ subject: "{fetch,upload}-pack: support filter in protocol v2",
+ date: "May 3, 2018",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_fetch",
+ thread:
+ "https://lore.kernel.org/git/cover.1525220786.git.jonathantanmy@google.com/",
+ email: "jonathantanmy@google.com",
+ desc: "Adds partial-clone filter specs to v2's fetch, which was first designed without them — --filter=blob:none and friends. Widely used: blobless and treeless clones, and the engine behind Scalar/VFS-style workflows.",
+ config: [
+ { key: "uploadpack.allowFilter", value: "true", side: "server" },
+ ],
+ },
+ {
+ cap: "fetch.ref-in-want",
+ ver: "2.19",
+ yr: "'18",
+ does: "Request refs by name",
+ enabled: false,
+ purpose: "Server farms",
+ tone: "new",
+ author: "Brandon Williams",
+ sha: "516e2b76bdcf53e757309481fa0e663217ee8039",
+ subject: "upload-pack: implement ref-in-want",
+ date: "Jun 27, 2018",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_fetch",
+ thread:
+ "https://lore.kernel.org/git/20180605175144.4225-1-bmwill@google.com/",
+ email: "bmwill@google.com",
+ desc: "Lets a client name wanted refs instead of object ids, closing the race where a ref changes between the advertisement and the fetch — the kind of thing that happens on busy, load-balanced server farms. Specialized; mostly relevant to large hosts.",
+ config: [
+ { key: "uploadpack.allowRefInWant", value: "true", side: "server" },
+ ],
+ },
+ {
+ cap: "fetch.sideband-all",
+ ver: "2.21",
+ yr: "'19",
+ does: "Multiplex the whole response",
+ enabled: false,
+ purpose: "Progress",
+ tone: "on",
+ author: "Jonathan Tan",
+ sha: "0bbc0bc5745ab8b294a5faf8c3b1d939ae8b6d10",
+ subject: "{fetch,upload}-pack: sideband v2 fetch response",
+ date: "Jan 16, 2019",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_fetch",
+ thread:
+ "https://lore.kernel.org/git/cover.1547244620.git.jonathantanmy@google.com/",
+ email: "jonathantanmy@google.com",
+ desc: "Extends the multiplexed side-band over the entire fetch response, not just the packfile, so the server can send progress and keepalives before the pack begins — a gap the CDN-offload work exposed. Opt-in and niche.",
+ config: [
+ { key: "uploadpack.allowSidebandAll", value: "true", side: "server" },
+ ],
+ },
+ {
+ cap: "object-format",
+ ver: "2.28",
+ yr: "'20",
+ does: "SHA-256 negotiation",
+ enabled: true,
+ purpose: "SHA-256",
+ tone: "on",
+ author: "brian m. carlson",
+ sha: "9de0dd361c9ea2ca6eca14a7dd43fe11d170a253",
+ subject: "serve: advertise object-format capability for protocol v2",
+ date: "May 25, 2020",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_object_format",
+ thread:
+ "https://lore.kernel.org/git/20200619175601.569856-1-sandals@crustytoothpaste.net/",
+ handle: "bk2204",
+ desc: "Negotiates the repository's hash algorithm during the handshake — the wire-level enabler for the SHA-256 transition. Always exchanged in v2, though SHA-256 repos themselves are still rare.",
+ },
+ {
+ cap: "fetch.packfile-uris",
+ ver: "2.28",
+ yr: "'20",
+ does: "CDN offload of pack data",
+ enabled: false,
+ purpose: "CDN offload",
+ tone: "new",
+ author: "Jonathan Tan",
+ sha: "dd4b732df73b06878b81fce050e7dcac4366a38e",
+ subject: "upload-pack: send part of packfile response as uri",
+ date: "Jun 10, 2020",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_fetch",
+ thread:
+ "https://lore.kernel.org/git/cover.1543879256.git.jonathantanmy@google.com/",
+ email: "jonathantanmy@google.com",
+ desc: "Lets the server hand back http(s) URIs for part of the pack so a CDN serves the bulk data, cutting Git-server load. Powerful but rare in the wild, and the in-tree implementation only covers blobs.",
+ config: [
+ {
+ key: "uploadpack.blobPackfileUri",
+ value: "git fetch --negotiate-only. Advertised as a fetch feature; niche.",
+ },
+ {
+ cap: "bundle-uri",
+ ver: "2.40",
+ yr: "'22",
+ does: "Seed a clone from bundle URLs",
+ enabled: false,
+ purpose: "Clone speedup",
+ tone: "new",
+ author: "Ævar Arnfjörð Bjarmason",
+ sha: "8b8d9a229888adb737851c4d7eeaa9a50e37afe1",
+ subject: 'protocol v2: add server-side "bundle-uri" skeleton',
+ date: "Dec 22, 2022",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_bundle_uri",
+ thread:
+ "https://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/",
+ handle: "avar",
+ desc: "Points the client at *.bundle files (often on a CDN) to seed a clone or fetch, so most objects arrive from bundles instead of the Git server — faster clones, less load. Still emerging; used by some large hosts.",
+ config: [
+ {
+ key: "uploadpack.advertiseBundleURIs",
+ value: "true",
+ side: "server",
+ },
+ { key: "transfer.bundleURI", value: "true", side: "client" },
+ ],
+ },
+ {
+ cap: "promisor-remote",
+ ver: "2.49",
+ yr: "'25",
+ does: "Advertise promisor remotes",
+ enabled: false,
+ purpose: "Large objects",
+ tone: "hot",
+ author: "Christian Couder",
+ sha: "d460267613da14eba959eb225e2cbf6a1e132eb1",
+ subject: "Add 'promisor-remote' capability to protocol v2",
+ date: "Feb 18, 2025",
+ doc: "https://git-scm.com/docs/gitprotocol-v2#_promisor_remotepr_info",
+ thread:
+ "https://lore.kernel.org/git/20240731134014.2299361-1-christian.couder@gmail.com/",
+ handle: "chriscool",
+ desc: "Lets the server tell the client which promisor remote it uses, so the client can lazily fetch missing (often large) objects straight from that better-connected source instead of passing -c remote.X… by hand. The newest and least-settled capability.",
+ config: [
+ { key: "promisor.advertise", value: "true", side: "server" },
+ { key: "promisor.acceptFromServer", value: "all", side: "client" },
+ ],
+ },
+ ]}
+/>
+
+Now, there is a lot to potentially dig into here, but what's really interesting is how _old_ some of these capabilities are and how very, very rarely used they are.
+
+Nearly all of them landed over 5 years ago and nearly none of them are available on any major code host. GitHub and GitLab, for example, only enable `fetch.filter` so you can do blobless clones. None of the rest of these capabilities are enabled _(well, GitLab can do bundle-uris on self-hosted instances, I think, but not by default)_.
+
+Honestly, _most_ of this is by and for Google. 11 of the 14 capabilities were contributed by Google and nearly all of the configurable options were also contributed by them. So, most of v2 seems to be so Google can scale Git weirdly internally.
+
+But it's cool.
+
+## See it for yourself
+
+If you want to check any of this out yourself, it's actually pretty easy to see what is going over the wire.
+If you set the GIT_TRACE_PACKET environment variable and run Git networking stuff, it will show you what it's doing.
+
+```small
+$ GIT_TRACE_PACKET=1 git ls-remote https://github.com/schacon/soe
+12:37:45.271670 pkt-line.c:85 packet: git< # service=git-upload-pack
+12:37:45.272656 pkt-line.c:85 packet: git< 0000
+12:37:45.272691 pkt-line.c:85 packet: git< version 2
+12:37:45.272716 pkt-line.c:85 packet: git< agent=git/github-cda1d7094a30-Linux
+12:37:45.272732 pkt-line.c:85 packet: git< ls-refs=unborn
+12:37:45.272747 pkt-line.c:85 packet: git< fetch=shallow wait-for-done filter
+12:37:45.272760 pkt-line.c:85 packet: git< server-option
+12:37:45.272773 pkt-line.c:85 packet: git< object-format=sha1
+12:37:45.272785 pkt-line.c:85 packet: git< 0000
+12:37:45.272956 pkt-line.c:85 packet: ls-remote< version 2
+12:37:45.274390 pkt-line.c:85 packet: ls-remote< agent=git/github-cda1d7094a30-Linux
+12:37:45.274425 pkt-line.c:85 packet: ls-remote< ls-refs=unborn
+12:37:45.274446 pkt-line.c:85 packet: ls-remote< fetch=shallow wait-for-done filter
+12:37:45.274463 pkt-line.c:85 packet: ls-remote< server-option
+12:37:45.274477 pkt-line.c:85 packet: ls-remote< object-format=sha1
+12:37:45.274491 pkt-line.c:85 packet: ls-remote< 0000
+12:37:45.274508 pkt-line.c:85 packet: ls-remote> command=ls-refs
+12:37:45.274552 pkt-line.c:85 packet: ls-remote> agent=git/2.52.0-Darwin
+12:37:45.274576 pkt-line.c:85 packet: ls-remote> object-format=sha1
+12:37:45.274594 pkt-line.c:85 packet: ls-remote> 0001
+12:37:45.274561 pkt-line.c:85 packet: git< command=ls-refs
+12:37:45.274612 pkt-line.c:85 packet: ls-remote> peel
+12:37:45.274656 pkt-line.c:85 packet: git< agent=git/2.52.0-Darwin
+12:37:45.274714 pkt-line.c:85 packet: ls-remote> symrefs
+12:37:45.274744 pkt-line.c:85 packet: git< object-format=sha1
+12:37:45.274779 pkt-line.c:85 packet: ls-remote> unborn
+12:37:45.274819 pkt-line.c:85 packet: git< 0001
+12:37:45.274893 pkt-line.c:85 packet: git< peel
+12:37:45.274919 pkt-line.c:85 packet: git< symrefs
+12:37:45.274860 pkt-line.c:85 packet: ls-remote> 0000
+12:37:45.274968 pkt-line.c:85 packet: git< unborn
+12:37:45.275097 pkt-line.c:85 packet: git< 0000
+12:37:45.490316 pkt-line.c:85 packet: git> 0002
+12:37:45.490332 pkt-line.c:85 packet: ls-remote< ee93348afd51e0d634d9e84a702af190d45cf9e3 HEAD symref-target:refs/heads/main
+12:37:45.490569 pkt-line.c:85 packet: ls-remote< ee93348afd51e0d634d9e84a702af190d45cf9e3 refs/heads/main
+12:37:45.490611 pkt-line.c:85 packet: ls-remote< 9cb163aaa43273f49de0182416d3a621b3ae8292 refs/heads/sc-branch-1
+12:37:45.490646 pkt-line.c:85 packet: ls-remote< 9cb163aaa43273f49de0182416d3a621b3ae8292 refs/pull/1/head
+12:37:45.490671 pkt-line.c:85 packet: ls-remote< 0000
+12:37:45.490874 pkt-line.c:85 packet: ls-remote< 0002
+ee93348afd51e0d634d9e84a702af190d45cf9e3 HEAD
+ee93348afd51e0d634d9e84a702af190d45cf9e3 refs/heads/main
+9cb163aaa43273f49de0182416d3a621b3ae8292 refs/heads/sc-branch-1
+9cb163aaa43273f49de0182416d3a621b3ae8292 refs/pull/1/head
+```
+
+Well, that's it. Hope you learned something, because I certainly did.