I Packed the Same Commit Twice, 200 Bytes Didn't Match

There is a single line in this blog's deploy workflow that every publication
passes through:

tar -czf - dist-deploy scripts/notify-subscribers.mjs package.json \ package-lock.json deploy/backup.sh | "$GATE" deploy 

The build finishes, the output flows into a pipe, it is unpacked on the server
and swapped in atomically. One line above it sits this: git rev-parse HEAD >
dist-deploy/client/_v. So I can ask the server "which commit are you running"
and get the right answer back.

Two days ago I wrote about
1,190 of my cover images being impossible to reproduce:
there the problem was generation itself — two machines, two font engines, a
different PNG from the same input. This article is the next stop on that road.
Generation aside, is the step that packages the output deterministic? The
_v file tells me the commit but says nothing about the bytes. If I packed
the same commit twice, would I get the same archive twice?

For years I had assumed the answer was obviously yes. I sat down and measured
it: no. Across three separate layers, and only one of them was my pipeline's
actual problem. The odd part: the first place I noticed the trouble was my own
Mac, but the cause there never happens in CI.

Layer one: gzip writes the clock into the archive

I started with the easiest experiment. I put the bundle's four non-dist-deploy
members — package.json, package-lock.json,
scripts/notify-subscribers.mjs, deploy/backup.sh, 399,720 bytes in total —
into one directory and packed it twice, two seconds apart. I didn't touch the
files and I didn't change the directory.

tar -czf a1.tgz -C src . sleep 2 tar -czf a2.tgz -C src . shasum -a 256 a1.tgz a2.tgz # 79881f54e8331048d53c29685f61f470576fdbf89cf81a6d9128b55ac209caa9 a1.tgz # ac4560028820aa02386b09e531695d36efe2f23ff934be6bd1493234f2176d7d a2.tgz 

Two different hashes. The sizes, however, are identical: 94,516 bytes each.
Same length, different content — so the difference isn't in the data, it's in
some fixed-width field. cmp named the exact spot:

a1.tgz a2.tgz differ: char 5, line 1 

The fifth byte. RFC 1952, the
definition of the gzip file format, lays out the first ten header bytes like
this: ID1, ID2, CM, FLG, then a four-byte MTIME, then XFL and OS.
That four-byte MTIME sits at offsets 4 through 7 — the fifth byte onward.
Let's read it:

od -An -tu4 -j4 -N4 a1.tgz # 1793713115 od -An -tu4 -j4 -N4 a2.tgz # 1793713117 date -r 1793713115 # 2026-11-03 16:38:35 +03 date -r 1793713117 # 2026-11-03 16:38:37 +03 

Two seconds. My own sleep 2, written into the archive's header. The RFC calls
this field "the most recent modification time of the original file being
compressed," then adds the critical sentence: if the compressed data did not
come from a file, MTIME is set to the time at which compression started. My
data comes from a pipe. So the clock of the moment gets stamped into my archive
every single run.

Separating compression from tar pinned the guilty layer:

tar -cf raw1.tar -C src . tar -cf raw2.tar -C src . shasum -a 256 raw1.tar raw2.tar # f73fc83e0f9a0a31941e4991a7f38397cd9332d22a2ef1c3256e5a936c4f7f11 raw1.tar # f73fc83e0f9a0a31941e4991a7f38397cd9332d22a2ef1c3256e5a936c4f7f11 raw2.tar 

The raw tar is byte-identical. gzip's -n flag exists for exactly this; the
gzip manual describes it as "do not
save the original file name and timestamp" when compressing:

tar -cf - -C src . | gzip -n -9 > n1.tgz sleep 2 tar -cf - -C src . | gzip -n -9 > n2.tgz cmp n1.tgz n2.tgz && echo SAME # SAME xxd -l 10 -g 1 n1.tgz # 1f 8b 08 00 00 00 00 00 02 03 

Four zeros in the MTIME field. The last two header bytes are meaningful too:
per the RFC, XFL=2 means "compressor used maximum compression, slowest
algorithm" — it doesn't carry the number 9, it's a two-state flag. OS=03
means "Unix".

There's a subtlety here that I missed on the first pass: the broken runs used
tar -czf, that is libarchive's own gzip filter, while I demonstrated the
fix with a separate gzip on the end of a pipe. Two different compressors.
For anyone who wants to keep using tar -czf, the equivalent of -n is
somewhere else:

tar --options 'gzip:!timestamp' -czf z1.tgz -C src . sleep 2 tar --options 'gzip:!timestamp' -czf z2.tgz -C src . cmp z1.tgz z2.tgz && echo SAME # SAME 

Layer one closed — or so I thought.

Layer two: CI doesn't work in the same directory

The experiment above had an unfair advantage: I did both packings on the same
files. That isn't how my pipeline actually works. The first step of the deploy
workflow is this:

git init . git fetch --no-tags --prune --depth=1 origin "+refs/heads/main:refs/remotes/origin/main" git checkout --force -B main refs/remotes/origin/main 

Every run is a checkout from scratch — I had already measured
what a --depth=1 clone really downloads,
but there was a side effect I didn't look at that day. Git doesn't store file
modification times; checkout gives every file it writes the clock of that
moment. An earlier measurement here looked at the same field from the other
side: git scanning 36,811 index entries.
There mtime was git's friend, the thing it read cache validity from. Here it
becomes the enemy.

I checked out the same commit twice, three seconds apart, then extracted the
bundle's four members while preserving their timestamps:

git checkout-index -a --prefix=co1/ # the whole repository sleep 3 git checkout-index -a --prefix=co2/ for d in co1 co2; do mkdir -p $d/sel/scripts $d/sel/deploy cp -p $d/package.json $d/package-lock.json $d/sel/ cp -p $d/scripts/notify-subscribers.mjs $d/sel/scripts/ cp -p $d/deploy/backup.sh $d/sel/deploy/ done stat -f '%Sm' co1/sel/package.json # 2026-11-03 16:39:13 stat -f '%Sm' co2/sel/package.json # 2026-11-03 16:39:17 

That -p isn't decoration: my first attempt went through a plain cp and all
four files picked up the copy time as their mtime — I had erased the very thing
I was trying to measure while setting the experiment up. A two-minute mistake,
but if I hadn't caught it the whole section would have reached the wrong
conclusion.

Same commit, same content, mtimes four seconds apart. Now pack with gzip -n —
the gzip header is clean this time, but:

tar -cf - -C co1/sel . | gzip -n -9 > co1.tgz tar -cf - -C co2/sel . | gzip -n -9 > co2.tgz cmp co1.tgz co2.tgz || echo DIFFERENT # DIFFERENT 

Still different. Let's count the differing bytes in the raw tars:

tar -cf e1.tar -C co1/sel . ; tar -cf e2.tar -C co2/sel . cmp -l e1.tar e2.tar | wc -l # 200 stat -f%z e1.tar # 419840 

200 of 419,840 bytes didn't match. My first explanation was this: each member's
512-byte header carries a 12-byte octal mtime field starting at offset 136 and
an 8-byte checksum at 148; multiply by the number of members. That arithmetic
doesn't work — tar -tf shows seven members in this archive, and seven times
twenty is a hundred and forty, not two hundred. So I mapped where the
differences actually land:

 26 bytes in ordinary member headers (mtime field + checksum) 174 bytes in extended pax headers 

The majority was in a place I never thought to look. macOS's tar writes a pax
header for each member too, and records the time there at nanosecond
resolution:

e1: 30 mtime=1793713158.104912441 e2: 30 mtime=1793713158.112318797 

The seconds are identical — 1793713158 in both. Only the nanoseconds differ. In
other words, even if the two checkouts had landed in the same second, the
archives still would not have matched. On top of that, when the digit count
shifts (29 instead of 30), the pax record's length prefix changes, and with
it the pax header's own size field and checksum:

e1: 00000000210 15272362006 016223 (body 136 bytes) e2: 00000000207 15272362006 016231 (body 135 bytes) 

A single nanosecond touches three separate fields. Meanwhile, looking at
e1.tar's first member, I saw something I wasn't expecting.

An aside: the seven members my Mac slipped into the archive

I read the first twenty bytes of e1.tar and a file name came back: ._.
Yet tar -tf wasn't showing me any such member:

tar -tf e1.tar | wc -l # 7 

Seven members: four files, two directories, and the root. Clean. But I handed
the same archive to Python's tarfile module:

python3 -c "import tarfile; print(len(tarfile.open('e1.tar').getnames()))" # 14 

Fourteen. There are seven more members in that archive and macOS's tar isn't
listing them for me. Each is 163 bytes:

._. ./._package-lock.json ./._deploy ./._package.json ./._scripts ./scripts/._notify-subscribers.mjs ./deploy/._backup.sh 

macOS's tar writes files' extended attributes as AppleDouble members, then
quietly reassembles them on read. A tool that doesn't show you what it wrote
can run unnoticed for years.

When I unpack that archive on my server, there is nobody to reassemble them:

$ tar -tf /tmp/e1.tar ._. ./ ./._deploy ./deploy/ ... tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance' ... (seven times) $ tar -xf /tmp/e1.tar && find . -name '._*' | wc -l 7 

Seven warnings, seven junk files. With --no-mac-metadata those members are
never written and the archive drops from 419,840 to 412,672 bytes — exactly
7,168 bytes, which is fourteen 512-byte blocks. My deploy doesn't pack from a
Mac; it packs from a runner in a Linux container, so this particular foot never
stepped on my server. But if I'm ever the person saying "CI is stuck, let me
deploy by hand," it's something I'd want to know.

The same Mac had one more surprise for me:

$ tar --sort=name -cf /dev/null -C co1/sel . tar: Option --sort=name is not supported $ tar --mtime=@0 -cf /dev/null -C co1/sel . tar: Option --mtime=@0 is not supported 

On macOS, tar is not GNU tar — it is libarchive's bsdtar. There's a fine
distinction here: the
libarchive manual does document
--mtime date as "(c, r, u modes only) Set the modification times of added
files to the specified date"; --sort genuinely isn't in the manual at all. So
the second rejection isn't a decision by the tool family, it's a version
matter: the CLI Apple ships is bsdtar 3.5.3, and --mtime landed in
libarchive later. Either way, I cannot express the fix on this machine. I
moved the rest of the measurement to where my pipeline actually runs.

Layer three: GNU tar breaks somewhere else

My own server has GNU tar 1.35 and gzip 1.12. I rebuilt the same setup there
with the same four files (the directories are again called co1/co2, but the
machine is different). The first measurement inverted the story from the Mac:

$ ( cd co1 && tar -czf - . ) > srv-a1.tgz $ od -An -tu4 -j4 -N4 srv-a1.tgz 0 

GNU tar's built-in gzip doesn't write the timestamp at all. Neither does a bare
gzip -9 on the end of a pipe. So the problem I described as layer one simply
does not exist in my CI; adding -n there fixes nothing. Apple gzip 479 on
the Mac, by contrast, writes the clock for data it reads from a pipe — two
different headers in two runs. Two programs implementing the same document
produce different headers for the same input; I won't claim which one reads it
"correctly," but for anyone who builds the same bundle on two machines, that
isn't a trade-off, it's a trap.

So what remains on Linux? Exactly layer two — the checkout mtime:

$ ( cd co1 && tar -cf - . ) | gzip -n -9 > srv-b1.tgz $ ( cd co2 && tar -cf - . ) | gzip -n -9 > srv-b2.tgz $ cmp -l srv-b1.tgz srv-b2.tgz | wc -l 91206 

91,206 bytes. A twelve-byte mtime difference diverges early in the compressed
stream and shifts everything after it, which is why the number is so large.

My first reflex was SOURCE_DATE_EPOCH — the shared variable of the
reproducible-builds world. I tried it:

( cd co1 && SOURCE_DATE_EPOCH=1793712451 tar -cf - . ) | gzip -n -9 > srv-c1.tgz ( cd co2 && SOURCE_DATE_EPOCH=1793712451 tar -cf - . ) | gzip -n -9 > srv-c2.tgz cmp srv-c1.tgz srv-c2.tgz || echo DIFFERENT # DIFFERENT 

It doesn't read it. Sure enough, SOURCE_DATE_EPOCH does not appear even once
in the GNU tar 1.35 manual. I have to
supply the time myself.

The recipe, and how much of it is actually needed

There is a standard command going around for reproducible tar. I set it up too,
taking the commit time as the reference:

E=$(git log -1 --format=%ct) tar --sort=name --format=gnu --owner=0 --group=0 --numeric-owner \ --mtime=@$E -cf - . | gzip -n -9 > bundle.tgz 

Twice, over two separate checkouts:

2432400b6cf31092195d970616d7b85ca5d73e4030ce87a3c19af9442fdde04a srv-d1.tgz 2432400b6cf31092195d970616d7b85ca5d73e4030ce87a3c19af9442fdde04a srv-d2.tgz 

I repeated it five times and five runs produced a single hash. The recipe
works.

But "it works" and "every part of it is necessary" are not the same claim. I
removed the flags one at a time and measured:

Only one of six flags turned out to be decisive. I'd like to read that table as
"delete the other five," but it's worth being careful about what the
measurement actually proves: both runs used the same user, the same machine,
and the same set of file names.

--sort=name wasn't needed because both directories got the same order from
the operating system — even though I deliberately created the files in reverse
order:

co1: ./ ./scripts/ ./scripts/notify-subscribers.mjs ./package-lock.json ./deploy/ ./deploy/backup.sh ./package.json co2: ./ ./scripts/ ./scripts/notify-subscribers.mjs ./package-lock.json ./deploy/ ./deploy/backup.sh ./package.json 

I'm not going to assert a mechanism for why that order came out the way it did;
the only measured fact is that the two directories agreed. The manual's
sentence about the default isn't a guarantee either: --sort=none, meaning
"stores archive members in the same order as returned by the operating system."
Which order it returns is its choice, not my guarantee.

To show that the ownership flags are not ineffective but merely untriggered,
I packed the same tree with two different owners:

tar ... --owner=0 --group=0 -cf - . | gzip -n -9 | sha256sum # 2432400b6cf31092195d970616d7b85ca5d73e4030ce87a3c19af9442fdde04a tar ... --owner=1000 --group=1000 -cf - . | gzip -n -9 | sha256sum # 7679e54a500b20f4b04bc29a1127dd7a90a5565b49adc7df505bf87f40fcb09d 

uid and gid live inside the archive's bytes. My runners run inside containers,
and if that container's user ever changes — an image upgrade, a rootless user
adjustment, take your pick — the hash shifts silently. Five flags aren't
unnecessary; all five are fuses that simply haven't blown yet.

Choosing a format: the smallest one silently drops files

I held the recipe fixed and changed only the format:

POSIX's standard format is the only non-reproducible option on the list. The
cause is the same one I hit in layer two — nanoseconds in extended headers,
this time in the two fields next to mtime:

--- first run --- 30 atime=1793713274.792094234 30 ctime=1793713270.227166462 --- second run --- 30 atime=1793713274.899092541 30 ctime=1793713274.402100404 

--mtime only forces mtime; access and status-change times come straight from
the filesystem. There is a cure:

tar ... --format=pax --pax-option='delete=atime,delete=ctime' -cf - . 

With that, pax produces byte-identical output too. Still, I have no reason to
pick pax.

v7 produced the smallest archive and looked reproducible. Its trap showed up
when I tried the two extremes:

# 198-character path, longest component 60 characters v7 tar: xxx.../fff...: file name is too long (max 99); not dumped ustar fine gnu fine # 124-character path, a single 120-character component v7 tar: tek/aaa...: file name is too long (max 99); not dumped ustar tar: tek/aaa...: file name is too long (cannot be split); not dumped gnu fine 

It skips the file — and it doesn't error out, it warns and carries on. ustar
is more generous: it splits the path at a / into a 155-byte prefix and a
100-byte name, reaching up to 256 characters in total. But when it finds
nowhere to split, it performs the same silent skip, only the message changes:
"cannot be split".

Does this hit my bundle? Not directly — what goes into the bundle isn't the
repository tree but the dist-deploy build output plus four files. I can,
however, derive the output's path lengths from the routing layout: the pattern
dist-deploy/client/en/blog/<category>/<slug>/index.html reaches 135
characters at its longest, with a longest single component of 94 characters. So
v7 would be a disaster for me, ustar is safe, and gnu has no limit at
all. "Produces the smallest output" and "is correct" rarely appear on the same
line.

Already solved: git archive

After assembling that whole list of flags, I realised a tool that gets this
right by design had been sitting under my hand the entire time.

git archive --format=tar HEAD package.json ... | gzip -n -9 > ga1.tgz sleep 2 git archive --format=tar HEAD package.json ... | gzip -n -9 > ga2.tgz cmp ga1.tgz ga2.tgz && echo SAME # SAME 

I passed no flags at all. Reading the member mtime made the reason obvious:
octal 15272360503, which is 1793712451 — the commit's own time, to the
second. The
git archive documentation
says so explicitly: when a commit or tag is given, the commit time recorded in
the commit object is used; when a tree is given, the current time is. I checked
that distinction too:

T=$(git rev-parse 'HEAD^{tree}') git archive --format=tar "$T" ... | gzip -n -9 > gt1.tgz sleep 2 git archive --format=tar "$T" ... | gzip -n -9 > gt2.tgz cmp gt1.tgz gt2.tgz || echo DIFFERENT # DIFFERENT 

Same tree, same content, non-reproducible output. The only difference is what I
asked git for. On top of that, the archive built from a commit has something
extra in its very first block:

commit archive, block 0: name='pax_global_header' type='g' block 1: '52 comment=b41c48d8e83b8c01bba87dcbb2a43fb339d16f20' tree archive, block 0: name='deploy/' type='5' 

The archive carries its own commit in a global pax header, and git can read it
back:

git get-tar-commit-id < ga_commit.tar # b41c48d8e83b8c01bba87dcbb2a43fb339d16f20 git rev-parse HEAD # b41c48d8e83b8c01bba87dcbb2a43fb339d16f20 

The tree-derived archive has no such header and the command returns nothing. A
curious detail: both raw tars are exactly the same size, 409,600 bytes —
tar's block padding swallows the extra 1,024. The difference only appears after
compression (81 bytes here), and that figure moves with the gzip version. So
"how much bigger" is the wrong question; the right one is "is the header
there".

git archive HEAD is both reproducible and arrives carrying its own identity —
the embedded version of what my _v file does by hand. Unfortunately my bundle
doesn't come entirely from git: the dist-deploy inside it is a build output,
not tracked. So I can't use git archive directly. But I can copy the recipe
from its behaviour, and --mtime=@$(git log -1 --format=%ct) is exactly that.

Four things the recipe doesn't cover

After setting up the recipe I went looking for what could still shift my hash.
Four of them were measurable.

The compressor itself isn't pinned. I compressed the same raw tar on two
machines with gzip -n -9:

Mac, Apple gzip 479 : 92,881 bytes 6d5f6f9453008a016c1a... VPS3, GNU gzip 1.12 : 93,152 bytes cf5e74f21f8eb1631352... 

Same input, same flags, 271 bytes apart and two different hashes. Making tar
deterministic doesn't pin the compressor. I warn about the hash shifting if a
uid changes, but a zlib upgrade is more likely than that.

--sort=name doesn't sort the command line. It sorts within directories
only; the order of the operands stays yours:

tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner -cf - b a | sha256sum # 70ac3934e858b4d70a835c2a5cffc5eb4fb6908d2a4f594397518c48c836616a tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner -cf - a b | sha256sum # 244709520989165eb0d890768155a7701fa61b05f0937b7d9266bbd33a6fdf66 

My deploy line names five separate operands. That order is fixed in the
workflow file, but it needs to be fixed deliberately.

Permissions aren't covered. There is a flag for uid/gid and one for mode
(--mode), but the recipe has neither for mode. If the runner's umask changes,
the archive changes.

--mtime overwrites everything. The bundle's largest part is untracked
build output; pulling all of it back to the commit's instant affects anything
on the server that runs find -newer or emits Last-Modified. Nothing in my
pipeline does, but for anyone copying this, it isn't free.

Checklist

For anyone who wants to trust an archive's bytes, in the order I worked
through them:

1. Separate compression first. Run tar -cf and gzip as distinct steps; it's the only way to see which layer is broken.
2. Look at the offset of the first differing byte with cmp. Between 4 and 7 means the gzip header; further in means tar member headers. The diagnosis takes a second.
3. Turn the timestamp off on macOS; Linux doesn't need it. -n for a gzip on the end of a pipe, --options 'gzip:!timestamp' for tar -czf. GNU tar and GNU gzip already don't write it.
4. Take mtime from the commit: --mtime=@$(git log -1 --format=%ct). In my measurement this was the only flag that mattered on its own.
5. SOURCE_DATE_EPOCH alone does nothing. tar doesn't read it; if you want to use the variable, pass it as --mtime=@$SOURCE_DATE_EPOCH.
6. --sort=name and --owner=0 --group=0 --numeric-owner are fuses. They weren't needed in my run; they will be the moment the user, machine, or file set changes.
7. Use --format=gnu. pax/posix break on nanosecond atime/ctime, and v7 silently drops paths longer than 99 characters.
8. Add --no-mac-metadata if you pack from macOS. Otherwise ._ files travel to your server and tar -tf won't show you them.
9. Pin the compressor version too. Same tar, two gzips, two hashes.
10. Write the hash down somewhere. A reproducible bundle whose hash is never compared is only a possibility.

Where this goes next

container-deploy.sh doesn't hash the incoming bundle. Its verification is
functional: does server/entry.mjs exist, does client/_astro exist, does the
server come up on a temporary port and serve /_v. That isn't bad
verification; the comment above the smoke-test block says why it was written —
it was added after an astro upgrade turned into a seventy-minute outage, and it
has done its job ever since. But "it works" and "it is the artefact that left
my build" are not the same question.

Today I pinned the envelope. The next job is named and obvious: is the output
of astro build itself byte for byte identical across two runs? None of
today's measurements answers that, and the experience from the cover-image
article tells me the odds of the answer being "no" are high. I'll measure it
before putting the recipe into the pipeline — because putting different content
into a deterministic envelope on every run is worse than not comparing hashes
at all; it makes you believe you compared something.

The real lesson isn't the list of flags. For years I looked at the _v file and
trusted that I was getting an answer to "which commit is live," when that file
tells me the commit's name, not the bundle's identity. The gap between the two
was invisible because I had never tried to measure it. Black boxes keep making
their quietest mistakes for as long as you leave the lid on — and this lid came
off with a command as simple as cmp.

Official Sources

• RFC 1952 — GZIP file format specification version 4.3
• GNU tar 1.35 manual page
• gzip manual page
• bsdtar (libarchive) manual page
• git archive documentation