The Python Reader I Measured Never Opened 34,557 Files. 310 Tests Still Needed Them.

Originally published on hexisteme notes.

On 2026-10-31 I quarantined 34,557 HTML files in a content-collection system because the reader I had measured never opened them. The collection round that ran next finished complete and a re-trace passed. Then I compared the full test suite to its sealed baseline and found 310 failures beyond it.

All of the files went back the same day, and that was cheap only because I had moved them instead of deleting them. The retry that came after is still in progress. As of 2026-11-04 its storage gate is red, three of its four nightly runs ended restored, and the step that actually deletes anything has never run.

132.8GiB and no alarm

The system's report/ directory had grown to 132.8GiB with no alarm. Jobs running every 15 minutes left a new folder each round, and nothing cleaned up after them.

The first pass was a reference-based pruner. It keeps anything cited in the database, the state or the source, and it sits under a budget gate of 140GiB. It took the directory from 132.8 to 115.1GiB by reclaiming 556 unreferenced units (10.7GiB) and cutting a 9.86GB SQLite file to 2.30GB. Collection runs were still 57.4GiB, and almost all of that was referenced, so a reference check had nothing left to take.

Measuring one reader

The second attempt ran the same day, 16:50 to 17:40 KST, and started from a plain question: isn't this data unnecessary once it's on the blog? The numbers seemed to say yes.

• Of 60,492 tasks, 626 (1%) had published.
• Of 46,643 collected HTML files (52.4GiB), the collection-state validator opened 1,920 (2.2GiB).
• The rest were the same ~2,100 videos re-fetched every 15 minutes, 33,434 history entries of them.

So the candidate set became HTML older than 7 days that the validator doesn't open: 34,557 files, 37.8GiB. At 07:52Z a release tool moved them to quarantine. Moved, not deleted. Afterward the collection round finished complete and a re-trace passed.

Both results were accurate, and neither covered the code that was about to fail. I've written before about checks that report accurately on what they examined and say nothing about what they could not see. This was another case of it.

The full suite disagreed

Comparing the full test suite to the sealed baseline (v20, 31 failures) showed 310 failures beyond the baseline. The causes were ValueError: YouTube page body unavailable, 312 times, and FileNotFoundError on runs/*.html.

Publish replay, re-judging blocked batches and joining past provenance all reopen old capture bodies. None of those appeared in what I had measured, because I had measured one reader.

That was the error. A read set tells you what one program opened. I took its complement and treated it as the files nobody needs, when the complement only describes the validator. One reader's read set was not every reader's, and a delete list has to be computed against all of them.

Recovery cost a rename

I moved all 34,580 quarantined files back, 0 missing, and re-ran the 908 failed tests: 0 failures beyond the baseline. Then I removed the tool, its launchd job, the gate branch and the line in the agent instructions, and locked the lesson with a regression test: a superseded capture cited only by history keeps its whole unit.

Because the move was a quarantine, recovery cost a rename and no re-crawl. The wrong candidate list would have been the same mistake either way. The quarantine changed what it cost to find out.

My record has two counts that don't line up with the earlier ones, and it explains neither. The rollback moved 34,580 files against the 34,557 I had quarantined, and the re-run covered 908 failed tests against 310 failures beyond the baseline. The results I relied on were the 0 missing and the 0 beyond baseline.

Third attempt: pins from running the readers

The third attempt ran 19:30 to 20:40 KST and began with an approved change of policy:

• Delete raw HTML for finished items and keep only receipts.
• Keep raw for in-progress items.
• Give up regenerating old published posts from raw.

The last item is a real loss, accepted on purpose.

I also found the operational reader behind the second failure. Two blocked publishes traced to publication queue replay, which reopens raw files for finished posts: 26 raw files, up to 28 days old. "Finished" did not mean "never read again", so the policy alone could not define the delete list.

The pin list, the files that have to stay, was built by actually running the readers and recording what they open, instead of reading the code. The recorder had two traps of its own.

Trap one: the read that didn't match

Path reads were captured with an audit hook. One module reads like this:

os.open(name, dir_fd=parent_fd) 

With the audit hook alone, the collection-state validator appeared to read 0 raw files. That is the same validator I had measured opening 1,920 files that afternoon. I checked the reason on Python 3.13.0 with a small script: the audit open event carries the relative name 'capture.html', so matching on absolute paths misses it.

The fix was an os.open wrapper plus fcntl F_GETPATH to recover the full path. In the final list the validator's pins include 1,898 raw files.

Trap two: reads collected at exit

Collecting test read records in atexit lost the main pytest process's reads entirely. The regression test for this uses a script ending in os._exit(1). My record stops at the symptom and the test. I haven't confirmed the exact path by which those records went missing, so I won't guess at it here.

Both traps undercounted, and in this design an undercount is the dangerous direction: every read the recorder misses is one fewer pin and one more candidate for deletion.

The list that came out

I applied it as one batch: 32,955 files, 36.6GiB, moved to a quarantine directory in 123 seconds. The self-check passed, and the publish planner passed for both pending publishes after quarantine.

What runs now

• A tripwire job runs every 15 minutes and watches logs for reads of missing files.
• A nightly release job recomputes pins, restores any held file that became pinned, deletes batches past 7 days, quarantines new candidates and self-checks.
• The storage gate fails on a stopped, refused or restored job, and on any tripwire restore within 7 days.
• Bytes in quarantine are excluded from the budget and shown separately.

What I'd carry into another cleanup

This comes from one system, so treat it as what I would try first:

• Move before you delete, on a fixed timer, so a wrong list costs a rename.
• Compute candidates against every reader you can run, and count the test suite as a reader. Mine held 15 raw pins of its own, and it was the check that caught the failure.
• Before trusting a recorder's zero, run it against a reader whose count you already have.
• Compare the whole suite to a baseline after the move. The checks closest to the change passed here.

Where it stands on 2026-11-04

The budget gate prints report/ at 49.1GiB of 140GiB, with 41.0GiB in quarantine awaiting deletion. The first number looks like success, but quarantined bytes are excluded from it, and those 41.0GiB have not been reclaimed.

The tripwire showed tripped [] in the checks I read on 2026-11-04.

The nightly job is where it isn't working. Of four runs since the first batch:

• 2026-10-31, 20:20Z: ended restored
• 2026-11-01: ended restored
• 2026-11-02: held
• 2026-11-03: ended restored

All three restored runs carry the same error: pin source collection failed: CaptureReleaseError: collection state stayed busy. That turned the storage gate red today, PASS -> REGRESSION. The gate is doing what I built it to do, since it fails on a restored job. The failure underneath is not resolved at the time of writing.

Nothing has reached the 7-day delete yet. The expired lists are empty in all receipts, so the deletion step has not run for real. Everything that has gone right so far was a move, and a move can be undone. The step that can't be undone is still ahead of me.

The limits I'm carrying into that step:

• I don't know why collection state stayed busy on three of four runs.
• The pin list covers the readers I ran. A reader I haven't listed, or a path those runs didn't take, isn't in it, which is the second attempt's mistake made smaller.
• After a real delete the tripwire can still report a read of a missing file, but there will be no quarantined copy left to restore.

The next thing to check is the first nightly run with a non-empty expired list, and what the tripwire reports after it.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.