A Truncated JSONL File Parses Fine. That's Why It Took Me 11 Days to Notice My Reports Were Empty.

My morning dashboard said the same thing for eleven days straight:

2026-10-14 anomalies: 0 status: PASS 2026-10-15 anomalies: 0 status: PASS ... 2026-10-25 anomalies: 0 status: PASS 

Zero anomalies. Eleven clean days. I actually felt good about it. I'd spent a weekend hardening that pipeline, and here was the payoff — a Raspberry Pi in my closet quietly watching a stream of sensor events and telling me, every morning at 06:50, that nothing was wrong.

On day twelve I opened one of the raw files by accident, looking for something unrelated. The file was 900 bytes. A normal day was 210 KB.

The reports weren't clean. They were hollow.

What the pipeline actually did

It's not complicated, which is the point. A writer process appends one JSON object per line to a dated file:

with open(path, "a") as f: f.write(json.dumps(event) + "\n") 

Bash on the Pi, tiny Python, a couple thousand events a day. A second process reads that file fifteen minutes later, aggregates, and writes the summary:

rows = [json.loads(line) for line in open(path) if line.strip()] 

This is the part I want you to sit with. When a writer and a reader touch the same file without any coordination, the reader can see a partial file. Not a corrupt file — a shorter one. My writer was still flushing when the reader opened it. The reader got whatever bytes had landed so far.

With JSON, that would have exploded. A truncated {... is a JSONDecodeError and I'd have known in seconds.

With JSONL it just... parses. Each line is independent. There's no footer, no closing brace, no record count, no checksum, nothing that says "this file is complete." A file with 40 lines out of 2,100 is a perfectly valid JSONL file. It's just a valid file that describes a much smaller world.

And so the aggregation ran over 40 rows, found no anomalies in those 40 rows, and wrote anomalies: 0, status: PASS.

The failure wasn't an exception. The failure was a successful run over data that wasn't there yet.

Failure #1: I fixed it with a sleep

My first fix was the kind of fix you make at 07:10 before a standup. I added a wait.

time.sleep(120) # let the writer finish 

It worked. For about a week. The reports came back with real row counts, real anomalies, real numbers I could act on.

Then it broke again, and this time it was worse, because now I trusted it.

The Pi's SD card got busy one morning — a backup job and the writer overlapping. The flush that usually finished in two seconds took four minutes. My reader woke up at its scheduled time, saw an incomplete file, and produced another hollow PASS. Same bug, new latency profile. A sleep doesn't create an ordering guarantee, it just moves the race condition to a time I'm not looking at. I had encoded an assumption about disk speed into a sleep() and then forgotten I'd done it.

If your correctness depends on "the other thing is probably done by now," you don't have a pipeline. You have a hope with a cron entry.

Failure #2: the check that agreed with the bug

The second fix was the one I was proudest of, and it's the one that should embarrass me most. I added a row-count sanity check:

prev = load_yesterday_count() if len(rows) < prev * 0.90: alert("volume drop") 

Compare today's volume to yesterday's, alert if it falls more than 10%. Reasonable! Standard! And completely defeated by the fact that yesterday's baseline was also truncated.

Day one: 2,100 rows. Day two got truncated to 1,940 — a 7.6% drop, under my 10% threshold, so no alert, and now the baseline was 1,940. Day three truncated to 1,790 — again under 10%. The baseline ratcheted downward, one silent step at a time, each step inside tolerance. Eleven days later my "normal" was 900 bytes, and every single day had passed a check designed to catch exactly this.

I built a drift detector and pointed it at a drifting reference. Relative checks only work if the denominator is anchored to something that can't quietly move. Mine moved every night and reported that all was well.

What I actually did, in order of how much it mattered

1. Stop writing in place. The writer now writes a sibling temp file and atomically renames it into position. On POSIX, os.replace is atomic — a reader either sees the old file or the new one, never half of either.

tmp = path + ".tmp" with open(tmp, "w") as f: f.write(payload) f.flush() os.fsync(f.fileno()) # survive power loss, not just process exit os.replace(tmp, path) 

For an append-only event stream where you can't rewrite the whole file, write to events.jsonl.tmp continuously and os.replace it into events.jsonl when the batch closes. Same guarantee, no rewriting cost at read time.

2. Make completeness explicit. Every file now ships with a sentinel:

{"rows": 2087, "sha256": "9f2c...", "closed_at": "2026-10-26T06:41:12Z"} 

The reader refuses to run without one. No sentinel, no report — and a missing report is an alert, not a silence.

3. Check the denominator, absolutely. Not "down 10% from yesterday." The writer knows how many events it emitted. The reader asserts it received exactly that many, or it fails loudly. A count that doesn't match is a bug, not a tolerance band.

4. Alert on absence, not on emptiness. This was the real lesson and it took me three incidents to learn it. My monitoring had one signal: anomalies > 0? No → quiet. But "zero anomalies" and "zero data" are two completely different states of the world that produced the identical output. Every pipeline needs a heartbeat that fires when there's nothing to say. I now have a dead-man's switch: if the daily summary doesn't arrive by 07:00, I get a message. Silence is only trustworthy when it's accompanied by proof of life.

5. Log the input, not just the decision. Every report now carries rows_read, rows_expected, file_bytes, file_mtime, sentinel_present. Eleven days of hollow reports would have been instantly visible in a single column, because 40 next to 2,100 is not a subtle discrepancy. I had been logging my conclusions and throwing away my evidence.

The general shape of this bug

This wasn't a JSONL bug, or a Raspberry Pi bug, or a cron bug. It's the pattern where the absence of data is indistinguishable from the absence of problems — and it shows up everywhere once you start looking for it.

An agent that fails to retrieve context and answers confidently from memory. A scraper that gets zero results and reports "no new items." A monitoring query that returns an empty set because the auth token expired, and the dashboard renders that empty set as green. Every one of those is a truncated file wearing a different costume. The system did not fail to produce an answer. It produced an answer about a smaller world than the one it was supposed to be looking at, and nobody asked how big the world was.

The question I ask now, of every automated report I read, is not "did it pass?" It's "how much did it look at, and who told it that was all of it?"

If you can't answer that in one line of output, you don't know whether the pass means anything.

The two-minute version I wish I'd had

• Write to temp, fsync, os.replace. Never read a file another process may still be writing.
• Ship a record count and a checksum with every artifact. No footer, no trust.
• Assert absolute counts, not relative deltas. Ratcheting baselines hide slow truncation perfectly.
• Alert when a report is missing, not only when it looks bad.
• Print the denominator next to the conclusion. rows_read / rows_expected would have caught this on day one.

Eleven days of PASS wasn't a monitoring failure. It was a measurement failure. The pipeline was working; it was just measuring a world that had quietly shrunk to 900 bytes.

The full checklist + scripts are in Ship Safe — The Launch-Day Security Kit — code LAUNCH90 at checkout makes it $1.50.