On Windows, four processes appending to one file lost 37 of 200 lines

My CI failed with this, only on the Windows runner, on a commit that changed nothing but a few GIFs:

receipts: Unexpected token '\', "\hostedtoo"... is not valid JSON 

The same test had passed on that runner a few hours earlier. Here is what it was, because it is a small bug that is easy to ship.

The setup

Receipts runs the tests a change added or edited twice: with the change, and with the changed source files put back the way they were. For pytest it loads a tiny plugin that writes one JSON line per test result into a file, and the Node side reads that file afterwards.

def _emit(record): out = os.environ.get("RECEIPTS_PYTEST_OUT") if out: with open(out, "a", encoding="utf-8") as fh: fh.write(json.dumps(record) + "\n") 

One file, opened in append mode, one line per record. Fine, until the project uses pytest-xdist. Then several worker processes run the tests, and at the end every one of them writes a record with the list of modules it imported. Those records are long, full of paths like \hostedtoolcache\windows\Python\.... That is the fragment in the error: the start of a line that got torn in half.

The repro

Four processes, each appending the same long JSON line 50 times to one file:

import json, os, multiprocessing as mp def worker(out, n): rec = {"files": [r"\hostedtoolcache\windows\Python\Lib\m%04d.py" % i for i in range(400)]} for _ in range(n): with open(out, "a", encoding="utf-8") as fh: fh.write(json.dumps(rec) + "\n") if __name__ == "__main__": out = "race.jsonl" ps = [mp.Process(target=worker, args=(out, 50)) for _ in range(4)] [p.start() for p in ps]; [p.join() for p in ps] print(sum(1 for _ in open(out, encoding="utf-8")), "lines") 

On my Windows machine: 163 lines instead of 200. 37 writes simply vanished. With records of different lengths you also get torn lines instead of clean losses, which is what CI hit.

Why

On POSIX, O_APPEND means "move to the end and write" happens as one step for each write(), so concurrent appenders don't overwrite each other. On Windows, the C runtime emulates append mode: it seeks to the end, then writes. Two processes can both seek to the same end and the second write lands on top of the first. Nothing errors, data is just gone.

It only showed up now because it needs two xdist workers to finish within the same moment. It had been passing for weeks.

The fix

Stop sharing the file. The controlling process keeps out.json, each xdist worker writes out.json.<worker id>, and the reader picks up all of them:

worker = getattr(_config, "workerinput", None) if worker is not None: out = "%s.%s" % (out, worker.get("workerid", os.getpid())) 

While I was there, the reader stopped throwing on a line it can't parse: a run killed for hanging can leave a half-written last line, and the records before it still count.

The regression test doesn't try to reproduce the race (that would be a flaky test about flakiness). It checks the mechanism directly: a worker writes to its own file, the controller to the main one, and a torn line at the end doesn't sink the read. It fails on the old plugin and passes on the new one, which, given what the tool is for, felt like the least I could do.

Takeaway

If several processes append to one file, it works on Linux and macOS and loses data on Windows, quietly and rarely. Give each process its own file, or take a lock.

I drafted this post with an AI assistant. The error, the numbers and the code are from the actual CI run, my machine and the commit: https://github.com/syntaxixr/receipts/commit/70bb9b2