My Worker Exited 0 With Jobs Still Marked "Running", and Nothing Retried Them

A deploy finished at 02:10, the supervisor logged a clean stop, and the worker exited with code 0. By 02:12 the queue depth graph was flat at zero, which looked like success and was actually the opposite. Work had been claimed, never finished, and the rows sat in running where no retry could see them. So what exactly happens to a job that is mid-flight when the platform sends SIGTERM?

The setup that made the bug invisible

My worker loop claims one row at a time from a jobs table, calls one outbound HTTP endpoint, then marks the row done. Each call has a 20-second client timeout, and the orchestrator allows 10 seconds between SIGTERM and SIGKILL. That mismatch alone guarantees the drain can never finish, but three smaller decisions turned it into a silent data problem instead of a loud crash.

• The signal handler set a boolean and returned, so Python exited normally and reported 0.
• My "shutdown complete" log line counted finished jobs, not released claims, so it was always cheerful.
• There was no lease and no heartbeat, so the reaper only touched rows whose claimed_at was older than 30 minutes.

Put those together and the failing path is boring: signal arrives, handler flips a flag, the main thread is still blocked inside a 20-second socket read, SIGKILL lands, and the row keeps status='running' for half an hour. Is your shutdown path covered by any test at all, or only by the deploy that happens to be quiet?

A harness that reproduces it in about a minute

The useful part of this bug is that it is cheap to reproduce, because you only need real signals and a real process table. Save the worker as worker.py, then drive it from this script:

#!/usr/bin/env bash set -uo pipefail python worker.py --drain-seconds 0 & pid=$! sleep 3 # let it claim a few rows kill -TERM "$pid" sleep 2 # grace window is shorter than the HTTP call kill -KILL "$pid" 2>/dev/null || true wait "$pid"; echo "exit=$?" python inspect.py # prints rows still in status='running' 

And inspect.py is deliberately dumb, because you want it to fail loudly:

import os, psycopg with psycopg.connect(os.environ["DATABASE_URL"]) as conn: rows = conn.execute( "SELECT id, claimed_at FROM jobs WHERE status = 'running' ORDER BY id" ).fetchall() print(f"stuck_running={len(rows)}") for job_id, claimed_at in rows: print(f" job={job_id} claimed_at={claimed_at.isoformat()}") raise SystemExit(0 if not rows else 1) 

Run it against the naive worker and you should see exactly the number of jobs that were in flight when the kill landed. Run it against a fixed worker and the count should be zero, or the rows should already be back in queued before your script checks. Label: the SQL below is reconstructed pseudo-Postgres, not a copy of production code; adapt the claim statement to your own store.

The fix is three separate changes, not one

The first temptation is to "handle the signal properly", which does not help when the kill arrives mid-socket. The version that survived my next deploy had three parts.

1. Stop claiming immediately. After STOP is set, the loop exits before the next UPDATE ... FOR UPDATE SKIP LOCKED.
2. Make the claim a lease, not a lock. A heartbeat refreshes claimed_at while work runs, and the reaper reclaims anything older than the lease window.
3. Tell the truth in the exit code and the log line. Report how much work was still in flight at exit instead of reporting only what finished.

import os, signal, sys, threading, time import psycopg STOP = False IN_FLIGHT: set[int] = set() def on_signal(signum, _frame): global STOP STOP = True print(f"event=stop_requested signal={signum} in_flight={len(IN_FLIGHT)}", flush=True) signal.signal(signal.SIGTERM, on_signal) signal.signal(signal.SIGINT, on_signal) HEARTBEAT_EVERY = 5.0 def run_with_heartbeat(dsn: str, job_id: int, fn): done = threading.Event() def beat(): with psycopg.connect(dsn, autocommit=True) as hb: # own connection: not shared  while not done.wait(HEARTBEAT_EVERY): hb.execute("UPDATE jobs SET claimed_at = now() WHERE id = %s", (job_id,)) t = threading.Thread(target=beat, daemon=True) t.start() try: return fn() finally: done.set() dsn = os.environ["DATABASE_URL"] lease_exit_code = 0 with psycopg.connect(dsn, autocommit=True) as conn: while not STOP: row = conn.execute( "UPDATE jobs SET status = 'running', claimed_at = now() " "WHERE id = (SELECT id FROM jobs WHERE status = 'queued' " " ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 1) " "RETURNING id, payload" ).fetchone() if row is None: time.sleep(0.5) continue job_id, payload = row IN_FLIGHT.add(job_id) try: run_with_heartbeat(dsn, job_id, lambda: do_work(payload)) conn.execute("UPDATE jobs SET status = 'done', claimed_at = NULL WHERE id = %s", (job_id,)) except Exception as exc: conn.execute("UPDATE jobs SET status = 'queued', last_error = %s WHERE id = %s", (str(exc), job_id)) finally: IN_FLIGHT.discard(job_id) if IN_FLIGHT: lease_exit_code = 143 # 128 + SIGTERM: stopped with work outstanding  print(f"event=stopped_with_work in_flight={sorted(IN_FLIGHT)}", flush=True) else: print("event=drained_cleanly", flush=True) sys.exit(lease_exit_code) 

The reaper then becomes a one-liner you can reason about, and its window is bounded by the heartbeat interval rather than by a 30-minute guess:

UPDATE jobs SET status = 'queued', claimed_at = NULL WHERE status = 'running' AND claimed_at < now() - interval '45 seconds'; 

Keep the lease comfortably larger than the heartbeat plus the slowest single call, and check the numbers instead of vibes. One caveat that bit me: do not reuse the main connection inside the heartbeat thread, because most drivers are not thread-safe and you will get a confusing protocol error that looks like a database bug.

Picking a deadline you can defend

The shutdown deadline, the per-job timeout, and the platform grace period are three numbers that must agree, and I now write them in one table before touching the code.

If the platform grace is shorter than your worst-case drain, stop pretending you can drain: shorten the per-job timeout, cut concurrency to one at a time, and let the lease do the recovery work. Which of those three numbers did you actually measure, and which one did you copy from a sample config file?

What the free model and free server actually changed

Two parts of this loop are annoying to set up on a laptop, so I used MonkeyCode's free model access and free server option for them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model was decent at the tedious part, turning a few thousand lines of claim/release log lines into an ordered timeline of stop_requested, last heartbeat, and kill, which is exactly the reading I am worst at doing patiently. The free server gave me a plain Linux shell where kill -TERM hits a real process and wait returns a real exit code, which is the whole point of the harness above.

What it did not do is decide the deadline numbers or tell me the reaper window was wrong, because a model reading logs only sees what I logged. The honest limitation is that if your worker never emits in_flight, no amount of summarising will recover it, so fix the log line first and paste it second.

What I would repeat, and who should skip this

The changes I would keep tomorrow are the harness (it runs in CI as a smoke test), the lease column with a heartbeat, and the event=stopped_with_work line that a grep can find during a post-deploy check. The sprint I would not repeat is the hour I spent adding a longer SIGTERM handler, which cannot help a thread that is blocked in a socket read.

Skip this approach if you are on a runtime with no signal semantics, if your grace period is under a second, or if you cannot add a lease column and a reaper. Also skip it if you need exactly-once processing, because a lease gives you at-least-once and your handler still has to be idempotent. If you want to rerun the harness without provisioning your own box, the free server option is a reasonable place to start, and the code above is the part worth copying.