CI Failure Triage Needs Evidence Before It Needs a Classifier

2026-11-14来源 HackerNoonEnglish

If you have ever owned a CI pipeline and cleared a wall of red builds before standup, you might recognize this routine. A nightly run finishes with thirty-seven failed tests, and by the time anyone looks, thirty have passed on a rerun and picked up a flaky tag while the other seven get tickets.

Nobody records why the thirty failed in the first place, so the same failures come back the next night under the same label. And it is easy to see why teams settle into this, because a rerun clears the build in minutes while finding the real cause of thirty failures could take the whole team a day.

That gap is where a new class of AI models starts to look tempting. Shawn Gordon’s recent HackerNoon explainer introduces Jev, a decision model from TypeSafe AI that writes nothing and only picks an answer. It chooses from up to 255 options or returns a yes or no probability in about half a second. TypeSafe also says those probabilities are calibrated, so an answer given at 80 percent confidence should be right about 80 percent of the time.

Sorting a failed test into a product defect, environment noise, or an agent error is a three-option choice question, which makes it look like a natural fit. But when Mastra, the company behind an open-source framework for building AI agents in TypeScript, tested Jev on a related triage job, the results were poor.

Jev labeled 78 of Mastra’s own GitHub issues as urgent, high, or low, using only each issue’s title and body, and only about 10 of its answers matched an engineer’s judgment. Mastra concluded that the model could not see the evidence that priority depends on, and Gordon put the risk plainly in his explainer. “A consistently wrong classifier that runs cheaply is just a faster way to tag things badly.”

CI triage has the same trap, because a model can sort failures well only when the pipeline has kept the evidence that separates one category from another. So the evidence has to come first and the classifier second.

Three Failure Categories Need Three Different Owners

Most pipelines record a failed test as one status, and that single status hides three kinds of failure that need different fixes from different teams.

Sai Krishna, Director of Engineering at TestMu AI (formerly LambdaTest), draws that line before any fix starts. “We should ideally treat these failures into three different categories,” he explains. “Whether it is a product defect, like an actual defect in the product. Or if it is environmental noise, maybe a network issue, a latency problem, a backend API that went down. Or very importantly now, since agentic pipelines are involved, whether it is an agent error.”

He also gives a working definition for the third category, which is the one teams find hardest to name. “If the agent’s reasoning or actions are inconsistent, that is the point where we can clearly say it is an agent error,” Krishna says. “We have to put them in different buckets.”

Each bucket leaves a different trail, and that trail is what a classifier will eventually read. A product defect reproduces on the same commit under the same conditions and usually traces back to a recent change in the application. Environment noise shows up as timeouts or 5xx responses from a dependency, and it tends to hit unrelated tests in the same time window.

An agent error happens in a healthy environment and with an unchanged commit, while the agent’s steps differ from the last passing run. So the first practical move is to write those signals down as rules for each bucket and to name an owner for each one. The feature team takes defects, the platform team takes noise, and whoever owns the prompt and the model takes agent errors.

Parallel Runs Hide a Fourth Category Inside the Noise

Three buckets cover most failures, but some failures look like environment noise only because of how the suite runs.

Srinivasan Sekar, also Director of Engineering at TestMu AI, sees this most often when teams move from local runs to large parallel grids. “The most common and unanticipated failure is shared state across parallel workers,” he highlights. “Locally, one or two tests run in parallel, and everything works, but in the cloud with 200 parallel workers you suddenly see tests depending on specific execution order.” The usual causes are global fixtures or singletons that several tests change at once, race conditions in test data setup, and ordering dependencies that nobody noticed while the suite ran one test at a time.

These failures are test defects. And a classifier that only knows three labels will file them as environment noise, because they come and go without any change to the code. The fix is to add a fourth label and give the classifier the one signal that separates it, which is whether the test still fails with a single worker.

Rerunning each failure once with one worker inside the triage job costs a few seconds and turns an ambiguous failure into a clear one. And the repair itself belongs in the test, through per-worker test data and idempotent setup and teardown, and not in the infrastructure.

Capture the Agent’s Steps Along With the Result

Agent errors are the hardest bucket to fill correctly, because the evidence is in decisions the agent made and not in any log the pipeline already keeps.

Tom Howe, Director of Insight Engineering, built his team’s approach around those decisions. His team captures a trace of every action an agent takes, with full context, in a cryptographically secure audit system. It then uses separate agent evaluators to review those traces for logic errors that have happened or might happen next.

“The steps that the non-deterministic system takes to get to an output are as important to test as the outputs themselves,” he explains. “By examining a detailed trace of the steps an agent takes and the tools it leverages to generate an output, you can get deep insight into where decisions are brittle and anticipate errors before they happen.”

That trace is also the missing input for automated triage. After its test, Mastra advised letting an agent gather the evidence first and letting the classifier judge what it gathered, and CI works the same way. A triage job can attach an evidence bundle to every failure before any model sees it.

The bundle carries the commit, the model and prompt versions, dependency health at the start of the run, the retry outcome, and a diff of the agent’s steps against the last passing run.

{ "test": "checkout > applies promo code", "commit": "9f3c2e1", "model": "provider-model-2026-09-15", "prompt_version": "checkout-agent@14", "attempts": [ {"status": "failed", "workers": 8}, {"status": "passed", "workers": 1} ], "dependency_health_at_start": { "payments-api": "degraded", "inventory-api": "ok" }, "network_errors": ["POST /payments 503 after 30012 ms"], "agent_steps_diff_vs_last_pass": [ "skipped: open_cart", "added: search_product" ] }

Every field in that bundle points at a bucket. A degraded payments API points at noise, a failure with eight workers followed by a pass with one points at a test defect, and a changed step sequence on a healthy environment points at an agent error. When the signals conflict, as they do in this example, the bundle shows the conflict instead of hiding it. A person reading this bundle can usually name the bucket quickly, and so can a decision model.

Route Failures by Label and Confidence

Once the evidence exists, the classifier’s job is narrow, and keeping it narrow is what makes it safe to automate.

Shahram Anver, Co-Founder and CEO of Cleric, which builds AI SRE agents for incident investigation, shows how far that labeling can go once a team has the evidence to support it. His team tracks 51 failure mode labels across its agent investigations. Those labels separate tool access problems from reasoning errors, omitted steps, and unfinished runs, because a status that only says the agent responded tells the team nothing about whether it did useful work.

Most CI teams do not need 51 labels on day one, but Anver’s taxonomy shows that the label set should grow as the evidence gets richer. A decision model makes that growth cheap, because a choice question over four labels or forty costs about the same to run. The harder design question is the rule for acting on each answer. The code below sends a failure straight to its owner only when the model is confident, and puts everything else in a human triage queue.

LABELS = ["product_defect", "environment_noise", "agent_error", "test_defect"] OWNERS = { "product_defect": "feature-team", "environment_noise": "platform-team", "agent_error": "agent-owners", "test_defect": "test-authors", } AUTO_THRESHOLD = 0.90 def route(evidence_bundle, classify): """classify() wraps the team's decision model or classifier.""" label, confidence = classify(state=evidence_bundle, options=LABELS) if confidence >= AUTO_THRESHOLD: return {"label": label, "route_to": OWNERS[label], "automatic": True} return {"label": label, "route_to": "human-triage", "automatic": False}

The threshold should come from the team’s own data and not from a vendor’s claim. Public write-ups on Jev note that most claims about the model come from TypeSafe itself and that no calibration results have been published yet. So run the classifier in shadow mode for two weeks, keep human labels for the same failures, and check whether answers given at 90 percent confidence are actually right about 90 percent of the time.

from collections import defaultdict def calibration_report(predictions): """predictions is a list of (confidence, predicted_label, human_label).""" buckets = defaultdict(lambda: [0, 0]) for confidence, predicted, actual in predictions: bucket = min(int(confidence * 10), 9) / 10 buckets[bucket][0] += 1 buckets[bucket][1] += predicted == actual for bucket in sorted(buckets): total, correct = buckets[bucket] upper = bucket + 0.1 accuracy = correct / total print(f"{bucket:.1f}-{upper:.1f} n={total} accuracy={accuracy:.2f}")

If the 0.9 bucket comes back at 0.70 accuracy, the model is overconfident on this team’s failures. In that case, raise the threshold or improve the bundle before letting it route anything on its own.

Krishna treats the classifier as the last step of triage, not the first. “A model can sort failures faster than anyone on the team, but only if the pipeline kept what that model needs to see,” he says. “Start by labeling a month of failures by hand. Once your own engineers agree on the buckets, a classifier has something real to learn from and something real to be checked against.”

The first week of this work needs no model at all. Pull the last month of failed and flaky tests, label each one by hand as a product defect, environment noise, an agent error, or a test defect, and note which piece of evidence decided the label. Then make the triage job attach those pieces of evidence to every new failure, run a decision model in shadow mode against the hand labels, and switch on automatic routing only for the labels where its confidence matches its accuracy.

Credits for Commentary

原文：https://www.a8q9.cn/en/article/ci-failure-triage-needs-evidence-before-it-needs-a-classifier-4c8ff6.html

相关文章