Make AI support triage reproducible

The first run of a support triage prompt can look excellent because the sample ticket is unusually clear. A week later the same prompt sees a billing complaint with two questions, no account ID, and a screenshot mentioned but never attached. A reproducible test starts with that messier input.

Build a fixture set

Write six synthetic tickets and save them next to the prompt. Include a duplicate request, a missing field, a billing and technical issue in the same message, a vague urgency claim, a clear outage report, and one clean control. Give each ticket an expected route and write down which facts the response may use. Keep real customer data out of the fixtures.

Record outcomes separately

Score routing and factual accuracy in separate columns. A ticket can reach the right team while the response invents an invoice date. For that case the routing score passes and the facts score fails. A third column should capture whether the model asks a useful question when the ticket lacks a required detail. Check cases individually so an average never hides a critical error.

Change one variable

Run the baseline first. Save the prompt text, fixture version, model identifier, and output. Next, change a single instruction such as asking for structured fields and repeat the same inputs. A workflow reference can help when sketching the sequence from input to review. Repeat any ambiguous failure to see whether it is consistent before changing the prompt again.

Decide what blocks a release

Treat an invented refund, guessed account ID, or confident misroute on an outage as a release blocker. Keep the failing ticket as a regression case, then add a nearby variation so the fix has to generalize. A good test report tells the next person exactly what to rerun and which result would change the release decision.

Define the fixture contract

A fixture can fit in a single JSON object with five fields: id, ticket, expected_route, required_facts, and forbidden_claims. Keep those last two arrays explicit. For a ticket that says “I see a duplicate charge but can’t find the receipt,” the required fact is that the customer reports two charges. A confirmed duplicate transaction and a promised refund both belong in forbidden_claims. The expected route might be billing, while the expected next step is to request the transaction reference through the approved channel. Avoid making the expected reply a verbatim paragraph. Two different replies can meet the same requirements.

Keep the evaluator small

If the model returns a route from a fixed set, compare it with expected_route mechanically. Report a parse failure separately from a wrong route. A malformed object and a valid object sent to the wrong team point to different fixes. For the free text fields, start with human review. A phrase match can miss a paraphrased promise and can also flag a sentence that explicitly says a refund cannot be promised. Give the reviewer a compact table with one row per run and a column for the exact phrase or missing fact that triggered each failure.

Make a retry policy before looking at the results

Suppose two runs on the same ticket yield different routes. Retrying until the desired route appears gives a misleading report. Decide in advance how many runs to record for each candidate, then keep every output with its run number. If the baseline is already inconsistent, write that down as part of the comparison. A prompt change that improves clean examples but makes an outage case less consistent deserves another iteration before it reaches production.

Save the decision beside the result

Record who reviewed the run and what they accepted. For example, a team might allow one noncritical style issue in six fixtures and still require every outage to be routed correctly. Give the next reviewer the fixture version and the baseline and candidate outputs. When a new type of failure comes in, reproduce it with a synthetic example and add the fixture before attempting a fix. The record then explains both how the prompt changed and which cases justified keeping the change.