How to turn AI failures into an evaluation dataset

Start with the mismatch

A useful evaluation record needs to preserve that mismatch. A label such as "bad answer" gives the next engineer very little to work with.

In the previous article, we tested those failures in a small investor-services workflow. This article turns its saved traces into a provisional evaluation dataset. The companion contains an AI-assisted review of all ten runs: five scripted scenarios, each before and after repair. They are fictional software tests, not production interactions or measured LLM behavior.

Define the task before labeling failures

The user is an investor-services analyst. They have already confirmed that bank-account evidence is required and outstanding for a particular application. The assistant should create one internal follow-up task against that application, or record why review is needed. It must leave bank details and the admission decision unchanged.

That is a plausible fund-admin operation, with a specific scope. The Harding Loevner application administered by Northern Trust requires bank evidence for relevant applicants and specifies an exemption on page 2. It does not establish a universal document checklist or prescribe our task table.

The distinction matters when labeling data. A missing document is an input condition. It becomes an assistant failure only when the assistant handles it incorrectly under the applicable requirement.

Write observations before choosing categories

Start with the request, available context, tool results, and stored outcome. Write what happened in plain language. This follows the open-coding approach described by Hamel Husain and Shreya Shankar.

The reviewed baseline traces produced these notes:

The last row belongs in the dataset too. An incomplete original task can still be the correct disposition. Otherwise, an evaluator might reward the assistant for acting when it should stop.

Keep observations separate from explanations. A wrong case ID shows a targeting error; it does not establish that a model confused two investor names. This demo supplies the wrong ID deliberately.

Give each label a boundary

The review groups the observed problems into four provisional labels: failed-write handling, duplicate operation, wrong-case write, and unsupported completion claim.

One trace can have several labels. The wrong-case run also claims completion on the intended case. Recording both preserves the operational error and the misleading confirmation.

Definitions need counterexamples. "Unsupported completion" initially sounds like "claims success when no task exists." The duplicate and wrong-case traces show why that is too narrow. The revised definition asks whether the claimed case, quantity, and action match the evidence. A task existing somewhere is insufficient.

A recorded review with an accurate explanation is a boundary example that should pass. The companion keeps that distinction explicit and records the definition change. A domain reviewer still needs to approve these proposed labels before they become an operational reference set.

Collect the cases this demo cannot answer

The current fixtures assume that missing evidence and conflicts have already been identified. They cannot test document interpretation or entity resolution.

For a fuller intake assistant, collect authorized examples that vary the applicable fund requirement, investor structure, document status, and available identifiers. Include incomplete submissions, apparent conflicts needing human assessment, similarly named entities, and ordinary successful cases. Have an investor-services reviewer check the expected handling against the fund's procedures.

Synthetic inputs can fill coverage gaps, but run them through the actual application and record their origin. Keep invented cases separate from observed incidents. A document mismatch should not automatically become a fraud finding or investor rejection.

Separate learning, regression, and measurement

A discovery collection helps find problems. Deliberately oversampling errors and ambiguous cases is useful here.

A regression set checks whether known failures return. Preserve the reviewed inputs, expected outcomes, and successful boundary cases. Keep related variants together when creating a separate holdout; paraphrases of a familiar case do not provide convincing evidence of generalization.

A representative measurement set answers a different question: how often does the system fail in a defined workload? Specify the population, time window, sampling method, and denominator. Keep it out of routine tuning. If you oversample a subgroup, report its results separately or account for the sampling design.

Our ten selected runs provide none of that population evidence. Their category counts describe these fixtures only.

For every dataset record, save the source trace, initial state, expected behavior, observed outcome, reviewer and review status, label definitions, and application version. Preserve unresolved judgments rather than forcing them into "pass."

Use the review package

Download the companion ZIP for the annotated traces, provisional taxonomy, annotation template, dataset schema, and saved evidence. The labels are AI-assisted proposals awaiting domain review.