Your AI agent said "done." Did anything actually happen?

A confirmation without a task

The assistant replies:

Created one bank-evidence correction task for CASE-101. No investor message was sent.

The wording is precise. It names the case and distinguishes an internal task from a message to the investor. It would look reassuring in a demo.

But the task table is empty.

That is the failure I want an agent evaluation to catch. Reading the answer tells us what the assistant claims. We need evidence from the application to establish what happened.

For this article, I used a small Python and SQLite example with fictional cases. The workflow responses are scripted, and the failures are deliberately injected. It makes no model calls. Its purpose is to show how to test the actions around an agent, including the cases where its final message sounds correct.

The example tests one internal step in an investor-services workflow. The analyst has already established that the evidence is required and outstanding. The script does not decide which documents a fund should require, contact an investor, or approve an application. It is a separate test harness, not a report of failures in a deployed client system.

Does this happen in fund administration?

There is a concrete basis for the example. The Harding Loevner Funds plc application, administered by Northern Trust, requires bank-account evidence for relevant applicants and specifies an exemption (page 2). That is a particular fund's requirement, not a rule for every investor.

Here, the evidence concerns the investor's nominated account. It is separate from the fund's account receiving subscription money, and from evidence explaining the source of the investor's wealth or funds.

The distinction between acknowledgment and completion also appears in actual dealing procedures. The same application's email addendum separates a task-number acknowledgment from placement of an instruction and warns about duplicate submissions (page 22). Those provisions concern dealing instructions; they do not establish the administrator's internal process for chasing documents.

Our task table is a proposed implementation of that internal follow-up. An administrator might instead use a case-management queue, an outstanding-document list, or an existing service system. The evaluation should query whichever record actually controls the work.

Give "done" a specific meaning

In this example, "correction task" is the local software label for following up on an outstanding document. There is no incorrect document to correct yet. Recording that follow-up and asking the investor for the document are separate actions.

An administrator might mean that someone drafted the request. Or that an open task exists against the correct investor case. Or that the investor can see it in the portal. "Sent" adds another claim about delivery. None of those establishes that the investor supplied the document, that a reviewer accepted the evidence, or that the investor was admitted to the fund.

I would keep those states separate in both the application and its evaluations. A document request should not silently advance an admission decision.

For the example here, the instruction is deliberately narrow: create one internal task for the confirmed outstanding bank-account evidence on CASE-101. This identifier represents one application in one fund, not every relationship the investor has. Leave bank instructions and the application decision unchanged. Do not send an external message.

The expected result is an open task with the right case ID and document requirement. Repeating the same logical request must not create another task. If the operation cannot complete, the workflow should record the problem for review and explain that the correction task remains uncreated.

That last condition matters. A workflow can handle an error correctly while leaving the user's original task unfinished. Those are two different results, and the evaluation should report both.

The wider process still needs an authorized request through the agreed communication channel, receipt of the evidence, and assessment against the applicable requirement. Uploading a file should move it into review; it should not automatically clear the outstanding item. Completing that item also does not, by itself, authorize acceptance of the subscription. HedgeServ's portal terms explicitly distinguish submission of transaction documents from written acceptance on behalf of the fund. The responsible parties and acceptance process depend on the fund's arrangements.

Follow the action past the tool call

There are several places to inspect when a confirmation does not match the record.

Start with tool selection. Did the assistant choose an operation that creates a task, or one that only drafts request text? Then inspect the arguments. A valid case ID can still identify the wrong investor. Next, check execution: did the service reject the write, commit it, or lose the response after committing it?

Finally, compare the confirmation with the evidence returned to the assistant. An error payload should not become a success message just because the tool call finished.

A trace helps locate the failure. The stored records tell us whether the outcome satisfies the request. Anthropic makes the same distinction in its guide to agent evaluations: the transcript records the interaction, while the outcome is the resulting state of the environment.

In the example, a storage failure occurs before the task is written. The baseline workflow catches the error and still produces the confirmation shown above. A separate database connection finds zero tasks. The outcome check rejects the run.

There is no reason to ask another language model whether that task exists. The application can answer the question directly.

A timeout can leave a completed write behind

A timeout creates a different problem. It tells the caller that it did not receive a timely response. It does not establish whether the write happened.

The second injected failure follows this sequence:

Create correction task for CASE-101
Database commits task 1
Response is lost; caller receives a timeout
Caller retries
Database commits task 2
Assistant confirms that one task was created

Now the task table contains two open requests for the same missing evidence.

In a larger system, those tasks might trigger separate reminders or appear in two people's queues. This local example stops at the task records; it does not simulate those downstream effects. The duplicate is already enough to fail the contract.

A prompt telling the agent to "avoid duplicates" leaves too much to interpretation. The write operation needs a stable identity for the logical request.

The repaired version carries the same operation ID across the retry. A unique constraint prevents two tasks from sharing that ID. When the retry finds the existing task, it checks that the case and document requirement match, then returns the existing task ID.

The operation ID belongs to this request. Using a permanent key such as "investor plus bank document" could incorrectly suppress a legitimate future request for updated evidence.

The example exercises a sequential retry. A production implementation also needs to handle concurrent calls, conflicting payloads, and failures between related writes. It should check whether another colleague already opened a follow-up for the same outstanding item. A new operation ID on a repeated instruction would bypass this example's retry protection. That business-level duplicate check needs to distinguish an existing request from a legitimate reminder or a new evidence requirement.

Check the case, not just the existence of a task

The wrong-record scenario creates a task successfully, but attaches it to CASE-202. The instruction was for CASE-101.

An assertion that checks only whether the task count increased would pass this run. So would a grader that trusted the assistant's confirmation without consulting the record.

The repair binds the proposed write to the case selected in the trusted application context. If the proposed case ID differs, the tool rejects it and the workflow records a review item against the intended case.

This is a deliberately narrow control. It catches a mismatch between a proposal and the selected case. It cannot establish that the administrator selected the right investor in the first place. A deployed workflow must also bind the instruction to the fund, registered holder, application, and the user's permitted scope. Investor name matching alone would be insufficient.

For the successful creation path, the outcome assertions can be expressed as the following pseudocode:

assert len(tasks_for_this_operation) == 1
assert task.case_id == expected_case_id
assert task.kind == "bank_evidence"
assert task.status == "open"
assert other_case_after == other_case_before
assert admission_after == admission_before
assert bank_details_after == bank_details_before

The expected case comes from the test fixture, independently of the assistant's answer. Otherwise, the evaluator risks accepting the same wrong ID that caused the failure.

These assertions specify the result. They do not require the agent to follow one exact sequence of tool calls. Reading the case first, reusing an existing task, or recovering from a timeout can all be acceptable if the resulting state meets the contract.

Where the process itself matters, test it separately. If an action requires independent approval, a correct final record does not prove that approval occurred before the action. The evaluation needs ordered evidence of the approval and the identity of the person who gave it.

Sometimes the right result is a review item

The fifth fixture represents conflicting bank evidence that a reviewer has already identified. The script routes that predefined case to review instead of creating a routine missing-document task. It does not read documents or discover the conflict itself.

The expected result is a review item on CASE-101 with the conflict recorded. Bank details remain unchanged, the application decision stays pending, and the response says that review is required. A mismatch does not automatically establish fraud: the reviewer must consider the account structure and applicable requirements.

The script uses one review table for simplicity. Operationally, a database outage should go to technical support or an operations exception queue. Conflicting documents should go to the investor-services reviewer or the relevant specialist under the firm's escalation rules. Neither event should be silently converted into an investor rejection. The demo records reasons but does not implement those separate queues or their ownership.

"I escalated this" needs evidence too. The test checks for the review record. In a deployed workflow, I would also check its owner or queue and whether it is visible to the people responsible for handling it.

This is where a single completion score becomes awkward. Creating a task is correct in the normal case. Creating that same task is wrong in the conflict case. The scenario's requirements determine the acceptable outcome.

What the local runs showed

I ran five scripted scenarios against the baseline and repaired versions. Each run saved its trace and final state in a separate SQLite database. The evaluator read the database through a fresh connection after execution.

The repaired workflow satisfies all five scenario contracts. It completes the correction-task request in two: normal execution and recovery from the lost response. The other three finish with recorded review work and no correction task.

Those counts describe deliberately constructed software tests. They say nothing about how often a model would choose the wrong case, ignore an error, or retry in production. Measuring that would require running the actual agent on representative inputs and evaluating its behavior across repeated trials where variability matters.

The repairs are ordinary engineering changes: preserve error status, check the target case, reuse the operation ID, and make the confirmation reflect the outcome. The evaluations give us a way to show that each change addresses the intended failure.

A bank-details change is a different workflow

Collecting missing evidence during onboarding is different from changing payment instructions already held on file. The latter can alter where money goes.

The Harding Loevner dealing form requires an authorized instruction, a change form, and callback verification to revise bank details. Other arrangements can differ. Pinnacle Fund Services describes both traditional review procedures and portal-based changes subject to confirmation and required approvals.

I would evaluate that as its own workflow. Receiving the instruction, verifying it, approving it, activating the revised bank record, and using it in a payment each needs separate evidence. This article's task-creation test covers none of those steps.

The useful habit is to take the verb in the assistant's answer, such as "created," "sent," "resolved," or "approved," and identify the evidence that would make it true.

I would start with one action in one workflow. Save its initial state, run the agent, and inspect the resulting records independently. Then repeat the test with a failed write, an uncertain response, and the wrong target record.

Run the example

Download the companion ZIP for the Python script, saved traces, SQLite databases, and a result summary. It uses Python 3 and the standard library. Extract the archive, open its folder, and run:

python3 workflow_demo.py

Each execution creates a new directory under runs/. No API key is required, and the script makes no network calls. The included fixtures are fictional. The source script and saved results are also available separately.

The error-analysis approach draws on Hamel Husain and Shreya Shankar's AI Evals for PMs & Engineers course. This example applies those ideas to a narrowly scoped investor-services operation.