How to turn AI failures into an evaluation dataset
Turn reviewed AI workflow failures into an evaluation dataset, with concrete annotations, provisional labels, and separate discovery, regression, and measurement sets.
Insights
Notes on building AI systems that can do real work.
Practical writing about workflow design, control boundaries, and how to evaluate complete systems—not just models.
Your AI agent said "done." Did anything actually happen?
A fund-admin follow-up task shows how to evaluate agent outcomes, catch failed writes, and prevent duplicate retries. Includes a runnable Python example.
How we know an AI system is ready for real work
How to evaluate probabilistic AI inside workflows that demand permissions, evidence, reliable actions, and human judgment.