> ## Documentation Index
> Fetch the complete documentation index at: https://laminar.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Turn Production Agent Failures into Evaluation Cases

The most valuable [agent evaluation](https://laminar.sh/docs/agent-evaluations/introduction.md) cases come from production: a trace where the agent did the wrong thing, replayed with the exact data that triggered it.

Agent evaluations exist to close the loop between what breaks in production and what you test before shipping:

1. **Find the traces.** Pick production traces with the failure you care about, plus a few that were correctly not flagged. If you track the failure with a [Signal](https://laminar.sh/docs/signals/introduction.md), query its events with `lmnr-cli signal list` and `lmnr-cli sql query`, or the [SQL editor](https://laminar.sh/docs/platform/sql-editor.md).
2. **Export the minimal input** for each into a `case.json`: the messages, tool calls, and records the app needs. Replace PII and secrets with plausible values, keep structure and ordering, and rebase timestamps.
3. **Set the reference answer** from the production outcome after a human check. Keep positive and negative cases paired against the same prompt and config.
4. **Copy a task directory**, drop in the new data and reference answer, and run `lmnr-cli eval run --task <new-case>`, then the whole suite. Commit it.

A new case should never need harness changes. If one does, generalize the shared seed or driver code so every case benefits.

## Next steps

> [card] [Track failures with Signals](https://laminar.sh/docs/signals/introduction.md)
>
> Detect a failure in production traces, then turn the flagged traces into cases.

> [card] [Build a production harness](https://laminar.sh/docs/agent-evaluations/production-harness.md)
>
> Run the real services, seed realistic data, and keep the eval honest.

> [card] [Compare runs](https://laminar.sh/docs/evaluations/comparing-runs.md)
>
> Every run of a suite is charted in its group. Diff two runs side by side to see which cases regressed.

> [card] [Run evals with lmnr-cli](https://laminar.sh/docs/agent-evaluations/running-evals.md)
>
> Local and remote modes, flags, CI, and sandbox limits.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.