Skip to main content
The most valuable agent evaluation cases come from production: a trace where the agent did the wrong thing, replayed with the exact data that triggered it. Agent evaluations exist to close the loop between what breaks in production and what you test before shipping:
  1. Find the traces. Pick production traces with the failure you care about, plus a few that were correctly not flagged. If you track the failure with a Signal, query its events with lmnr-cli signal list and lmnr-cli sql query, or the SQL editor.
  2. Export the minimal input for each into a case.json: the messages, tool calls, and records the app needs. Replace PII and secrets with plausible values, keep structure and ordering, and rebase timestamps.
  3. Set the reference answer from the production outcome after a human check. Keep positive and negative cases paired against the same prompt and config.
  4. Copy a task directory, drop in the new data and reference answer, and run lmnr-cli eval run --task <new-case>, then the whole suite. Commit it.
A new case should never need harness changes. If one does, generalize the shared seed or driver code so every case benefits.

​
Next steps

Track failures with Signals

Detect a failure in production traces, then turn the flagged traces into cases.

Build a production harness

Run the real services, seed realistic data, and keep the eval honest.

Compare runs

Every run of a suite is charted in its group. Diff two runs side by side to see which cases regressed.

Run evals with lmnr-cli

Local and remote modes, flags, CI, and sandbox limits.