Agent Evaluations in Real Environments with Laminar

Agent evaluations test your agent end to end, the way it runs in production. Every eval case gets its own isolated container: your real application, its real services (database, queue, cache), and the data that reproduces one specific scenario. A driver calls the app through an entry point it already has, and a verifier scores what the agent left behind. You run them with lmnr-cli eval run, locally on Docker or in Laminar’s cloud sandboxes, and every trial lands in Laminar with its score next to the agent’s full trace.

​

Agent evaluations and LLM evaluations

Laminar has two ways to evaluate, named after what they look at:

• Agent evaluations (this section) set up a working environment for the agent, let it run, and score the whole outcome, including the state changes, and the agent trajectory.
• LLM evaluations call a function in your process with evaluate(), then compare its output to target data with evaluator functions.

You can evaluate an agent with LLM evaluations, and for a prompt, a classifier, or a short pipeline that’s often the fastest option. Once the agent is large, uses tools, and acts on real systems, the answer to did it do the right thing lives in the environment and in the trace, not in a single return value. That’s where agent evaluations fit. The two coexist in the same project and share the same evaluation views, run comparison, and SQL access.

​

Why evaluate the agent in its environment

Unit-style evals become hard once the thing you’re testing is an agent that acts on the world:

• The outcome is not just the final message. Agent may issue refunds, write database rows, and affect the world around them. Scoring just the final message misses whether the right thing actually happened, and more importantly whether bugs not happened.
• The bugs are in the integration. Queries time out; downstream services may experience transient issues; agents may or may not self-heal. Fixtures hide these kinds of issues.
• Production failures need their exact context. Reproducing a failure means the same conversation, records, and account state. Each case needs its own data, and cases can’t share a database without interfering with each other.

Agent evaluations answer all three by giving every case a fresh, complete copy of the stack:

• Production fidelity. The real code and the real services run in the container, started the way production starts them. You seed them and mock them only if absolutely necessary.
• Isolation. Each case boots its own stack with its own data, so cases run in parallel with no shared state and no ordering.
• No or minimal production changes. The driver calls an entry point the app already has (an HTTP route, a CLI command, a queue message, an exported function), so you test the code path users hit.
• Local and cloud from one folder. lmnr-cli eval run runs trials on your Docker. Add --remote and the same folder runs in Laminar sandboxes, one per trial.
• No lock-in. The case format is Harbor’s, so tasks, image, agent, and verifier are plain Harbor. Laminar adds the trajectories, run history, comparisons, SQL access, and managed execution.

​

How a run works

You keep an evals/ folder next to your code. Each case in it is a Harbor task: which image to start, the data for this scenario, and a verifier. When you run lmnr-cli eval run, Laminar:

1. Builds your image from the live source tree.
2. Starts one container per trial, uploads the case’s data, and runs the driver against your app.
3. Runs the verifier over what the app produced and records its reward as the score.
4. Attaches the app’s LLM and tool calls to the trial’s trace, so a failing score takes one click to explain.

Every run of a suite is charted in one evaluation group, local and remote together, so a regression shows up the moment the run lands. Concepts maps each piece to what you see in Laminar.

​

Next steps