> ## Documentation Index
> Fetch the complete documentation index at: https://laminar.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Write the Agent and Verifier for an Agent Evaluation

Every trial runs two pieces of your code: a Harbor agent that starts the stack and drives your app's entry point, and a verifier that grades what the app produced. Both sit on top of the [production harness](https://laminar.sh/docs/agent-evaluations/production-harness.md).

## The Harbor agent

Subclass Harbor's `BaseInstalledAgent`, so the agent's commands run inside the container next to your app. (`BaseAgent`, which runs on the host and reaches the container through `environment.exec`, is only for a product that can't run in the container.) `lmnr-cli` pins Harbor 0.25.0; see Harbor's [custom agents](https://docs.harborframework.com/agents/custom-agents) guide for the full interface.

* **`install()` stays empty** with a prebuilt image. It runs in every trial, and anything it would install belongs in the image. Per-trial setup goes at the start of `run()`.
* **`run()` stays thin**: start the stack, seed baseline and case data, run the driver.
* **Run commands with `exec_as_agent`**, or `exec_as_root` when they need root. The `agents[].env` values from `harbor.yaml` reach every command the agent runs, so use `env=` only for per-command extras.
* **Read the model from `self.model_name`**, so `--model` works.
* **Import only `harbor`, the standard library, and modules in the suite directory.** The class runs in the Harbor process (your machine or Laminar's orchestrator), not in the container; everything else belongs in the image.
* **Optionally override `populate_context_post_run(context)`**, which runs after the trial's logs are copied to the host, to parse `result.json` into the agent context.

## The driver

The driver the agent runs does two things: trigger, then wait for a terminal state.

* For a **synchronous** entry point (an HTTP call or a function that returns the result), call it with a timeout and record the response.
* For an **asynchronous** one (enqueue a job, the app writes a status later), poll the status the app writes until it's terminal, with a hard timeout. Don't sleep for a fixed time.
* Map the result to an outcome the verifier can grade, with `failed` and `timeout` as distinct outcomes so they're never mistaken for a model verdict. Write the full detail (status, error, ids, duration) to `/logs/agent/result.json`.
* Exit `0` whenever you produced an outcome, including `failed` and `timeout`. A non-zero exit makes the trial an error, which should mean the harness broke.

Keep the driver's timeout below the exec `timeout_sec`, and both below the task's `[agent] timeout_sec`.

## The verifier

`tests/test.sh` is the entry point. It can call anything: a Python script comparing against `expected.json`, a rubric, an LLM judge with its key in `[verifier.env]`. It writes the reward to `/logs/verifier/reward.txt`. A discrete outcome (pass or fail, event or no event) is the easiest place to start.

Where it runs is set by `environment_mode` under `[verifier]` in `task.toml`:

* **`"shared"`** (the default) runs in the agent's container after the agent finishes and can read anything the run left behind, such as `/logs/agent/result.json`.
* **`"separate"`** runs in a fresh container that sees only what the task declares: files listed in the top-level `artifacts = [...]`, plus snapshots of runtime state taken by `[[verifier.collect]]` commands before the environment is torn down. Use it when the verifier needs its own dependencies or must not see the app's internals.

Keep reference answers in `tests/`, never in `environment/`.

## Next steps

> [card] [Capture the trajectory](https://laminar.sh/docs/agent-evaluations/trajectories.md)
>
> Get the app's LLM and tool calls into every trial's trace.

> [card] [Build a production harness](https://laminar.sh/docs/agent-evaluations/production-harness.md)
>
> Run the real services, seed realistic data, and keep the eval honest.

> [card] [The evals folder](https://laminar.sh/docs/agent-evaluations/evals-folder.md)
>
> Reference for `lmnr.yaml`, `harbor.yaml`, `task.toml`, and secrets.

> [card] [Troubleshooting](https://laminar.sh/docs/agent-evaluations/troubleshooting.md)
>
> Fixes for stale images, missing env vars, and empty traces.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.