Hosted Evals for AI Agents with Harbor and lmnr-cli

​

Why hosted evals

• The outcome is state, not a return value. A support agent issues a refund, a coding agent edits files, a pipeline writes rows. Scoring the final message misses whether the right thing actually happened.
• The bugs are in the integration. The query that returns the wrong rows, the tool that times out, the retry loop, the config that’s only set in production. Mocks and fixtures in a unit-style eval hide exactly these.
• Production failures need their exact context. Reproducing a failure means the same conversation, the same records, the same account state. Each case needs its own data, and cases can’t share a database without interfering with each other.

• Production fidelity. The real code and the real services run in the container, started the way production starts them. You seed them; you don’t mock them.
• Isolation. Each case boots its own stack with its own data, so cases run in parallel with no shared state and no ordering.
• No production changes. The driver calls an entry point the app already has (an HTTP route, a CLI command, a queue message, an exported function), so you test the code path users hit.
• Local and cloud from one folder. lmnr-cli eval run runs trials on your Docker. Add --remote and the same folder runs in Laminar sandboxes, one per trial.
• No lock-in. Tasks, image, agent, and verifier are plain Harbor. Laminar adds the trajectories, run history, comparisons, SQL access, and managed execution.

​

Core concepts

• Task (case): one scenario. A directory with a task.toml (which image, resources, timeouts), an instruction.md, the case’s data in environment/, and a verifier in tests/. In production evals, tasks are variations of the same scenario: same code and services, different data.
• Environment: the container a trial runs in. Here it’s one prebuilt image holding your app, its services, and shared scripts. Each task’s environment/ files (for example case.json) are copied into it when the trial starts.
• Agent: the code that drives the trial. In Harbor’s original use this is a general-purpose agent like Claude Code. In hosted evals of your product, it’s a thin class that starts your stack, seeds the case, and calls your app’s entry point. Your AI agent is the app under test, not the Harbor agent.
• Verifier: tests/test.sh, run after the agent finishes. Any language, any grading method (exact match, a rubric, an LLM judge). It writes a reward, such as 1 or 0, to /logs/verifier/reward.txt.
• Trial: one attempt at one task: start the environment, run the agent, run the verifier. Run a task several times with --n-attempts to measure variance.
• Suite: a directory in evals/ with a harbor.yaml (a Harbor job config) and its tasks. Most projects have one suite.
• Trajectory: what the app did during the trial: its LLM calls and tool calls. Laminar shows it under the trial’s trace.

​

Quickstart

import os

from anthropic import Anthropic
from lmnr import Laminar, observe

Laminar.initialize()
client = Anthropic()

CATEGORIES = ["billing", "bug", "account", "other"]


@observe()
def triage(ticket: str) -> str:
    response = client.messages.create(
        model=os.environ.get("TRIAGE_MODEL", "claude-sonnet-5-5"),
        max_tokens=16,
        system=f"Classify the support ticket. Reply with one word: {', '.join(CATEGORIES)}.",
        messages=[{"role": "user", "content": ticket}],
    )
    answer = response.content[0].text.strip().lower()
    return answer if answer in CATEGORIES else "other"

npx lmnr-cli@latest setup

evals/
  lmnr.yaml                     # the image tasks use, and how to build it
  image/
    Dockerfile
    shared/
      run_case.py               # drives the entry point, writes result.json
  support/                      # a suite
    harbor.yaml
    agent/
      __init__.py               # empty
      agent.py
    tasks/
      refund-double-charge/
        task.toml
        instruction.md
        environment/
          case.json             # this case's data
        tests/
          test.sh               # the verifier
          expected.json         # the reference answer

images:
  triage-evals:
    dockerfile: image/Dockerfile
    context: ..

FROM python:3.12-slim-bookworm
COPY requirements.txt /srv/app/
RUN pip install --no-cache-dir -r /srv/app/requirements.txt
COPY evals/image/shared/ /opt/harness/
COPY app/ /srv/app/app/
WORKDIR /case

import json
import sys
import time

sys.path.insert(0, "/srv/app")
from app.triage import triage  # the production code path, unchanged

case = json.load(open("/case/case.json"))
started = time.time()
try:
    result = {"outcome": triage(case["ticket"])}
except Exception as e:
    result = {"outcome": "failed", "error": repr(e)}
result["duration_s"] = round(time.time() - started, 2)

json.dump(result, open("/logs/agent/result.json", "w"), indent=2)
print(json.dumps(result))

agents:
  - import_path: agent.agent:TriageAgent
    model_name: anthropic/claude-sonnet-5-5
    env:
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
datasets:
  - path: tasks
n_concurrent_trials: 4

from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext


class TriageAgent(BaseInstalledAgent):
    @staticmethod
    def name() -> str:
        return "triage"

    async def install(self, environment: BaseEnvironment) -> None:
        pass  # everything the agent needs is already in the image

    @with_prompt_template
    async def run(
        self, instruction: str, environment: BaseEnvironment, context: AgentContext
    ) -> None:
        # "anthropic/claude-sonnet-5-5" -> "claude-sonnet-5-5"
        model = (self.model_name or "").split("/", 1)[-1]
        await self.exec_as_agent(
            environment,
            command="python /opt/harness/run_case.py",
            env={"TRIAGE_MODEL": model},
            timeout_sec=120,
        )

[environment]
docker_image = "triage-evals"
workdir = "/case"

[agent]
timeout_sec = 180

Triage the support ticket in case.json.

{"ticket": "Hi, I was charged twice for my October invoice (two identical charges of $49 on Oct 3). Can you refund one of them?"}

{"category": "billing"}

#!/bin/bash
expected=$(python3 -c 'import json; print(json.load(open("/tests/expected.json"))["category"])')
actual=$(python3 -c 'import json; print(json.load(open("/logs/agent/result.json"))["outcome"])')

if [ "$expected" = "$actual" ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo "expected $expected, got $actual"
  echo 0 > /logs/verifier/reward.txt
fi

.env
evals/*/jobs/

npx lmnr-cli@latest eval run

​

Run evals with lmnr-cli

• Local is the iteration loop. Run it after every change.
• --remote scales out. lmnr-cli builds or pulls every image the selected suites use for linux/amd64, uploads only the layers Laminar doesn’t have yet, starts the run, and returns. Results appear in the project’s evaluations as trials finish. Harbor’s job output stays in the sandbox, so debug a failure that only happens remotely from the trial’s trace in Laminar.

lmnr-cli eval run                                    # every suite in ./evals
lmnr-cli eval run --suite support --task "refund-*" --n-attempts 3
lmnr-cli eval run --task refund-double-charge        # one case
lmnr-cli eval run --remote --n-concurrent 20
lmnr-cli eval run --remote --model anthropic/claude-sonnet-5-5

​

In CI

​

Remote limits and image requirements

• One container per trial. docker-compose tasks don’t run remotely. cpus and memory_mb above the sandbox limit are clamped, and a run that exceeds the time limit is stopped.
• The folder upload is capped at 1 GiB. Symlinks, .git, node_modules, virtualenvs, caches, and jobs/ directories are skipped, and .env* files are never uploaded. Keep large fixtures out of evals/.
• Images need tar, gzip, and bash, plus su if a task runs as a non-root user. They’re built for linux/amd64 (emulated, slowly, on Apple silicon), so architecture-specific downloads should use ARG TARGETARCH.
• Concurrency is capped per project (50 sandboxes by default). A higher --n-concurrent queues rather than running more at once.
• Large images may need a higher build_timeout_sec under [environment] in task.toml (default 600 seconds), which bounds image preparation plus sandbox start.
• Self-hosted Laminar: set LMNR_GRPC_PORT in your shell or .env for --remote runs, or the trial traces don’t arrive.

​

The evals folder

evals/
  README.md                       # how to run, how to add a case
  lmnr.yaml
  image/
    Dockerfile                    # services, app, shared scripts, baseline data
    shared/
      start_stack.sh              # boot dependencies + the app, wait for readiness
      seed_baseline.*             # (optional) data every case shares
      seed_case.*                 # load this case's data from environment/
      run_case.*                  # drive the entry point, write result files
      export_trajectory.*         # (optional) rebuild the trajectory if the app isn't instrumented
  <suite>/
    harbor.yaml
    agent/
      __init__.py
      agent.py
    tasks/
      <case>/
        task.toml
        instruction.md
        environment/
          case.json               # any format seed_case reads
        tests/
          test.sh
          expected.json           # (optional) reference answer

​

lmnr.yaml and the image

• Order layers from rarely to often changing: system packages, then dependencies, then shared scripts, then app code. An app change then rebuilds, and for --remote uploads, one small layer.
• COPY specific paths, never COPY . . from the repo root. Docker only sends what COPY references, so a large repo costs nothing, while COPY . . would bake in everything, including every suite’s tests/ and their reference answers. From evals/, copy image/shared/ only.
• Build from the live source, not a vendored copy, so the app and its migrations always come from the same revision as the evals.

​

harbor.yaml

​

task.toml

[environment]
docker_image = "myapp-evals"
workdir = "/case"          # where environment/ is uploaded; defaults to the image's WORKDIR
cpus = 2
memory_mb = 4096

[agent]
timeout_sec = 600

​

Secrets and env vars

• agents[].env in harbor.yaml, for the agent and the app it starts.
• [verifier.env] in task.toml, for example the key an LLM-graded verifier needs.
• [environment.env] in task.toml, for the container.

​

Build a harness that matches production

​

Principles

1. Production fidelity over convenience. Run the real code and the real services for as much of the stack as you can. Seed them; don’t mock them. Mock only what you can’t run (a third-party SaaS, a private internal service), at the network edge through the app’s existing base-URL config, never by editing the code path. Every mock is a gap in what the eval measures, so write each one down.
2. Zero production changes. Drive an entry point the app already exposes. A harness-only code path tests the harness. If nothing existing reaches the behavior you want to test, treat adding an entry point as a deliberate decision (for example an internal endpoint disabled unless a config flag is set), not something to slip in.
3. One case, one environment. Every container boots its own stack with its own data. Never share a database between cases.
4. Evals live next to the code, built from the same source tree and revision.
5. Turn the run into artifacts, then verify the artifacts. The driver leaves its observable result in files (a result.json, an exported row, a transcript), and the verifier grades those.

​

Find the entry point and the outcome

• the entry point you can call without modification (a route, a CLI command, a queue consumer, a function);
• the inputs it reads (rows, files, messages) and where they live;
• the terminal state it produces (a response body, a status row, an emitted record) and how to read it;
• the external calls it makes (LLM, HTTP) and how they’re configured.

​

Boot the services in the container

• Databases start as background processes with the engine’s own bootstrap (initdb, redis-server, clickhouse server). Install the version production runs, from the distro, the vendor’s apt repo, or a multi-stage COPY --from=<official image>:<version>.
• Schemas are created the way the app’s own startup creates them: roles, databases, migrations in order, then anything created after migrations in code (views, grants, seed rows). If the app has no migrations, create only the tables the path under test needs, and note that the schema was inferred.
• Queues, caches, and object stores with an in-memory or local-disk fallback can use it. That’s production-supported code, not a mock.
• The app runs as its production artifact: the real package, or a compiled binary built in a builder stage of the same Dockerfile (a binary built on a macOS host is the wrong format).
• Readiness: poll the health endpoint with a timeout, and on failure print the app log to stderr so it shows up in the job output.

​

Seed the data

• Shared baseline data is the same for every case: reference tables, a default org and user, config rows, an LLM profile. Seed it in the image build or in a shared script that runs at startup. Each container still gets its own copy.
• Per-case data is what reproduces one scenario: the conversation, the records, the account state. It lives in the task’s environment/ and is loaded by the same shared seed script for every case.

​

Make seed data look real

• Timestamps. Data exported weeks ago reads as stale when replayed today. Default to relative timestamps (shift on load so the latest event is near now, keeping the intervals). Keep absolute dates only when the case depends on one, such as an end-of-month or DST boundary. Check dates outside timestamp columns too: free text, filenames, and time-ordered ids (UUIDv7, ULID).
• Ids. Patterned ids like user_1 or 12000000-0000-4000-8000-0123456789ab give the game away. Generate them the way the app does and keep references consistent.
• Names and content. No John Doe, [email protected], Acme, or lorem ipsum. Replace scrubbed PII with plausible values, not placeholders.
• The harness itself. If the agent can see the filesystem, process list, or environment, keep harness files out of sight or neutrally named: no /eval, EVAL_MODE=1, or test_case in anything it can read.

​

Write the agent

• install() stays empty with a prebuilt image. It runs in every trial, and anything it would install belongs in the image. Per-trial setup goes at the start of run().
• run() stays thin: start the stack, seed baseline and case data, run the driver.
• Run commands with exec_as_agent, or exec_as_root when they need root. The agents[].env values from harbor.yaml reach every command the agent runs, so use env= only for per-command extras.
• Read the model from self.model_name, so --model works.
• Import only harbor, the standard library, and modules in the suite directory. The class runs in the Harbor process (your machine or Laminar’s orchestrator), not in the container; everything else belongs in the image.
• Optionally override populate_context_post_run(context), which runs after the trial’s logs are copied to the host, to parse result.json into the agent context.

• For a synchronous entry point (an HTTP call or a function that returns the result), call it with a timeout and record the response.
• For an asynchronous one (enqueue a job, the app writes a status later), poll the status the app writes until it’s terminal, with a hard timeout. Don’t sleep for a fixed time.
• Map the result to an outcome the verifier can grade, with failed and timeout as distinct outcomes so they’re never mistaken for a model verdict. Write the full detail (status, error, ids, duration) to /logs/agent/result.json.
• Exit 0 whenever you produced an outcome, including failed and timeout. A non-zero exit makes the trial an error, which should mean the harness broke.

​

Use the real model

​

Write the verifier

• "shared" (the default) runs in the agent’s container after the agent finishes and can read anything the run left behind, such as /logs/agent/result.json.
• "separate" runs in a fresh container that sees only what the task declares: files listed in the top-level artifacts = [...], plus snapshots of runtime state taken by [[verifier.collect]] commands before the environment is torn down. Use it when the verifier needs its own dependencies or must not see the app’s internals.

​

Capture the trajectory

• Each message is a step with a sequential step_id (1, 2, 3…), a source (system, user, or agent), a message, and an ISO 8601 timestamp.
• model_name, tool_calls, and metrics are only allowed on agent steps. Tool results go in that step’s observation.results, paired by source_call_id.
• History seeded from case.json gets is_copied_context: true, so it’s shown but not mistaken for the agent’s work.
• Leave metrics unset rather than estimating: stored history usually lacks token counts, costs, and the exact prompt of each call. Say where the trajectory came from in notes.

​

Check that the harness can fail

• Run a positive and a negative case that share the same prompt and config and differ only in input.
• Flip a reference answer once and confirm the reward drops to 0. A harness that can’t fail is broken.
• Open a trial’s trace and confirm the LLM and tool calls are there.
• Run --remote once to confirm the folder works in Laminar sandboxes.

​

Turn production failures into cases

1. Find the traces. Pick production traces with the failure you care about, plus a few that were correctly not flagged. If you track the failure with a Signal, query its events with lmnr-cli signal list and lmnr-cli sql query, or the SQL editor.
2. Export the minimal input for each into a case.json: the messages, tool calls, and records the app needs. Replace PII and secrets with plausible values, keep structure and ordering, and rebase timestamps.
3. Set the reference answer from the production outcome after a human check. Keep positive and negative cases paired against the same prompt and config.
4. Copy a task directory, drop in the new data and reference answer, and run lmnr-cli eval run --task <new-case>, then the whole suite. Commit it.

​

Troubleshooting

​

Next steps