The Evals Folder Reference for lmnr.yaml, harbor.yaml, and task.toml

This page is the reference for the files lmnr-cli eval run reads. For a guided first suite, start with the quickstart. lmnr-cli eval run treats every directory directly inside evals/ that has a harbor.yaml as a suite. A fuller layout, for an app with services and a database:

evals/
  README.md                       # how to run, how to add a case
  lmnr.yaml
  image/
    Dockerfile                    # services, app, shared scripts, baseline data
    shared/
      start_stack.sh              # boot dependencies + the app, wait for readiness
      seed_baseline.*             # (optional) data every case shares
      seed_case.*                 # load this case's data from environment/
      run_case.*                  # drive the entry point, write result files
      export_trajectory.*         # (optional) rebuild the trajectory if the app isn't instrumented
  <suite>/
    harbor.yaml
    agent/
      __init__.py
      agent.py
    tasks/
      <case>/
        task.toml
        instruction.md
        environment/
          case.json               # any format seed_case reads
        tests/
          test.sh
          expected.json           # (optional) reference answer

lmnr.yaml and the image

lmnr.yaml lists the images tasks can use, keyed by the name a task puts in docker_image:

evals/lmnr.yaml

images:
  myapp-evals:                  # referenced as docker_image = "myapp-evals"
    dockerfile: image/Dockerfile
    context: ..                 # the repo root, so COPY can reach the app's source
  myapp-evals-browser:          # a second image, e.g. with a headless browser
    dockerfile: image/browser/Dockerfile
    # no context: builds from image/browser/

Key	Required	What it does
`images.<name>`	Yes	The image name tasks reference with `docker_image`.
`images.<name>.dockerfile`	Yes	Path to the Dockerfile, relative to `evals/`.
`images.<name>.context`	No	Docker build context, relative to `evals/`. Defaults to the Dockerfile’s directory.

A run builds only the images its selected tasks reference, so an unused entry costs nothing. lmnr-cli rebuilds them on every run (a cached build when nothing changed), so there are no tags to bump and no registry to push to. lmnr.yaml is optional: without it, tasks pull docker_image from a registry or build their own environment/Dockerfile. When you write the Dockerfile:

Order layers from rarely to often changing: system packages, then dependencies, then shared scripts, then app code. An app change then rebuilds, and for --remote uploads, one small layer.
COPY specific paths, never COPY . . from the repo root. Docker only sends what COPY references, so a large repo costs nothing, while COPY . . would bake in everything, including every suite’s tests/ and their reference answers. From evals/, copy image/shared/ only.
Build from the live source, not a vendored copy, so the app and its migrations always come from the same revision as the evals.

.dockerignore applies, and builds that lmnr-cli runs never receive .git or .env* files.

harbor.yaml

A suite’s harbor.yaml is a plain Harbor job config: agents, datasets, n_concurrent_trials, and so on. Relative paths resolve from the suite directory, and the suite directory is on PYTHONPATH, so import_path: agent.agent:ClassName points at <suite>/agent/agent.py.

task.toml

Set docker_image to an image name from lmnr.yaml and leave the Dockerfile out of environment/. Every trial then starts from the same built image, and Harbor uploads the contents of environment/ into workdir after the container starts. That is how case.json reaches a shared image.

task.toml

[environment]
docker_image = "myapp-evals"
workdir = "/case"          # where environment/ is uploaded; defaults to the image's WORKDIR
cpus = 2
memory_mb = 4096

[agent]
timeout_sec = 600

A full stack per case is heavy, so set cpus and memory_mb. Locally all trials share your machine, so keep n_concurrent_trials to what it can run; remotely each trial gets its own sandbox, so raise concurrency for that run with --n-concurrent. An image name that isn’t in lmnr.yaml is pulled from its registry, so public images work as-is. A per-task Dockerfile in environment/ is the exception, for a case that genuinely needs a different system setup. Remove docker_image from that task: while it’s set, Harbor uses the image and ignores the Dockerfile. With a Dockerfile, environment/ is the build context and nothing is uploaded at start, so the Dockerfile has to COPY the case data itself. It also can’t FROM an image that only exists in lmnr.yaml; start from a public base.

Secrets and env vars

Declare every secret and setting as a ${VAR} placeholder:

agents[].env in harbor.yaml, for the agent and the app it starts.
[verifier.env] in task.toml, for example the key an LLM-graded verifier needs.
[environment.env] in task.toml, for the container.

${VAR:-default} makes one optional. Values come from your shell first, then .env.local / .env in the current directory. For --remote they travel with the run as env vars; files containing secrets are never uploaded. Commit a .env.example listing the names so teammates know what to fill in.

Harbor redacts agent env values from its output files. A very short value, like a one-character placeholder x, turns every x in the job output into [REDACTED]. Keep placeholder values long and distinctive.

Next steps

Run evals with lmnr-cli

Local and remote modes, flags, CI, and sandbox limits.

Build a production harness

Run the real services, seed realistic data, and keep the eval honest.

Write the agent and verifier

Drive the entry point, wait for a terminal state, and grade the result.

Troubleshooting

Fixes for stale images, missing env vars, and empty traces.

Overview

Tracing

Signals

Debugger

Agent evaluations

LLM evaluations

Datasets

Platform

The Evals Folder Reference for lmnr.yaml, harbor.yaml, and task.toml

lmnr.yaml and the image

harbor.yaml

task.toml

Secrets and env vars

Next steps

Run evals with lmnr-cli

Build a production harness

Write the agent and verifier

Troubleshooting

​lmnr.yaml and the image

​harbor.yaml

​task.toml

​Secrets and env vars

​Next steps

Run evals with lmnr-cli

Build a production harness

Write the agent and verifier

Troubleshooting

lmnr.yaml and the image

harbor.yaml

task.toml

Secrets and env vars

Next steps