Skip to main content
lmnr-cli eval run runs every agent evaluation suite in your evals/ folder, locally on Docker or in Laminar’s cloud sandboxes with --remote, and records each trial in Laminar with its trace. lmnr-cli eval run has two modes. Run it from the repo root: .env and .env.local are read from the current directory, and every mode needs Docker on the machine that runs lmnr-cli, because it builds the images there.
  • Local is the iteration loop. Run it after every change.
  • --remote scales out. lmnr-cli builds or pulls every image the selected suites use for linux/amd64, uploads only the layers Laminar doesn’t have yet, starts the run, and returns. Results appear in the project’s evaluations as trials finish. Harbor’s job output stays in the sandbox, so debug a failure that only happens remotely from the trial’s trace in Laminar.

​
In CI

Use --json and the exit code. A local run prints {mode, suites, failed, url}; a --remote run prints {mode: "remote", uploadId, sandboxId, suites, url} once the run has started. Set LMNR_PROJECT_API_KEY and the secrets your harbor.yaml references as CI secrets.

​
Remote limits and image requirements

  • One container per trial. docker-compose tasks don’t run remotely. cpus and memory_mb above the sandbox limit are clamped, and a run that exceeds the time limit is stopped.
  • The folder upload is capped at 1 GiB. Symlinks, .git, node_modules, virtualenvs, caches, and jobs/ directories are skipped, and .env* files are never uploaded. Keep large fixtures out of evals/.
  • Images need tar, gzip, and bash, plus su if a task runs as a non-root user. They’re built for linux/amd64 (emulated, slowly, on Apple silicon), so architecture-specific downloads should use ARG TARGETARCH.
  • Concurrency is capped per project (50 sandboxes by default). A higher --n-concurrent queues rather than running more at once.
  • Large images may need a higher build_timeout_sec under [environment] in task.toml (default 600 seconds), which bounds image preparation plus sandbox start.
  • Self-hosted Laminar: set LMNR_GRPC_PORT in your shell or .env for --remote runs, or the trial traces don’t arrive.

​
Next steps

The evals folder

Reference for lmnr.yaml, harbor.yaml, task.toml, and secrets.

Compare runs

Every run of a suite is charted in its group. Diff two runs side by side to see which cases regressed.

Read a failing trial

Open a low-scoring datapoint and read the app’s run as a transcript.

Troubleshooting

Fixes for stale images, missing env vars, and empty traces.