lmnr-cli eval run.
A change to the app or the image doesn't show up
A change to the app or the image doesn't show up
lmnr-cli rebuilds the lmnr.yaml images on every run, but Docker’s layer cache can still hide a change, for example a RUN step that downloads “latest”. Check the file inside a running container, then clear the cache with docker builder prune or bump a cache-busting ARG placed before the stale step.A service won't start as root
A service won't start as root
Some services, such as Postgres, refuse to run as root. Start them as a service user with
gosu or su, and make sure that user can write the log directory (chmod 777 /logs/agent).The app can't see an env var from harbor.yaml
The app can't see an env var from harbor.yaml
agents[].env reaches every command the agent runs. If the start script launches the app with env -i, it must forward each name explicitly.The trial trace has no LLM or tool calls
The trial trace has no LLM or tool calls
Either the app isn’t instrumented with Laminar, or it didn’t receive
LMNR_TRAJECTORY_PATH. See Capture the trajectory.Job output shows [REDACTED] in odd places
Job output shows [REDACTED] in odd places
A secret’s value is very short, so every occurrence of it in the output is redacted. Use long, distinctive placeholder values.
Where are the logs for a failed trial?
Where are the logs for a failed trial?
For local runs, in
evals/<suite>/jobs/<timestamp>/. For --remote runs, Harbor’s job output stays in the sandbox: open the trial’s trace in Laminar instead.Each local trial has agent/ (your logs and result.json), verifier/ (reward.txt, test-stdout.txt), and exception.txt with the agent’s traceback if it raised.Next steps
Run evals with lmnr-cli
Local and remote modes, flags, CI, and sandbox limits.
Capture the trajectory
Get the app’s LLM and tool calls into every trial’s trace.
Write the agent and verifier
Drive the entry point, wait for a terminal state, and grade the result.
Harbor integration
Already running
harbor run on a benchmark? Add the Laminar plugin directly.