How to Run Terminal-Bench with Harbor and Debug Failed Trials

Terminal-Bench is a benchmark of tasks an AI agent must complete in a real terminal. The Harbor framework runs each task in a sandbox, gives the instruction to an agent, then runs the task's verifier and records a reward. In our six-task run, Claude Code found a lost Git commit but asked for permission instead of merging it. On another task, it claimed its HTML filter preserved clean pages; the verifier found that it changed them.

Harbor reported a mean reward of 0.667: four passes, two failures, zero exceptions. That score doesn't distinguish an unfinished merge from a broken filter. This guide covers installing Harbor, running a small Terminal-Bench sample, and using Laminar traces to debug those two failures. We used Claude Code with Claude Sonnet 5, then compared it with Opus 5, on Harbor 0.24.0 and lmnr 0.7.67.

What Terminal-Bench and the Harbor framework are#

Terminal-Bench 2.0 has 89 tasks, from recovering lost Git commits to writing an XSS filter. Each task is a directory: an instruction.md in plain English, an environment/ with the Dockerfile the agent works in, a tests/ folder, a reference solution/, and a task.toml with metadata and timeouts.

The filter-js-from-html task, for example, gives the agent 30 minutes and the verifier 15. The verifier runs tests/test.sh, which usually invokes pytest to check the container's end state: file contents, script output or repository state. The agent never sees the tests.

Harbor runs agents on datasets like this one. Terminal-Bench 2.0 is in its registry as terminal-bench@2.0, so you don't need to clone the tasks yourself. Harbor ships adapters for agents including claude-code and codex. It runs each task in Docker on your machine by default, or with a cloud sandbox provider such as Daytona using --env daytona.

You'll see these terms in the results:

• A job is one harbor run invocation: one agent and model on a set of tasks.
• A trial is one agent attempt at one task, scored by that task's verifier.
• A reward is the verifier's score for a trial. On Terminal-Bench it is 1.0 or 0.0.

How to install Harbor with the Laminar plugin#

Harbor installs as a Python tool. Laminar's third-party plugin ships in Laminar's Python SDK (lmnr). It is not an official Harbor integration, so install the two packages together:

uv tool install harbor --with lmnr

For Daytona instead of local Docker, install Harbor's Daytona extra and set DAYTONA_API_KEY:

uv tool install 'harbor[daytona]' --with lmnr

The plugin needs a Laminar project API key, available under Project Settings → Project API keys. Your agent also needs its provider key: ANTHROPIC_API_KEY for Claude Code.

export LMNR_PROJECT_API_KEY=your-project-api-key
export ANTHROPIC_API_KEY=your-anthropic-key

Check that Harbor found the plugin:

harbor plugins list

                   Harbor Plugins
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name    ┃ Import path                            ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ laminar │ lmnr.integrations.harbor:LaminarPlugin │
└─────────┴────────────────────────────────────────┘

The plugin defaults to Laminar Cloud. Its options are in the Laminar Harbor integration docs. Harbor's own documentation, at harborframework.com, covers agents, environments and datasets.

How to run Terminal-Bench: a few tasks first#

Start with a handful of tasks. Every trial spends model tokens, and a long task can run for half an hour before its verifier starts. Pick tasks by name with -i, or take the first few with -l:

harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-sonnet-5 \
  --env daytona \
  -i fix-git -i overfull-hbox -i regex-log \
  -i sqlite-db-truncate -i chess-best-move -i filter-js-from-html \
  -n 6 \
  --job-name tb2-sonnet5 \
  --plugin laminar

-n 6 runs all six trials concurrently. --job-name gives the job a readable name instead of a timestamp, and --plugin laminar enables the plugin. At the start of the job, the plugin prints a Laminar evaluation: link. Trials appear there as they finish.

Our run took 16 minutes 33 seconds and ended like this:

terminal-bench • claude-code • claude-sonnet-5
Trials 6 | Exceptions 0 | Mean 0.667
Reward 1.0: 4 ; 0.0: 2
Results written to jobs/tb2-sonnet5/result.json

These costs and token counts come from the agent_result field in each trial's result.json:

Six trials cost $1.67 in model spend, between $0.08 and $0.48 each, plus sandbox time. The cheapest trial failed, as did the third most expensive. Neither low nor high spend reliably indicated success in this sample.

Use a sample like this to estimate your budget before running all 89 tasks. Task length and model choice affect per-trial cost. When you're ready for the full benchmark, drop the -i filters and set -n to fit your sandbox capacity and provider rate limits.

Reading Terminal-Bench results in Laminar#

Harbor writes results to jobs/<job-name>/: one directory per trial with the agent's logs, its trajectory, the verifier's output and a result.json. harbor view jobs starts a local web server for viewing them.

You can debug directly from those files. For agent evaluation across multiple runs, the plugin maps the same data into a Laminar evaluation:

The trace connects the score to the execution. Its root span holds the trial's rewards. Under it are four spans with Harbor's timings: environment_setup, agent_setup, agent and verifier.

When the agent writes an ATIF trajectory, as Claude Code does, the plugin replays it under the agent span. Each model turn becomes an LLM span with messages, tokens and cost. Each command becomes a tool span with its input and output. The verifier span's output is the reward.

Open a datapoint that scored 0 to read what the agent did on that task.

How to debug failed Terminal-Bench tasks#

fix-git was the cheapest failure in our run: $0.08, five model turns, about 20 seconds of agent time. Its instruction:

I just made some changes to my personal site and checked out master, but now I can't find those changes. Please help me find them and merge them into master.

The trace header puts the whole trial at 1 minute 15 seconds. The timeline shows that about 49 seconds went to agent_setup, installing Claude Code in the sandbox. The agent itself ran for a short stretch at the end.

The transcript shows each model turn and its tool calls in order:

1. The agent runs git status, git stash list, git reflog and git branch -vv --all in one command. The reflog shows the lost work: commit 650dba4, "Move to Stanford", made on a detached HEAD before a checkout back to master.
2. It runs git show --stat 650dba4 and sees that the commit changed _includes/about.md and _layouts/default.html.
3. It reads the full diff with git show 650dba4.
4. It runs git merge-tree against the merge base and finds a conflict in about.md: master has a newer commit, "off the job market woo", that edited the same paragraph.
5. It stops. The final message explains the conflict, recommends keeping the Stanford text, and asks: "How would you like to proceed? 1. Merge … 2. Something else … Let me know and I'll do it."

Nobody answers in a benchmark. Claude Code ended its turn without changing the repository, and the verifier failed both file checks:

FAILED test_about_file
FAILED test_layout_file
AssertionError: File /app/resources/patch_files/default.html is not in the correct state
2 failed in 0.08s

The agent found the orphaned commit, identified both files and detected the conflict. It never performed the merge. The trace distinguishes "couldn't find the commit" from "found it and waited for permission", even though both receive a reward of 0.

For this failure, the next experiment is an instruction change: tell the agent it runs unattended and must finish without asking for confirmation. That targets the point where this trial stopped.

The other failure, filter-js-from-html, needs a different investigation. The agent spent about four minutes and $0.39 writing /app/filter.py. It checked its work by writing one test page, running the filter on it, then running it again on the output. It finished with "Both checks pass: the filter is idempotent, and fully benign HTML (tables, headers, entities, inline styles, links with query strings) passes through byte-for-byte unchanged."

The verifier then ran for about 11 minutes. test_filter_blocks_xss failed because some attack vectors got through. test_clean_html_unchanged failed because the filter altered clean pages it should have left alone.

The agent's checks on one hand-written file missed both failure modes. Here, inspect the filter implementation and how the agent tested it. Completing the task without asking permission would not address either verifier failure.

For any failed trial, check:

• Did the agent stop early? Read the last LLM span. Look for a question to the user, a plan with no action, or a completion claim without a test run. Then check what it actually changed.
• Did the verifier run at all? Check for a verifier span and an exception on the root span before treating the result as a task failure.
• Did it run out of time? Compare the agent span's duration with the task's agent timeout in task.toml.
• What did the verifier see? The verifier span holds the rewards. For assertion messages like those above, read verifier/test-stdout.txt in the trial's directory.

The transcript view shows model turns and tool calls in order, with inputs and outputs inline. For the same debugging method applied to production agents, see how to detect AI agent failures.

How to compare agents and models on Terminal-Bench#

Compare runs on the same tasks to see where an agent or model gained or lost passes. The plugin's group_name option puts runs in one group so Laminar charts them together. evaluation_name names each run.

We ran the same six tasks twice more, once with Sonnet 5 and once with Opus 5, both in a group named tb2-models. The Opus command:

harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-opus-5 \
  --env daytona \
  -i fix-git -i overfull-hbox -i regex-log \
  -i sqlite-db-truncate -i chess-best-move -i filter-js-from-html \
  -n 3 -r 2 --retry-include AgentSetupTimeoutError \
  --job-name tb2-models-opus-5 \
  --plugin laminar \
  --pk evaluation_name=claude-code-opus-5 \
  --pk group_name=tb2-models

The Sonnet job used the same command with --model anthropic/claude-sonnet-5, --job-name tb2-models-sonnet-5 and --pk evaluation_name=claude-code-sonnet-5. For any other agent or model, change evaluation_name and keep group_name. Without group_name, the plugin groups runs by dataset name and version, so every Terminal-Bench 2.0 run sent to the same project lands in one terminal-bench@2.0 group, smoke tests and aborted jobs included.

The -n 3 -r 2 --retry-include AgentSetupTimeoutError line comes from our first attempt. With both jobs at -n 6, two trials in each job failed with AgentSetupTimeoutError: installing Claude Code in the sandbox took longer than 360 seconds. That is an infrastructure failure, not a model result, so we lowered concurrency and told Harbor to retry only that exception, up to twice. Without --retry-include, Harbor retries every exception type, including agent and verifier timeouts that would cost the same again.

Each job becomes one evaluation in the group. The group page charts the mean reward of every run and lists the runs below the chart:

The tb2-models group: one evaluation per Harbor job, six datapoints each. Sonnet 5 averaged 0.667 and Opus 5 averaged 0.833.

Opening an evaluation shows its datapoints: one row per trial, with the task and instruction in the Data column and the verifier's reward in the score column. Selecting a row opens that trial's trace beside the table:

The Sonnet 5 evaluation: six trials, four passes. The selected fix-git trial repeats the first run's failure: the agent found the lost commit, then asked whether to cherry-pick it or merge it.

Sonnet scored 0.667 again and failed the same two tasks. On fix-git it found commit 650dba4 within three model turns, recommended a cherry-pick, and ended with "Let me know and I'll proceed."

Opus 5 averaged 0.833, in 44 minutes 31 seconds against 25 minutes 21 seconds for Sonnet. From the Opus evaluation, pick the Sonnet run in the "Select compared evaluation" menu at the top of the page to see both side by side:

Sonnet 5 → Opus 5 on the same six tasks: mean reward 0.67 → 0.83. fix-git flips from 0 to 1. The red cross on filter-js-from-html marks the Opus trial's verifier timeout.

The comparison view shows each datapoint's score as an old → new delta. fix-git is the only flip. The Opus trace on the right shows why: it found the same dangling commit, created a safety branch at it, cherry-picked it onto master, resolved the conflict in favor of the Stanford text and reported "Done. Your changes are back on master." Both verifier checks passed.

Cost tells a different story. Opus spent $8.80 in model tokens across six trials, against $1.64 for Sonnet, and $7.06 of that went on one trial. On filter-js-from-html the Opus agent worked for almost 24 minutes and read 7.6 million input tokens, then the verifier ran past its 900-second limit and raised VerifierTimeoutError. Harbor counted the trial as an exception and scored it 0, so the most expensive trial in either run earned nothing. Run a few tasks per model before committing to all 89.

Keep the task list fixed. Different subsets mix model differences with task difficulty. A single task flipping between runs can also be noise, since agents don't take the same path on every attempt. Sonnet failing fix-git in both of our runs is stronger evidence than Opus passing it once.

Laminar's run comparison works for any two evaluations, and clicking a row opens that task's trace.

FAQ#

Does the Laminar plugin work with agents written in TypeScript?#

Yes. Harbor loads plugins in Python, so the plugin ships in the Python SDK, but it reads Harbor's trial results and the agent's trajectory file from outside the sandbox. Claude Code, which runs on Node.js, is the agent in this guide. The Laminar plugin installs nothing inside the sandbox. See the Harbor integration docs for the agents covered.

What happens to Harbor trials that are retried or cancelled?#

A retried trial overwrites its datapoint, so Laminar shows the final attempt. A trial that never produced a reward, for example because the agent crashed before the verifier ran, scores 0, matching how Harbor counts it in the mean. Cancelled trials get no score and are marked cancelled in the datapoint's metadata.

Can I send Harbor results to a self-hosted Laminar instance?#

Yes. Pass --pk base_url=http://your-host, plus --pk http_port=... and --pk grpc_port=... if you changed the defaults. We used these settings to run the jobs in this guide against a local Laminar instance. They are the same settings the SDK uses for self-hosted evaluations.

Will a Laminar outage stop my Harbor run?#

Not by default. The plugin logs Laminar errors and lets the Harbor job continue, so a connection failure doesn't stop the run. Set --pk fail_fast=true to fail the job when Laminar returns an error, for example in CI where missing evaluation results should fail the build.

How do I find which Terminal-Bench tasks every agent fails?#

Query the evaluation_datapoints table in the SQL editor. Each row has the trial's scores and its task name in data, so you can group by task across runs. For patterns inside traces, such as "the agent asked the user a question and stopped", define a Signal and Laminar counts it across trials. For choosing an evaluation setup more broadly, see how to evaluate AI agents.