Skip to main content
A dataset is the set of tasks behind an agent evaluation, stored and versioned in Laminar. Your repo keeps one small file per dataset, tasks/dataset.toml, which pins one version of each task. The task folders themselves, with their case data and reference answers, live in Laminar. lmnr-cli eval dataset push uploads your tasks and pins them, and lmnr-cli eval dataset pull downloads the versions that dataset.toml pins. Cases built from production traces often hold customer data and large fixtures, which you may not want in your git history. With the tasks in Laminar, every trial also records which version of its task it ran, so you can open that exact version from the evaluation.
Datasets need lmnr-cli 0.5.8 or later. Datasets created before it are now legacy datasets.

What lives in git and what lives in Laminar

A dataset is a folder in evals/ with a harbor.yaml, as described in The evals folder:
The first push adds these lines to evals/.gitignore, so git ignores every task folder and keeps dataset.toml:
evals/.gitignore

Name the dataset and its tasks

dataset.toml is Harbor’s dataset manifest. Its [dataset] name is the dataset’s name in Laminar, and the default evaluation group for its runs:
evals/support/tasks/dataset.toml
Each task is known by the [task] name in its task.toml, not by its folder name:
evals/support/tasks/duplicate-charge/task.toml
Both names have Harbor’s org/name form. The org part can be anything, such as your app or team name. Every task in a dataset needs its own name, so when you copy a task folder to make a new case, change the name in the copy’s task.toml.

Push tasks to Laminar

From the repo root:
push hashes each task, uploads the versions Laminar doesn’t have yet, and pins them in dataset.toml:
evals/support/tasks/dataset.toml
Commit dataset.toml. That commit is what gives your teammates and CI the new version. If the task folders were committed to git before, the new .gitignore rules don’t untrack them. Remove them from the index once, keeping the files on disk:
A task’s version is the hash of its content, computed the same way as Harbor’s harbor sync. These files count: task.toml, instruction.md, README.md, trajectory.json, and the environment/, tests/, solution/, and steps/ folders, minus anything the task’s .gitignore or .harborignore excludes. Other files in the task folder are neither hashed nor uploaded. push refuses a task that contains .env* files, and a task can be at most 1 GiB once packed.

Pull tasks from Laminar

pull makes each task folder match its pin in dataset.toml. Run it after you clone the repo, and after a git pull that changed dataset.toml. It never changes dataset.toml itself.

How push and pull treat each task

Laminar stores every version of a task side by side and has no “latest” version. dataset.toml is the only record of which version a dataset uses. For each task, push and pull compare three things: the task folder on disk, its pin in dataset.toml, and whether Laminar has the version on disk. These rules cover the everyday changes:
  • Edit a task: change its files, push, and commit dataset.toml.
  • Add a task: copy a task folder, give the copy a new [task] name, push, and commit dataset.toml.
  • Remove a task: delete its [[tasks]] entry from dataset.toml and commit. The next pull deletes the folder. push never removes a pin.
Laminar keeps every version it receives, so earlier evaluations still link to the task versions they ran.

When two people edit the same task

Each push stores its own version, so both edits are kept in Laminar. The two branches then change the same digest line in dataset.toml, and git shows a merge conflict there. Keep the digest you want, commit the merge, and run pull to get that version on disk.

Local edits and --force

pull checks every task before it writes anything. If it would overwrite a task whose version Laminar doesn’t have, it stops with exit code 22 and names the task, because that edit exists only on your disk. Push the edit to keep it, or pass --force to discard it and take the pinned version:

Preview, filter, and script

Run a dataset

lmnr-cli eval run runs the tasks on disk as they are, including edits you haven’t pushed. It never pulls, so run pull first on a fresh clone or after dataset.toml changes:
If a pinned task is missing on disk or differs from its pin, eval run warns and prints the push and pull commands. If a dataset has pins and no task folders at all, as in a fresh clone, it stops and tells you to pull. With --remote, only the tasks Laminar doesn’t have are uploaded with the run. The sandbox downloads the others from the dataset. In CI, pull before you run. Both commands use LMNR_PROJECT_API_KEY:

Browse datasets in Laminar

The project’s Datasets page lists your datasets. Open one to see its tasks and every version of each task, and open a version to browse its files.
The refund-agent/support dataset in Laminar with the refund-agent/duplicate-charge task open. Two versions, v2 and v1, are listed; v1 is selected and its task.toml is shown next to its environment and tests folders.

A task in a dataset: every version Laminar stores, and the files of the selected version.

An evaluation of a dataset links to the dataset from its header. Each trial records the name and version of the task it ran, so the trial’s Files view shows the task at that exact version under task/, with a link to it in the dataset. For --remote runs, the Files view also has the dataset’s agent/ folder and the trial’s Harbor output under logs/.
An evaluation of the refund-agent/support dataset with a trial's Files tab open. The tree shows task/ with the task's files, a collapsed agent/ folder, and logs/ with agent/result.json, verifier/reward.json, and verifier/test-stdout.txt, which is open and shows the refund check and the judge's verdict.

A trial's Files view: the task version it ran, linked to the dataset, next to Harbor's output for the trial. Here the verifier's output shows the LLM judge's reasoning.

On a self-hosted deployment, datasets need an S3 bucket. Set S3_DATASETS_BUCKET, with AWS credentials, for both the app server and the frontend. Without it, push and pull fail with storage_not_configured.

Next steps

The evals folder

Reference for harbor.yaml, dataset.toml, task.toml, and secrets.

Run evals with lmnr-cli

Local and remote modes, flags, CI, and sandbox limits.

Compare runs

Every run of a dataset is charted in its group. Diff two runs to see which cases regressed.

Track failures with Signals

Find the production traces worth turning into new tasks.