> ## Documentation Index
> Fetch the complete documentation index at: https://laminar.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Manual evaluation API

`evaluate()` is the right entry point for 95% of cases. For the other 5% (wiring evaluations into an existing pipeline, streaming datapoints from a long-running job, scoring production traffic after the fact) use the lower-level `LaminarClient.evals` API, or call the HTTP endpoints directly if you're not on a Laminar SDK.

## When to use this

Reach for the manual API when you need to:

* Create an evaluation now and append datapoints to it over hours or days, as work completes.
* Register a datapoint in the UI *before* the executor runs, so a row is visible while the run is still in progress.
* Run the executor in one process and write scores from another (for example, an async judge that posts results back later).
* Score production traces without re-running the call: save the executor output and scores against a new datapoint.
* Write evaluations from a language or runtime with no Laminar SDK. The SDK methods below are thin wrappers over four HTTP endpoints you can call directly: see the [HTTP API reference](#http-api-reference).

If none of those apply, use [`evaluate()`](/docs/evaluations/quickstart).

## The three-phase pattern

The manual API is designed around three distinct moments in an evaluation's lifecycle. Call them in order:

1. **Create the evaluation** with `create_evaluation` / `create`. Returns an `eval_id`. Do this once per run.
2. **Pre-register each datapoint** with `create_datapoint`. Returns a `datapoint_id`. A row appears in the UI immediately, even though the executor hasn't run yet.
3. **Update the datapoint** with `update_datapoint`: link it to a trace, then (once the executor and evaluators finish) write the executor output and scores.

Pre-registering is the pattern that matters. It's how a long-running evaluation stays observable while it runs, and how a separate scoring process can write results back to rows created by the executor.

Each phase is one HTTP call underneath, so the same lifecycle is available without an SDK. See the [HTTP API reference](#http-api-reference) for full request and response shapes.

| Phase                   | SDK method                     | Endpoint                                                                    |
| ----------------------- | ------------------------------ | --------------------------------------------------------------------------- |
| Create the evaluation   | `create` / `create_evaluation` | [`POST /v1/evals`](#create-an-evaluation)                                   |
| Pre-register datapoints | `create_datapoint`             | [`POST /v1/evals/{eval_id}/datapoints`](#add-datapoints)                    |
| Write output and scores | `update_datapoint`             | [`POST /v1/evals/{eval_id}/datapoints/{datapoint_id}`](#update-a-datapoint) |
| Update the evaluation   | `update` / `update_evaluation` | [`POST /v1/evals/{eval_id}`](#update-an-evaluation)                         |

## Setup

<CodeGroup>
  ```typescript TypeScript theme={null}
  import { Laminar, LaminarClient, observe } from '@lmnr-ai/lmnr';
  import { OpenAI } from 'openai';

  Laminar.initialize({
    projectApiKey: process.env.LMNR_PROJECT_API_KEY,
    instrumentModules: { OpenAI },
  });

  const client = new LaminarClient({
    projectApiKey: process.env.LMNR_PROJECT_API_KEY,
  });

  const openai = new OpenAI();
  ```

  ```python Python theme={null}
  import os
  from lmnr import Laminar, LaminarClient, observe
  from openai import AsyncOpenAI

  Laminar.initialize(project_api_key=os.environ["LMNR_PROJECT_API_KEY"])

  client = LaminarClient(project_api_key=os.environ["LMNR_PROJECT_API_KEY"])
  openai_client = AsyncOpenAI()
  ```
</CodeGroup>

## Build executor and evaluator spans

Wrap the executor and each evaluator in `observe()` with the matching `spanType`. The evaluation UI uses `EXECUTOR` and `EVALUATOR` to know which spans hold the input, output, and score for each row.

<CodeGroup>
  ```typescript TypeScript theme={null}
  const runExecutor = async (testCase: { data: { country: string }; target: string }) =>
    observe(
      { name: 'executor', spanType: 'EXECUTOR', input: testCase.data },
      async () => {
        const response = await openai.chat.completions.create({
          model: 'gpt-5-mini',
          messages: [
            {
              role: 'user',
              content:
                `What is the capital of ${testCase.data.country}? ` +
                'Answer only with the capital, no other text.',
            },
          ],
        });
        return response.choices[0].message.content ?? '';
      },
    );

  const accuracy = async (output: string, target: string) =>
    observe(
      { name: 'accuracy', spanType: 'EVALUATOR', input: { output, target } },
      async () => (output.toLowerCase().includes(target.toLowerCase()) ? 1 : 0),
    );

  const lengthOk = async (output: string) =>
    observe(
      { name: 'length_ok', spanType: 'EVALUATOR', input: { output } },
      async () => (output.length > 0 && output.length < 50 ? 1 : 0),
    );
  ```

  ```python Python theme={null}
  @observe(name="executor", span_type="EXECUTOR")
  async def run_executor(test_case):
      response = await openai_client.chat.completions.create(
          model="gpt-5-mini",
          messages=[
              {
                  "role": "user",
                  "content": (
                      f"What is the capital of {test_case['data']['country']}? "
                      "Answer only with the capital, no other text."
                  ),
              }
          ],
      )
      return response.choices[0].message.content or ""


  @observe(name="accuracy", span_type="EVALUATOR")
  async def accuracy(output, target):
      return 1 if target.lower() in output.lower() else 0


  @observe(name="length_ok", span_type="EVALUATOR")
  async def length_ok(output, target=None):
      return 1 if 0 < len(output) < 50 else 0
  ```
</CodeGroup>

## Create the evaluation and datapoints

Open the evaluation up front, then loop over the test data. For each row, pre-register the datapoint, run the executor inside an `EVALUATION` span, and write scores back once the evaluators finish.

<CodeGroup>
  ```typescript TypeScript theme={null}
  const testData = [
    { data: { country: 'France' }, target: 'Paris' },
    { data: { country: 'Germany' }, target: 'Berlin' },
    { data: { country: 'Japan' }, target: 'Tokyo' },
  ];

  const evalId = await client.evals.createEvaluation(
    'Manual capitals eval',
    'capitals-manual',
    { model: 'gpt-5-mini', run_by: 'manual-api-demo' },
  );

  for (let i = 0; i < testData.length; i++) {
    const testCase = testData[i];

    await observe(
      { name: 'evaluation', spanType: 'EVALUATION', input: { testCase } },
      async () => {
        // Pre-register the datapoint with the current trace ID.
        // The row shows up in the UI now, before the executor runs.
        const datapointId = await client.evals.createDatapoint({
          evalId,
          data: testCase.data,
          target: testCase.target,
          index: i,
          traceId: Laminar.getTraceId(),
        });

        const output = await runExecutor(testCase);
        const accuracyScore = await accuracy(output, testCase.target);
        const lengthOkScore = await lengthOk(output);

        await client.evals.updateDatapoint({
          evalId,
          datapointId,
          scores: { accuracy: accuracyScore, length_ok: lengthOkScore },
          executorOutput: { response: output, model: 'gpt-5-mini' },
        });
      },
    );
  }

  await Laminar.flush();
  ```

  ```python Python theme={null}
  test_data = [
      {"data": {"country": "France"}, "target": "Paris"},
      {"data": {"country": "Germany"}, "target": "Berlin"},
      {"data": {"country": "Japan"}, "target": "Tokyo"},
  ]


  async def run_one(eval_id, index, test_case):
      # Phase 2: pre-register the datapoint. A row appears in the UI now.
      datapoint_id = client.evals.create_datapoint(
          eval_id=eval_id,
          data=test_case["data"],
          target=test_case["target"],
          index=index,
      )

      @observe(name="evaluation", span_type="EVALUATION")
      async def run_inside_span():
          # Link the datapoint to the evaluation trace that is now active.
          trace_id = Laminar.get_trace_id()
          client.evals.update_datapoint(
              eval_id=eval_id,
              datapoint_id=datapoint_id,
              scores={},
              trace_id=trace_id,
          )

          output = await run_executor(test_case)
          accuracy_score = await accuracy(output, test_case["target"])
          length_ok_score = await length_ok(output, test_case["target"])
          return output, {"accuracy": accuracy_score, "length_ok": length_ok_score}

      output, scores = await run_inside_span()

      # Phase 3: write the final executor output and scores.
      client.evals.update_datapoint(
          eval_id=eval_id,
          datapoint_id=datapoint_id,
          executor_output={"response": output, "model": "gpt-5-mini"},
          scores=scores,
      )


  async def main():
      eval_id = client.evals.create_evaluation(
          name="Manual capitals eval",
          group_name="capitals-manual",
          metadata={"model": "gpt-5-mini", "run_by": "manual-api-demo"},
      )

      for index, test_case in enumerate(test_data):
          await run_one(eval_id, index, test_case)

      Laminar.flush()
  ```
</CodeGroup>

<Note>
  The two SDKs link traces to datapoints in slightly different places. TypeScript accepts `traceId` on `createDatapoint` only, so call it from inside the `EVALUATION` span and pass `Laminar.getTraceId()` there. Python's `update_datapoint` accepts `trace_id`, so you can register the datapoint *before* the span opens and link the trace once it's running. The Python pattern is what you want when the row needs to be visible before you know which trace will own it.
</Note>

## Renaming or re-tagging a finished run

A long-running job often doesn't know everything about itself when the run is created: the final status, the git SHA it ends up testing, how many rows it processed. `update` (TypeScript) / `update_evaluation` (Python) writes the evaluation's `name` and `metadata` after the fact, any time after `create`:

<CodeGroup>
  ```typescript TypeScript theme={null}
  await client.evals.update({
    evalId,
    name: 'Manual capitals eval (complete)',
    metadata: { model: 'gpt-5-mini', run_by: 'manual-api-demo', status: 'complete' },
  });
  ```

  ```python Python theme={null}
  client.evals.update_evaluation(
      eval_id=eval_id,
      name="Manual capitals eval (complete)",
      metadata={"model": "gpt-5-mini", "run_by": "manual-api-demo", "status": "complete"},
  )
  ```
</CodeGroup>

Fields you leave out are unchanged, so you can rename a run without touching its metadata. When you do send `metadata`, it replaces the stored object wholesale rather than merging: include the keys you want to keep, as the examples above do. The group cannot be changed.

## Decoupling the executor from the scorer

Because `update_datapoint` can be called any time after `create_datapoint`, executor and scorer can live in different processes. A common shape:

1. A worker runs the agent, produces a trace, and calls `update_datapoint` with `executor_output` and an empty `scores={}` dict.
2. A judge process reads `executor_output` from the dataset or from the agent's output store, scores it, and calls `update_datapoint` again with the filled-in `scores`.

Both writes target the same `datapoint_id`. The UI updates in place each time.

## Backfilling without running the executor

For pure backfills (rows you already have outputs and scores for, no live executor), loop `create_datapoint` + `update_datapoint` over the pre-scored rows:

```python theme={null}
eval_id = client.evals.create_evaluation(
    name="Backfilled capitals",
    group_name="capitals",
)

for i, row in enumerate(rows):
    datapoint_id = client.evals.create_datapoint(
        eval_id=eval_id,
        data=row["data"],
        target=row["target"],
        index=i,
    )
    client.evals.update_datapoint(
        eval_id=eval_id,
        datapoint_id=datapoint_id,
        executor_output=row["output"],
        scores=row["scores"],
    )
```

Over HTTP a backfill skips the update phase entirely: [`POST /v1/evals/{eval_id}/datapoints`](#add-datapoints) accepts `executorOutput` and `scores` on each point, so fully-scored rows land in one batched call.

No `EVALUATION` span is opened, so no trace is attached. The row shows `data`, `target`, `executor_output`, and `scores`, which is enough for the list view, progression chart, and side-by-side comparison. Use this when you want the numbers in Laminar but don't need per-row transcript drill-down.

## HTTP API reference

All four endpoints authenticate with a project API key as a Bearer token and take a JSON body. Base URL is `https://api.lmnr.ai` on Laminar Cloud, or your app-server's HTTP origin (including port) when [self-hosting](/docs/evaluations/self-hosted).

**Headers**

| Header          | Value                      |
| --------------- | -------------------------- |
| `Authorization` | `Bearer <PROJECT_API_KEY>` |
| `Content-Type`  | `application/json`         |

All request and response field names are camelCase. A bad or revoked key returns `401`.

The examples below assume:

```bash theme={null}
export LMNR_PROJECT_API_KEY="<your project API key>"
export LMNR_BASE_URL="https://api.lmnr.ai"

AUTH="Authorization: Bearer ${LMNR_PROJECT_API_KEY}"
JSON="Content-Type: application/json"
```

Spans are the one thing these endpoints don't carry: emit them with any [OpenTelemetry](/docs/tracing/otel) SDK and pass that trace's id as `traceId` on the datapoint to get per-row transcripts.

### Create an evaluation

`POST /v1/evals`

| Field       | Type   | Required | Description                                                                              |
| ----------- | ------ | -------- | ---------------------------------------------------------------------------------------- |
| `name`      | string | no       | Name shown in the UI. A readable name is generated when omitted.                         |
| `groupName` | string | no       | Group identifier. Only runs sharing a group name can be compared. Defaults to `default`. |
| `metadata`  | object | no       | Arbitrary JSON on the evaluation row, filterable in the UI.                              |

```bash theme={null}
curl -sS -X POST "${LMNR_BASE_URL}/v1/evals" \
  -H "${AUTH}" -H "${JSON}" \
  --data-raw '{
    "name": "HTTP capitals eval",
    "groupName": "capitals-http",
    "metadata": { "model": "gpt-5-mini", "runBy": "http-api-demo" }
  }'
```

Returns `200` with the created evaluation:

```json theme={null}
{
  "id": "c381a582-6ae0-4b62-81a4-55d450d9d65d",
  "createdAt": "2026-07-27T12:42:33.209382Z",
  "name": "HTTP capitals eval",
  "projectId": "0cce3ee3-d6bb-437d-a2fa-bbfd72a935e2",
  "groupId": "capitals-http",
  "metadata": { "model": "gpt-5-mini", "runBy": "http-api-demo" }
}
```

Keep the `id`: every later call is scoped to it.

### Update an evaluation

`POST /v1/evals/{eval_id}`

| Field      | Type   | Required | Description                                                               |
| ---------- | ------ | -------- | ------------------------------------------------------------------------- |
| `name`     | string | no       | New name. Omit to leave unchanged.                                        |
| `metadata` | object | no       | Replaces the existing metadata object wholesale. Omit to leave unchanged. |

```bash theme={null}
curl -sS -X POST "${LMNR_BASE_URL}/v1/evals/${EVAL_ID}" \
  -H "${AUTH}" -H "${JSON}" \
  --data-raw '{
    "name": "capitals @ a1b2c3d",
    "metadata": { "model": "gpt-5-mini", "commit": "a1b2c3d", "status": "complete" }
  }'
```

Returns `200` with the updated evaluation in the same shape as create, or `404` when the id doesn't exist in the project.

Fields you leave out are unchanged, so you can rename a run without touching its metadata. `metadata` is replaced wholesale rather than merged: send the whole object, including keys you want to keep. `groupId` cannot be changed.

### Add datapoints

`POST /v1/evals/{eval_id}/datapoints`

| Field                     | Type     | Required | Description                                                                                                                                                                       |
| ------------------------- | -------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `points`                  | array    | yes      | One or more datapoints. Batch them rather than sending one request per row.                                                                                                       |
| `points[].data`           | any JSON | yes      | The executor input, shown in the datapoint table.                                                                                                                                 |
| `points[].id`             | UUID     | no       | Datapoint id. Mint it yourself so you can update the row later; otherwise the server generates one and does not return it.                                                        |
| `points[].index`          | integer  | no       | Position of the row within the run. Also the key rows are matched on when comparing two runs. Defaults to `0` and is **not** auto-incremented, so number your points yourself.    |
| `points[].target`         | any JSON | no       | Expected output, for evaluators and side-by-side diffing.                                                                                                                         |
| `points[].executorOutput` | any JSON | no       | Skip the update phase and write the output now.                                                                                                                                   |
| `points[].scores`         | object   | no       | Score name → number. Same shortcut as `executorOutput`.                                                                                                                           |
| `points[].metadata`       | object   | no       | Arbitrary JSON on the datapoint.                                                                                                                                                  |
| `points[].traceId`        | UUID     | no       | Trace that owns this row. Clicking the row opens that trace's transcript.                                                                                                         |
| `groupName`               | string   | no       | Group written onto these datapoints. Pass the evaluation's group name: it defaults to `default` per request, and a mismatch leaves the rows out of the group's progression chart. |

```bash theme={null}
curl -sS -X POST "${LMNR_BASE_URL}/v1/evals/${EVAL_ID}/datapoints" \
  -H "${AUTH}" -H "${JSON}" \
  --data-raw '{
    "groupName": "capitals-http",
    "points": [
      {
        "id": "9d7efa44-77d2-4dde-a1e3-9e8e1b7bfaa6",
        "index": 0,
        "data": { "country": "France" },
        "target": "Paris"
      },
      {
        "id": "d5936651-31a2-4c24-bb5f-44eac6bf5e26",
        "index": 1,
        "data": { "country": "Germany" },
        "target": "Berlin"
      }
    ]
  }'
```

Returns `200` with the evaluation id.

<Warning>
  Two fields the SDKs fill in for you and the API does not:

  * **`groupName` is not inherited from the evaluation.** It defaults to `default` on every request, so if you created the evaluation with a `groupName`, send the same value on each datapoints call. Otherwise the rows land in a different group and won't appear in the group's progression chart.
  * **`index` is not auto-incremented.** Every point you send without one gets `0`. Since [comparing runs](/docs/evaluations/comparing-runs) matches rows across runs on `index`, leaving them all at `0` makes each row match every row in the other run instead of its counterpart. Number your points as you build the batch.
</Warning>

### Update a datapoint

`POST /v1/evals/{eval_id}/datapoints/{datapoint_id}`

| Field            | Type     | Required | Description                                                          |
| ---------------- | -------- | -------- | -------------------------------------------------------------------- |
| `scores`         | object   | yes      | Score name → number. Send `{}` when you only have the output so far. |
| `executorOutput` | any JSON | no       | The executor's output. Omit to keep the stored value.                |
| `traceId`        | UUID     | no       | Links (or re-links) the row to a trace.                              |

```bash theme={null}
# Executor: record the output before any scores exist.
curl -sS -X POST \
  "${LMNR_BASE_URL}/v1/evals/${EVAL_ID}/datapoints/${DATAPOINT_ID}" \
  -H "${AUTH}" -H "${JSON}" \
  --data-raw '{ "executorOutput": { "response": "Paris" }, "scores": {} }'

# Judge, later: fill in the scores. The stored output is preserved.
curl -sS -X POST \
  "${LMNR_BASE_URL}/v1/evals/${EVAL_ID}/datapoints/${DATAPOINT_ID}" \
  -H "${AUTH}" -H "${JSON}" \
  --data-raw '{ "scores": { "accuracy": 1, "length_ok": 1 } }'
```

Returns `200` with the datapoint id. Scores are merged into the existing scores, so a second call adding a new score name keeps the earlier ones. Call it as many times as you like against the same `datapoint_id`; the UI updates in place.

## Result

Manual evaluations show up in the same evaluations list, progression chart, and comparison UI as `evaluate()` runs. Groups, per-datapoint deltas, and CSV export all work the same way.

<Frame caption="Manual evaluation detail page. Progression chart and datapoint table match what evaluate() produces">
  <img src="https://mintcdn.com/laminarai/-q9WJgn2x9iWK3Su/images/evaluations/manual-evaluation-trace.png?fit=max&auto=format&n=-q9WJgn2x9iWK3Su&q=85&s=40cb7248a7f922162d33e4e750715afd" alt="Evaluation detail page for Manual capitals eval with length_ok averaging 1.00 across six datapoints" width="1280" height="577" data-path="images/evaluations/manual-evaluation-trace.png" />
</Frame>

Clicking a row opens the transcript for that datapoint's trace, with the full `EVALUATION` root, `EXECUTOR`, and `EVALUATOR` nesting you'd expect from `evaluate()`.

<Frame caption="One datapoint's transcript: EVALUATION root, executor, the gpt-5-mini call, and accuracy / length_ok scores">
  <img src="https://mintcdn.com/laminarai/-q9WJgn2x9iWK3Su/images/evaluations/manual-evaluation.png?fit=max&auto=format&n=-q9WJgn2x9iWK3Su&q=85&s=e1d66c500215f41a1dadbf656a736c4e" alt="Manual evaluation trace with EVALUATION, EXECUTOR, gpt-5-mini, accuracy, and length_ok spans" width="1280" height="577" data-path="images/evaluations/manual-evaluation.png" />
</Frame>

## Next steps

<CardGroup cols={2}>
  <Card title="Quickstart" href="/docs/evaluations/quickstart" icon="play">
    The high-level `evaluate()` API, which is the right starting point for most cases.
  </Card>

  <Card title="Compare runs" href="/docs/evaluations/comparing-runs" icon="chart-line">
    Group manual runs so you can compare them like any other evaluation.
  </Card>

  <Card title="Concepts" href="/docs/evaluations/concepts" icon="boxes">
    The datapoint / executor / evaluator / group model the manual API maps onto.
  </Card>

  <Card title="SDK reference" href="/docs/sdk/client" icon="code">
    Full parameters for `LaminarClient.evals` methods.
  </Card>
</CardGroup>