Distributed Tracing for AI Agents Across Services and Queues

Distributed tracing for AI agents follows one run across the processes that execute it. Your API can finish successfully while its research worker appears in a separate trace, with no parent or session connecting the work. For a run where each step has one upstream caller, the target is one trace with one root span.

Laminar's Python and TypeScript SDKs do not inject or extract HTTP or queue context themselves. Unless another integration handles propagation, you must serialize the parent context, carry it across the boundary and restore it downstream. This guide covers tested HTTP, queue and approval-resume patterns, plus Temporal's documented integration. It also explains when you need W3C traceparent alongside Laminar's context to include framework-generated spans.

Illustrative: the application spans share one trace when each boundary carries context. Framework-generated HTTP spans may also need W3C propagation, as the HTTP example explains.

Why agent runs split into several traces#

Within a process, Laminar carries the current span across function calls, async code and threads. A separate worker cannot read that process-local context. The parent must travel through the HTTP request, queued message or stored state that starts its work.

OpenTelemetry instrumentation libraries usually handle propagation for ordinary HTTP requests. Queues and stored resumes need their own context handoff unless an integration provides it. Laminar's Temporal integration is one such exception.

The measurements below come from exported spans in ClickHouse on a local Laminar stack built from main on 2026-10-22, using Python lmnr 0.8.0 and TypeScript @lmnr-ai/lmnr 0.8.51. These results describe that SDK/backend combination, not every OpenTelemetry service or future Laminar version.

The pattern: serialize, carry, restore#

Laminar.serialize_span_context() returns a string containing the current span's trace id, span id, ancestor paths, and associated session id, user id and metadata. Downstream, deserialize it and pass the result as parent_span_context when starting the application span. Spans created inside that context, including auto-instrumented LLM calls, nest beneath it.

The continuing traces docs recommend treating incoming context as untrusted input and failing open: missing or malformed context should not fail the work. Within one service, pass span objects rather than serializing them.

Every Python receiver below uses this helper, saved as context_utils.py:

from lmnr import Laminar


def restore_parent(raw: str | None):
    if not raw:
        return None
    try:
        return Laminar.deserialize_span_context(raw)
    except Exception:
        return None

Install the Python packages with pip install lmnr fastapi uvicorn httpx celery. Set LMNR_PROJECT_API_KEY in every process, and keep context_utils.py importable by each Python worker.

HTTP: an API service calling a research worker#

The caller, api.py, starts answer_question and sends its context to a research worker in a separate process:

import httpx
from lmnr import Laminar, observe

Laminar.initialize()


@observe(name="answer_question", session_id="sess-42", user_id="user-7")
def answer_question(question: str):
    headers = {}
    span_context = Laminar.serialize_span_context()
    if span_context:
        headers["X-Laminar-Span-Context"] = span_context
    resp = httpx.post(
        "http://research-worker:9100/research",
        json={"query": question},
        headers=headers,
    )
    return resp.json()


if __name__ == "__main__":
    print(answer_question("context propagation"))
    Laminar.flush()

The worker, research_worker.py, restores that context before starting its application span:

from fastapi import FastAPI, Request
from lmnr import Laminar, observe
from context_utils import restore_parent

Laminar.initialize()
app = FastAPI()


@observe()
def search_sources(query: str):
    return [f"source for {query}"]


@app.post("/research")
async def research(request: Request):
    body = await request.json()
    parent = restore_parent(request.headers.get("x-laminar-span-context"))
    with Laminar.start_as_current_span(
        name="research", input=body, parent_span_context=parent
    ):
        sources = search_sources(body["query"])
        Laminar.set_span_output(sources)
    return {"sources": sources}

For a same-machine test, make research-worker resolve to 127.0.0.1. Start uvicorn research_worker:app --port 9100, then run python api.py.

The three application spans share one trace: answer_question is the root, research is its child, and search_sources sits beneath research. All three inherit sess-42 and user-7, which keeps the worker's application spans visible when filtering by session. Missing or malformed headers returned HTTP 200 in our tests, with research starting a new root.

The file above carries only Laminar context. FastAPI 0.161.1 also emitted OpenTelemetry framework spans in our test. With only the Laminar header, its POST /research span landed in a separate trace.

To include that framework span in the run, also add from opentelemetry.propagate import inject to the caller and call inject(headers) after setting the Laminar header. This tested dual-header variant produced one trace with one root. The FAQ explains its hierarchy and session-id limit.

A TypeScript worker behind a Python API#

The Laminar header works across languages. Install @lmnr-ai/lmnr with npm and run this worker.ts with npx tsx worker.ts, instead of the Python worker:

import http from "node:http";
import { Laminar, observe } from "@lmnr-ai/lmnr";

Laminar.initialize();

const rankSources = (query: string) =>
  observe({ name: "rank_sources" }, async () => [`ranked results for ${query}`]);

http.createServer(async (req, res) => {
  let raw = "";
  for await (const chunk of req) raw += chunk;
  const body = JSON.parse(raw);

  const header = req.headers["x-laminar-span-context"];
  const span = Laminar.startSpan({
    name: "research",
    input: body,
    parentSpanContext: typeof header === "string" ? header : undefined,
  });
  const sources = await Laminar.withSpan(span, () => rankSources(body.query), true);

  res.writeHead(200, { "content-type": "application/json" });
  res.end(JSON.stringify({ sources }));
}).listen(9100);

The Python answer_question span remained the root, with TypeScript's research and rank_sources nested beneath it and session/user ids inherited. The third argument to withSpan ends the span when the callback settles, including on rejection.

The TypeScript SDK handles malformed parent context itself. Our malformed-header requests succeeded and started new roots. A TypeScript caller serializes with Laminar.serializeLaminarSpanContext(); the context utilities reference covers both SDKs.

Laminar span context vs traceparent#

W3C Trace Context defines traceparent as version-traceid-parentid-flags. OpenTelemetry's default propagator injects and extracts it.

We tested four carrier/downstream combinations. Three used Laminar's SDK downstream; one used a plain OpenTelemetry SDK exporting to Laminar. The comparison tracks the application spans, excluding framework-generated telemetry. As the HTTP test showed, the Laminar header alone did not connect FastAPI's server span.

Measured application-span results on the local 2026-10-22 stack, with lmnr 0.8.0 and @lmnr-ai/lmnr 0.8.51. Framework spans are outside this diagram's scope.

A plain OpenTelemetry service that extracts traceparent creates spans with the correct parent, and Laminar preserves that nesting. Laminar's SDK also builds ancestor paths from its own context. With a bare traceparent, it started a fresh path in our test, and the backend stored research as a second root.

Neither traceparent case carried session id, user id or trace metadata downstream. The header has no fields for those values, and we propagated nothing else.

Use W3C propagation for plain OpenTelemetry services. Session/user filters will miss their spans unless you carry those values separately. Between Laminar SDKs, carry Laminar context; add traceparent when other instrumentation also needs the standard context. Laminar's Temporal integration writes both and prefers Laminar context when restoring.

Queues and background jobs#

For a queue, carry the serialized context as a task argument and restore it inside the task. In tasks.py, triage_inbox fans out one job per document:

from celery import Celery
from lmnr import Laminar, observe
from context_utils import restore_parent

Laminar.initialize()
app = Celery("jobs", broker="redis://localhost:6379/0")


@observe()
def summarize_document(doc_id: str):
    return f"summary of {doc_id}"


@app.task
def summarize(doc_id: str, trace_context: str | None = None):
    with Laminar.start_as_current_span(
        name="summarize_job",
        input={"doc_id": doc_id},
        parent_span_context=restore_parent(trace_context),
    ):
        Laminar.set_span_output(summarize_document(doc_id))


@observe(name="triage_inbox", session_id="sess-queue-1")
def triage_inbox(doc_ids: list[str]):
    for doc_id in doc_ids:
        summarize.delay(doc_id, trace_context=Laminar.serialize_span_context())

The displayed configuration requires Redis running at localhost:6379 and its Celery transport dependencies, available through pip install "celery[redis]". Start celery -A tasks worker, then call triage_inbox(["doc-1", "doc-2"]) from your application.

We tested the task code with Celery 5.6.3's filesystem broker, not the displayed Redis configuration. With the default prefork pool and two worker processes, the trace contained one triage_inbox root, two summarize_job spans and their two summarize_document children. All carried sess-queue-1.

The jobs started after the parent span ended and still nested correctly. A parent need not remain open for a child to attach to it.

With default batching and the worker kept running, spans exported within 20 seconds without flushing inside tasks. Flush before a short-lived process exits, including a producer that enqueues work and immediately stops. Avoid flushing every task in a busy worker; see flushing and shutdown.

Fan-in and span links#

A batch containing messages from independent runs cannot give its consumer span multiple parents. OpenTelemetry's messaging conventions use span links to represent those relationships.

The Laminar backend we tested does not persist or display span links. If you need native links between a consumer and several independent producers, make link support a backend-selection criterion.

With Laminar, you can correlate manually: start the batch as a root and record upstream trace ids in metadata. This function assumes an initialized worker with Laminar and restore_parent imported, and your application's embed function available:

def embed_batch(batch: list[dict]):
    contexts = [restore_parent(m.get("trace_context")) for m in batch]
    source_ids = sorted({str(c.trace_id) for c in contexts if c})
    with Laminar.start_as_current_span(
        name="embed_batch", metadata={"source_trace_ids": ",".join(source_ids)}
    ):
        embed([m["text"] for m in batch])

Our test skipped malformed and missing contexts and recorded both valid upstream trace ids. This SQL editor query found the batch: SELECT trace_id FROM spans WHERE name = 'embed_batch' AND position(attributes, '<upstream trace id>') > 0.

That is searchable metadata, not stored span links or parent-child edges.

Temporal workflows#

Laminar's Temporal integration handles propagation between the client, workflow and activities. We did not test Temporal end to end here; this section follows Laminar's integration guide and source.

For Python lmnr 0.7.53 or later, Laminar.initialize() instruments Temporal when temporalio is installed. Initialize in both the starting process and worker before Client.connect. For TypeScript @lmnr-ai/lmnr 0.8.28 or later, register Laminar's client, activity and workflow interceptors. The Temporal integration guide supplies complete examples.

Do not create spans in workflow code. Temporal replays workflows, and an open span cannot survive replay; put instrumented work in activities. A scheduled workflow has no caller span, so each scheduled run starts its own root.

Human approval: resume from stored state#

If the starting process exits while waiting for approval, store its context beside the state you will resume. This approvals.py implements both steps:

import sqlite3
import sys
from lmnr import Laminar, observe
from context_utils import restore_parent

Laminar.initialize()
db = sqlite3.connect("approvals.db")
db.execute(
    "CREATE TABLE IF NOT EXISTS approvals "
    "(id TEXT PRIMARY KEY, draft TEXT, trace_context TEXT)"
)


@observe(name="draft_weekly_report", session_id="sess-approval-1")
def draft_weekly_report(approval_id: str):
    draft = "Revenue up 4% week over week."
    db.execute(
        "INSERT OR REPLACE INTO approvals VALUES (?, ?, ?)",
        (approval_id, draft, Laminar.serialize_span_context()),
    )
    db.commit()


@observe()
def send_report(draft: str):
    return f"sent: {draft}"


def resume_after_approval(approval_id: str):
    draft, trace_context = db.execute(
        "SELECT draft, trace_context FROM approvals WHERE id = ?", (approval_id,)
    ).fetchone()
    with Laminar.start_as_current_span(
        name="resume_after_approval",
        input={"approval_id": approval_id},
        parent_span_context=restore_parent(trace_context),
    ):
        send_report(draft)


if __name__ == "__main__":
    step, approval_id = sys.argv[1], sys.argv[2]
    if step == "request":
        draft_weekly_report(approval_id)
    else:
        resume_after_approval(approval_id)
    Laminar.flush()

Run python approvals.py request appr-7, then python approvals.py resume appr-7 from the same directory so both processes use the same database.

In our separate-process test, resume_after_approval started 20.5 seconds after the original root. It and send_report nested beneath draft_weekly_report, with the session id on every span. A corrupted stored context allowed the resume to complete as a new root.

The serialized context contains no timestamp or expiry field. That does not establish behavior for arbitrary pause lengths: we tested about 20 seconds, not hours or days. The earlier trace must also remain within your project's data retention. These measurements make no claim about trace-duration calculations or timeline rendering.

How to verify one run has one root#

Start with a known run. Check that its entry span is the only top-level span and inspect the worker hierarchy in the span tree and transcript view. Compare expected work with exported spans: two jobs without retries should produce two summarize_job spans. If retries occurred, account for the attempts.

Do not use session filtering as a completeness check. An orphaned worker trace may lose its session id and disappear from that view.

In the SQL editor, a top-level span has a zero parent_span_id. This query finds traces with multiple roots:

SELECT trace_id,
       countIf(parent_span_id = '00000000-0000-0000-0000-000000000000') AS root_spans,
       count() AS spans
FROM spans
WHERE start_time > now() - INTERVAL 1 DAY
GROUP BY trace_id
HAVING root_spans > 1

It caught our traceparent-only Laminar SDK case. It cannot catch two separate, single-root traces. For that signature, search for worker entry spans that became roots:

SELECT trace_id, start_time
FROM spans
WHERE name = 'research'
  AND parent_span_id = '00000000-0000-0000-0000-000000000000'
  AND start_time > now() - INTERVAL 1 DAY

This returned our no-header and traceparent-only runs, but not the Laminar-context run.

Both queries have limits. The window must include the producer and slowest consumer. A worker root indicates a bug only if that entry point always belongs to an upstream run. Clean results do not prove that every expected span exported. For instrumentation fundamentals, see OpenTelemetry for AI agents.

FAQ#

Does Laminar propagate trace context over HTTP automatically?#

No. Neither Laminar SDK instruments HTTP clients, web frameworks or task queues, so neither adds a header to outgoing requests or reads one from incoming requests. Propagation is automatic within a process and across Temporal. Other libraries in your stack may still create spans: FastAPI 0.161's built-in telemetry produced server spans in our test. At every other boundary, add X-Laminar-Span-Context yourself, as in the continuing traces docs.

Can I send traceparent and the Laminar header together?#

Yes. In the tested dual-header HTTP variant, FastAPI's POST /research and Laminar's research were siblings beneath answer_question. The framework span used W3C context; the application span restored Laminar context. Everything shared one root, but the framework server span still lacked a session id. One trace does not guarantee identical attributes on every span.

What should I log when context restoration fails?#

Record the extraction failure without copying the raw context into logs or responses. The Python helper deliberately returns None without logging; the TypeScript SDK logged a warning for malformed context. Missing, garbage and structurally invalid JSON headers all allowed our worker requests to succeed. See the fail-open guidance in continuing traces.

Does trace metadata travel in the header?#

Yes. Session id, user id and trace metadata are included in serialized Laminar context. Keep metadata small because it crosses each boundary with the request or message, and avoid forwarding sensitive values to less-trusted services. Treat incoming user/session ids as correlation data, not proof of identity or permission.

Can an approval resume intentionally start a new trace?#

Yes. Start a new root rather than restoring the stored parent, then assign the same session id to group the runs. The documented Laminar.set_trace_session_id() must run inside an active span context. A session groups traces; it does not create a parent-child relationship between them.

How do I trace a subagent running as a subprocess?#

The Python SDK reads LMNR_SPAN_CONTEXT during Laminar.initialize(). Set that environment variable to the parent's serialized context before launching the subprocess. With lmnr 0.8.0, our subagent nested correctly but did not inherit the session id. To set it explicitly, call Laminar.set_trace_session_id() inside the subagent's first span. In-process subagents do not need this process-boundary handoff; see tracing the OpenAI Agents SDK.