OpenTelemetry for AI agents: Attributes, Trace Structure, and Export

OpenTelemetry lets AI agents share a tracing pipeline with the services they call. An agent run, its model requests, and its tool executions can appear in the same trace as an incoming HTTP request or a database query. To make that trace useful for debugging agent behavior, the spans also need model information, token usage, messages, and meaningful parent-child relationships.

This guide covers those attributes, trace structure, OTLP export, and sending traces to multiple backends. The examples use the OpenTelemetry SDK directly, with Laminar as the receiving backend. Laminar-specific attributes are identified separately from OpenTelemetry conventions.

Why use OpenTelemetry for agent tracing?#

OpenTelemetry gives agent telemetry a common path through your infrastructure:

• Connected traces. With trace context propagated across calls, model and tool spans stay connected to the request or job that started the agent.
• Reusable instrumentation. OTLP exporters let you send the same spans to different compatible backends without rewriting the agent's tracing code.
• Shared attribute names. GenAI semantic conventions describe model operations and usage, reducing the amount of custom interpretation each backend needs.

Transport compatibility and feature support are separate questions. A backend may accept a span without using its message attributes to render a conversation or its token counts to calculate cost. Check both the endpoint and the attributes the destination understands.

Which attributes should an LLM span carry?#

A model-call span should describe the operation, provider, requested model, and result. Record usage when the provider reports it, and capture messages when you need to inspect the conversation.

The GenAI conventions are evolving. Older instrumentation may emit gen_ai.system for the provider; current conventions use gen_ai.provider.name. Keep the convention version and backend support aligned, especially for cache and reasoning usage fields.

For the Python examples below, serialize message arrays as JSON strings. A text message looks like this:

messages = [{
    "role": "user",
    "parts": [{"type": "text", "content": "Find a hotel in Paris"}],
}]
span.set_attribute("gen_ai.input.messages", json.dumps(messages))

Record tool calls and tool results using their corresponding message parts, including the call ID that connects them. Messages are optional content capture; model and usage attributes remain useful when message bodies are omitted.

Input token totals include cached input tokens. Normalize provider usage accordingly rather than adding cache reads to an already inclusive total. For cost calculation, include the model name as well as usage: the backend needs a matching price to estimate the charge. Missing pricing information should not be treated as a zero-cost call.

Structuring an agent trace#

Use a span for the agent run, with child spans for each model call and tool invocation. If the agent starts inside an existing request or job, keep its run span under that parent.

Each subagent's work should sit beneath a span representing its invocation. When a tool invokes the subagent, that tool span can serve as the parent for its model and tool calls. An additional wrapper is only useful if it represents a distinct operation.

request
└── agent.run
    ├── chat
    ├── research_destination [tool invoking a subagent]
    │   ├── chat
    │   ├── search_hotels [tool]
    │   └── chat
    └── chat

This structure shows which agent performed each action and what caused the work to begin. Preserve trace context when execution crosses task, thread, or service boundaries so the children remain attached to the correct parent.

OpenTelemetry's gen_ai.operation.name describes operations such as chat, execute_tool, and invoke_agent. Backends can also use their own attributes for presentation. In the Laminar examples, lmnr.span.type identifies LLM, TOOL, or DEFAULT spans, while lmnr.span.input and lmnr.span.output hold JSON-stringified inputs and outputs.

Keep run-level identifiers, such as the session and user, on the run span using the destination's supported association fields. Use stable span names; put changing identifiers in attributes.

Exporting spans with the OpenTelemetry SDK#

Configure tracing once at application startup. If your application already has a tracer provider, add the exporter to that setup instead of creating a competing provider.

This Python configuration sends spans to Laminar over OTLP/gRPC:

import json
import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import SpanKind

provider = TracerProvider(
    resource=Resource.create({"service.name": "travel-agent"})
)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://api.lmnr.ai:8443/v1/traces",
            headers={
                "authorization": f"Bearer {os.environ['LMNR_PROJECT_API_KEY']}"
            },
        )
    )
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("travel-agent")

The following instrumentation skeleton separates the agent run, model calls, and tool execution. call_model, execute_tool, and build_followup represent your application code. The model adapter returns model, input_tokens, output_tokens, and messages; messages use the GenAI schema. The first response also includes the selected tool's tool_name and tool_arguments.

def traced_model_call(model, messages):
    with tracer.start_as_current_span(
        f"chat {model}",
        kind=SpanKind.CLIENT,
        attributes={
            "gen_ai.operation.name": "chat",
            "gen_ai.provider.name": "openai",
            "gen_ai.system": "openai",  # Legacy compatibility key
            "gen_ai.request.model": model,
            "gen_ai.input.messages": json.dumps(messages),
            "lmnr.span.type": "LLM",
        },
    ) as span:
        response = call_model(model, messages)
        span.set_attributes({
            "gen_ai.response.model": response["model"],
            "gen_ai.usage.input_tokens": response["input_tokens"],
            "gen_ai.usage.output_tokens": response["output_tokens"],
            "gen_ai.output.messages": json.dumps(response["messages"]),
        })
        return response


def run(model, messages):
    with tracer.start_as_current_span(
        "agent.run",
        attributes={
            "lmnr.span.type": "DEFAULT",
            "lmnr.span.input": json.dumps(messages),
        },
    ) as run_span:
        response = traced_model_call(model, messages)
        tool_name = response["tool_name"]
        arguments = response["tool_arguments"]

        with tracer.start_as_current_span(
            f"execute_tool {tool_name}",
            attributes={
                "gen_ai.operation.name": "execute_tool",
                "gen_ai.tool.name": tool_name,
                "lmnr.span.type": "TOOL",
                "lmnr.span.input": json.dumps(arguments),
            },
        ) as tool_span:
            result = execute_tool(tool_name, arguments)
            tool_span.set_attribute("lmnr.span.output", json.dumps(result))

        followup = build_followup(messages, response, result)
        answer = traced_model_call(model, followup)
        run_span.set_attribute("lmnr.span.output", json.dumps(answer["messages"]))
        return answer

This illustrates one tool call followed by a final model call. A full agent loop also handles responses without tool calls, multiple tool calls, and iteration limits. Apply the same span boundaries to each iteration. If a tool runs a subagent, its instrumented calls inherit the active tool span's context.

With the default Python context-manager behavior, an exception escaping the span is recorded and marks the span as an error. If you catch a tool failure inside the span and let the agent recover, explicitly record that failure and set the tool span's error status.

For Node.js, pass gRPC authorization through a Metadata object:

import { Metadata } from '@grpc/grpc-js';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';

const metadata = new Metadata();
metadata.set('authorization', `Bearer ${process.env.LMNR_PROJECT_API_KEY}`);

const exporter = new OTLPTraceExporter({
  url: 'https://api.lmnr.ai:8443/v1/traces',
  metadata,
});

Attach the exporter to your provider through a BatchSpanProcessor. For runtimes where gRPC is unsuitable, use OTLP/HTTP with https://api.lmnr.ai:443/v1/traces and an authorization header. Laminar's transport documentation lists supported formats and deployment endpoints.

Batching and flushing#

Agent runs can produce many spans in a short burst. Three batch-processor settings control how those spans leave the application:

A larger queue can absorb a burst, but it cannot fix a destination that consistently receives data more slowly than the application produces it.

Before a CLI process exits or a serverless invocation freezes, allow pending spans to export. In Python, provider.force_flush(timeout_millis=5000) requests a bounded flush; check its result if delivery matters. Call provider.shutdown() when the process is finished. For long-running services, drain telemetry during graceful shutdown.

Sending traces to Laminar and an existing APM backend#

You can send the same spans to Laminar and your existing Datadog or Grafana setup. For a single service, attach two batch processors with different exporters to the same provider. Each receives the completed spans.

For multiple services, an OpenTelemetry Collector centralizes routing and backend credentials. Applications export to the Collector once; the Collector forwards the traces to each destination.

The configuration below sends traces to Laminar and Grafana Cloud. Set the Grafana endpoint and credentials for your stack:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 2s
    send_batch_size: 512

exporters:
  otlp/laminar:
    endpoint: api.lmnr.ai:8443
    headers:
      authorization: "Bearer ${env:LMNR_PROJECT_API_KEY}"
  otlphttp/grafana:
    endpoint: ${env:GRAFANA_OTLP_ENDPOINT}
    headers:
      Authorization: "Basic ${env:GRAFANA_OTLP_BASIC_AUTH}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/laminar, otlphttp/grafana]

For Datadog, use a Collector distribution containing its exporter, such as the contrib distribution, and add a datadog exporter with your API key and site. Include that exporter in the traces pipeline. Use separate pipelines when destinations need different filtering or sampling.

This gives teams access to the same execution in different contexts: service dependencies and infrastructure in their APM workflow, and model messages, tool results, and usage in their agent-debugging workflow.

Sampling agent traces#

Head sampling decides whether to retain a trace before its outcome is known. It can therefore discard failed runs along with successful ones. Keeping all agent traces is useful while debugging or when volume permits, but retention should reflect traffic, payload size, and cost.

When sampling is necessary, tail sampling lets a Collector consider the spans it has received before deciding. For example, a policy can retain error traces and slow runs while sampling the remainder. Size the decision window and buffering for agent duration: a fixed 30-second wait may miss the outcome of a multi-minute run.

If your agent backend needs every run while your APM backend needs a sample, route them through separate pipelines and apply sampling only to the APM pipeline. Upstream instrumentation must still record and export those spans; a Collector cannot recover spans already dropped by head sampling.

Using an SDK alongside OpenTelemetry#

Manual instrumentation gives you control over span boundaries and attributes. An agent-observability SDK reduces the work of capturing provider responses, token usage, and framework operations.

Laminar's SDK uses OpenTelemetry, so SDK-generated spans and manual OTel spans can participate in the same trace when they share tracing context. Use automatic instrumentation for supported model and framework calls, then add manual spans for application logic or integrations it does not cover. Avoid instrumenting the same model call twice.

Troubleshooting export#

For Laminar cloud, start by checking the transport, port, path, and authorization. These symptoms suggest checks rather than uniquely identifying a cause:

Laminar's troubleshooting reference includes exporter-specific examples. If export succeeds but spans are missing, check sampling and process shutdown. If spans arrive but model calls render as ordinary operations, check their type and message attributes.

What Laminar adds to the trace#

Laminar accepts OpenTelemetry traces without requiring its SDK. With supported attributes, it can present model messages and tool inputs and outputs alongside the execution tree, and calculate model costs from reported usage. The instrumentation examples above use lmnr.* fields where Laminar-specific rendering is needed.

The trace structure and exported attributes remain available to other destinations. You can use Laminar for agent debugging while retaining the same trace in your existing observability pipeline.

FAQ#

Does OTLP support mean every backend will display the same agent trace?

No. OTLP standardizes how spans are transported, but backends differ in how they interpret attributes. Two backends can receive identical spans and present different views of messages, tool calls, and usage. Check both OTLP ingestion and support for the attributes your instrumentation emits, including any backend-specific fields needed for rendering or cost calculation.

Why do my tool calls or subagent runs appear as separate traces?

A common cause is lost parent context when execution moves into another task, thread, or service. Check that the child span starts with the intended parent active. Across service or queue boundaries, pass trace context with the request or message and restore it before creating child spans. Adding span-type attributes cannot repair a missing parent relationship. Laminar's continuing traces guide shows how to pass context between services using its SDK.

How can I tell whether an OpenTelemetry integration captures the whole agent run?

Inspect a test run that includes a model call, a tool invocation, and a final model response. Confirm that each operation appears once, under the expected parent, and that each model call carries its own model identifier and reported usage. Include a subagent invocation if your application uses one, then test a tool failure to verify that the failed operation is visible even if the agent recovers. This reveals gaps that a successful export alone will not catch.