> ## Documentation Index
> Fetch the complete documentation index at: https://laminar.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# OpenTelemetry
Laminar uses OpenTelemetry for tracing. That is, Laminar's backend is compatible with OpenTelemetry.
This page details the methods to send your OpenTelemetry traces to Laminar, even if you are not using Laminar's SDK.
If you are using Laminar's SDK, refer to the [tracing](/tracing/introduction) page to get started.
Still feel free to read this page to get basic understanding of the internals of OpenTelemetry and how it works with Laminar.
## Introduction
OpenTelemetry is a framework for tracing and monitoring distributed systems. OpenLLMetry is
an extension of OpenTelemetry that is specifically designed for LLM tracing. The standards
introduced in OpenLLMetry are gradually being adopted by OpenTelemetry.
OpenTelemetry covers tracing, metrics, and logs. Laminar focuses on tracing today; limited log ingestion
is available for some SDK integrations and can be queried in the SQL Editor. For more details about
tracing in Laminar, and a quick intro to the core concepts, see the [Tracing](/tracing/introduction) page.
Processing of OpenTelemetry traces has many concepts and components, namely, exporters,
processors, propagators, samplers, and collectors.
For the purposes of this page, we will focus on the exporters.
### Exporters
Exporter is the client-side component that sends spans (and thus traces) to an OpenTelemetry-compatible backend.
It is responsible for the payload format and the transport protocol.
The default protocol is [OTLP](https://opentelemetry.io/docs/specs/otel/protocol/),
which underneath uses protobuf in one of the following formats:
* Protobuf in gRPC (commonly known as OTLP/gRPC)
* Protobuf over HTTP with binary encoding (commonly known as OTLP/HTTP/proto)
* Protobuf over HTTP with JSON encoding (commonly known as OTLP/HTTP/json, is used rarely).
Laminar's backend (both cloud and self-hosted) supports OTLP/gRPC and OTLP/HTTP/proto formats.
OTLP/gRPC is recommended for performance and reliability reasons.
Laminar's backend does not support OTLP/HTTP/json.
Here's a quick comparison of formats that Laminar supports:
| | OTLP/gRPC | OTLP/HTTP/proto |
| ------------------------------------ | --------------------- | --------------------- |
| Supported | ✅ | ✅ |
| Recommended | ✅ | ❌ |
| Underlying protocol | gRPC over HTTP/2 | HTTP/1.1 |
| Encoding format | protobuf | protobuf |
| Base URL for Laminar cloud | `https://api.lmnr.ai` | `https://api.lmnr.ai` |
| Port at Laminar cloud | `8443` | `443` |
| Default port for self-hosted backend | `8001` | `8000` |
| Path | `/v1/traces` \[1] | `/v1/traces` \[1] |
\[1] In OpenTelemetry Node and Python SDKs,
this path is appended (unless specified explicitly) in gRPC exporter, but not in the HTTP exporter.
## Getting started
Usually, OpenTelemetry tracing in other libraries is initialized once at the start of the application
by calling a function like `initTracer()`. Functions like this usually accept a configuration object
or a set of parameters, including the exporter configuration.
To send traces to Laminar, you need to configure the endpoint and the authorization.
### Example
```javascript theme={null}
import { Metadata } from '@grpc/grpc-js';
// it is important to import the OTLP exporter from the
// `@opentelemetry/exporter-trace-otlp-grpc` package
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
const metadata = new Metadata();
metadata.set('authorization', `Bearer ${process.env.LMNR_PROJECT_API_KEY}`);
const exporter = new OTLPTraceExporter({
url: "https://api.lmnr.ai:8443/v1/traces",
metadata,
});
functionThatInitiatesTracer({
exporter
});
```
```python theme={null}
# it is important to import the OTLP exporter from the
# `opentelemetry.exporter.otlp.proto.grpc.trace_exporter` package
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter
)
import os
exporter = OTLPSpanExporter(
endpoint="https://api.lmnr.ai:8443/v1/traces",
# important: `authorization` starts with a lowercase letter
headers={"authorization": f"Bearer {os.getenv('LMNR_PROJECT_API_KEY')}"},
)
function_that_initiates_tracer(
exporter=exporter,
# other parameters
)
```
### Endpoint
The default base url for the endpoint is `https://api.lmnr.ai` listening
for gRPC traffic on port `8443`.
For the self-hosted backend, the base url is `http://`
and the default port is `8001` for gRPC or `8000` for HTTP, unless you have changed the configuration.
The `/v1/traces` path is the default OpenTelemetry trace endpoint, and Laminar listens
at this path. In both JavaScript (OpenTelemetry Node SDK) and Python OpenTelemetry SDKs,
the gRPC implementation appends `/v1/traces` to the base url, if you don't specify it.
Be careful though if you are using the HTTP exporter, as the HTTP implementation does not append it.
### Authorization
The authorization header is required to send traces to Laminar. We use bearer authentication,
so the header should start with `Bearer ` and the token is your
[project API key](/tracing/introduction#getting-started).
The right way to set the headers for gRPC requests is to use the metadata object. Even though
gRPC is sent over HTTP/2, and metadata is sent as HTTP headers, it is different from
raw HTTP headers. Learn more about metadata in the [gRPC documentation](https://grpc.io/docs/guides/metadata/).
If you specify the raw HTTP headers for a gRPC request, Laminar will not be able to process them.
This is especially relevant for the exporter in Node.js, which accepts both
`metadata` and `headers` parameters. The latter is effectively ignored by Laminar backend.
gRPC metadata has a restricted set of keys, and so many client implementations,
including the OpenTelemetry Python SDK, check the keys for validity,
effectively imposing case-sensitivity. That is, `authorization` has to start with
lowercase `a`.
At the time of writing, Python SDK throws an error if the key is
`Authorization` with capital `A`. If you are specifying the key as `Authorization`
and not seeing an error saying `TypeError: not all arguments converted during string formatting`,
you are probably using the HTTP exporter, which is not recommended.
### Troubleshooting
If you are facing issues, please refer to the [Troubleshooting OpenTelemetry](/tracing/troubleshooting-opentelemetry) page.
## Span reference
This section is solely for information purposes.
As a user of Laminar, you don't have to deal with the internals of OpenTelemetry.
### Span object
| Attribute | Description | Type | Laminar representation (if different) | Example |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------- | -------------------------------------- |
| trace\_id | Unique identifier for the trace | 16-bytes | UUID \[1] | `01234567-89ab-4def-0123-456789abcdef` |
| span\_id | Unique identifier for the span | 8-bytes | UUID \[1] | `00000000-0000-0000-0123-456789abcdef` |
| parent\_span\_id | Unique identifier for the parent span | 8-bytes | UUID \[1] | `00000000-0000-0000-0123-456789abcdef` |
| name | Name of the span | string | | `my_function` |
| events | Events associated with the span | Array\ | | |
| attributes | Attributes associated with the span. Fully compatible with the `gen_ai` [semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-spans.md) | Key-value pair. Key must comply to semantic conventions, value must be of `AttributeType` \[2] | | `{"gen_ai.usage.output_tokens": 369}` |
| start\_time\_unix\_nano | Start time of the span in nanoseconds \[3] | number | timestamp with UTC timezone | `1630000000000000000` |
| end\_time\_unix\_nano | End time of the span in nanoseconds \[3] | number | timestamp with UTC timezone | `1630000000000000000` |
\[1] 13th digit in hex UUID depends on UUID version, so technically converting [OTel/WC3's](https://www.w3.org/TR/trace-context/#trace-id) 16-byte and, especially, 8-byte IDs
to UUIDs is not exactly UUID.
\[2] `AttributeType` is a union of `string`, `number`, `boolean`, `Array`, `Array`, `Array`
\[3] In most OpenTelemetry client implementations, you don't have to convert the timestamp to nanoseconds manually,
you can simply pass the `Date` / `datetime` object and the client will convert it to nanoseconds.
### Span attributes
Span attribute values that Laminar bases some of its functionality on:
| Attribute | Description | Type | Example |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | --------------------------------- |
| `lmnr.span.type` | Type of the span | string (`LLM` or `DEFAULT`) | `LLM` |
| `lmnr.span.path` | Path of the span | string | `agent.generate.openai.chat` |
| `gen_ai.system` | Model provider | string | `openai` |
| `gen_ai.usage.output_tokens` | Number of tokens the LLM produced | number | 369 |
| `gen_ai.usage.input_tokens` | Number of tokens in the LLM input | number | 42 |
| `llm.usage.total_tokens` | Total number of tokens in the LLM input and output. If not specified, sum of `input_tokens` and `output_tokens` | number | 411 |
| `gen_ai.usage.cost` | Cost of the LLM call. If not specified, `system`, `input_tokens`, and `output_tokens` are used to estimate the cost Laminar-side in the background | number | 0.012 |
| `gen_ai.usage.input_cost` | Cost of the inputs to the LLM call. If not specified, `system` and `input_tokens` are used to estimate the cost Laminar-side in the background | number | 0.003 |
| `gen_ai.usage.output_cost` | Cost of the outputs of the LLM call. If not specified, `system`, and `output_tokens` are used to estimate the cost Laminar-side in the background | number | 0.009 |
| `gen_ai.usage.request_model` | Model name specified in the request | string | `gpt-4o` |
| `gen_ai.usage.response_model` | Model name returned in the response | string | `gpt-4o-2024-08-06` |
| `gen_ai.prompt.{i}.content` | Content of the input message number i (starting from 0) | string | `write a poem about laminar flow` |
| `gen_ai.prompt.{i}.role` | Role of the input message number i (starting from 0) | string | `user` |
These *and* all other attributes are stored as key-value pairs in the `attributes` field of the span.