Skip to main content
AI agents often fail without throwing an error. They cite the wrong document, click the wrong button, or present an unsupported claim as fact. These mistakes are an inherent risk of relying on probabilistic models, yet a run can look healthy to traditional observability systems: every tool call succeeds, latency stays normal, and no errors are logged. An alert based on those metrics cannot tell you that a legal agent cited the wrong document. Detecting these failures requires understanding what the agent was asked to do and how it went about doing it. That evidence lives in its trace: the inputs, recorded reasoning, tool calls, and outputs that show how the run unfolded. Laminar’s Signals analyze agent traces to identify failures and turn them into structured events you can monitor and alert on.

​
From traces to structured events

Signals are powered by our trace analysis agent. The Signals agent takes three inputs:
  • Trace: the agent run to analyze.
  • Prompt: what to look for.
  • JSON schema: what to record when it finds a match.
A trace and a Signal's prompt and schema go into the Signals agent, which outputs a signal event. In the trace, a coding agent runs tests: one test fails on a missing function, the end-to-end test fails on a different error, and the unit tests pass. Its final report blames both failures on the same missing function. The prompt asks whether the agent's final report states a conclusion that its own tool outputs contradict, with a schema of claim, evidence and claim_type. The Signals agent outputs an event: claim_type root_cause, evidence 'e2e test: different error', severity critical.

A coding agent's report contradicts its own test output, and the Signals agent records it as a structured event.

A prompt and a schema together define a Signal. The Signals agent reads the relevant steps in each trace and returns a signal event with your schema fields and links to its evidence. If it finds no match, it returns no event.
The lifecycle of a Signal run. A trace finishes and is checked against the Signal's trigger, sample rate and filters; a trace that fails them is skipped and no run is recorded. A trace that passes goes to the Signals agent; if it finds no match, the run completes with no event. If it finds a match, it returns a signal event with its trace_id, claim_type root_cause and severity critical. The event joins a cluster, is stored in the signal_events table for SQL, and is checked against alert rules, which notify Slack or email on live runs.

Each finished trace is checked against the Signal's trigger and filters, then read by the Signals agent. A matching trace produces an event that is grouped into a cluster, stored for SQL, and checked against your alerts.

To set when Signals agent starts processing your trace you can configure triggers and trace filters, or use a backfill to analyze historical traces. Once a signal event is created, its structured fields are available to query in SQL. You can filter events, count how often a behavior occurs, and compare outcomes across traces. Laminar automatically groups similar events into clusters based on the meaning of their findings. Clusters help you spot patterns across thousands of events, see which issues recur, and understand how common they are. On top of that, you can configure alerts to notify you in Slack or by email when a new signal event matches your conditions or a new cluster forms.

​
Example: catch a report that contradicts its own evidence

In the trace above, a coding agent reports its work as done despite two failing tests. It blames both failures on the same pre-existing problem, but one test log shows a different error. Every tool call succeeded. The mistake is in the agent’s reasoning, and if you trust its report, you’ll look for the wrong bug. To find reports that contradict the agent’s own evidence, we created a Signal with the following prompt:
Does the agent’s final report state a conclusion that its own tool outputs contradict? Record the claim and the evidence against it.
The schema defines three fields:
  • claim: the statement in the final report.
  • evidence: the tool output that contradicts it.
  • claim_type: one of root_cause, completion, verification, or other.
For this trace, the event looks like this:
Now, we can use claim_type across traces to see which kinds of claims our agent gets wrong most often.

​
Cost efficient and reliable analysis on every trace

To monitor an agent in production, trace analysis needs to be accurate enough to trust and cheap enough to run on every trace. Sampling leaves failures unseen, while noisy results make alerts hard to act on. Off-the-shelf models often force a tradeoff: cheaper models miss failures or flag problems the evidence doesn’t support, while more capable models make analyzing every trace too expensive at production scale. We trained flow-1 with reinforcement learning specifically for agent trace analysis to meet both requirements:
  • Quality: it matches GPT-6-sol in failure detection quality on our benchmark, while being 23x cheaper.
  • Cost: It would cost just $3.5 on average to process 1k traces of 100k tokens each. Learn more about Signals pricing here, and cap your spend with usage warnings and hard limits.
flow-1 gives GPT-6-sol level trace analysis with 23 times more traces processed per $1. One dot per trace analyzed for $1: GPT-6-sol analyzes 38 traces, flow-1 analyzes 888.

One dot per trace analyzed for $1.

That makes it practical to analyze every production trace with the quality needed to catch subtle failures.

​
Define your own trace analysis

Signals are a general way to analyze traces. You decide what to look for and what to record. A prompt can ask a broad question, combine several conditions, or compare evidence across different steps. The output schema gives you control over the fields and categories in the result. For example, you could ask a Signal to:
  • Detect a specific behavior: Did the agent promise a refund without calling the refund tool? Record the promise and the missing action.
  • Classify a conversation: What was the user trying to accomplish, and was it resolved? Return an intent and an outcome.
  • Extract information: Which integrations did the user ask for? Collect the integration names and the requests they made.
  • Compare evidence: Do the amounts in the contract agree with the accompanying memo? Record any mismatches and where they appear.

​
Get started

If you’re new to Laminar, start by sending your agent’s traces. Every new project includes a Failure Detector Signal. Open Signals in your project to inspect its findings. When you create your own Signal, review its findings on representative traces before relying on its alerts. The results are model judgments, and the linked evidence lets you check them.

Quickstart

Create your first Signal and inspect its events.

CLI

Define and manage Signals from your terminal.

Clusters

Explore recurring patterns across traces.

Alerts

Get notified when a Signal finds an issue or a new event cluster forms.