How it works

Apgar turns the traces of an agent into turns, groups turns by release, and compares each new release with a baseline release until it can say whether the new release is worse.

​

Ingest

Traces arrive in one of two ways:

• An OpenTelemetry SDK or collector pushes OTLP/HTTP to /v1/traces with an ingest token. See OpenTelemetry.
• The worker pulls observations from a Langfuse project. See Langfuse.

Both sources go through the same normalizer. It classifies each span (workflow, agent, chat, tool, retrieval, or other), keeps a small set of attributes, and drops message content after reducing it to a character count and a JSON validity flag. Tool arguments are replaced by a keyed hash. See Data handling.

​

Turns

A turn is one trace: one request to the agent and everything it did to answer. The worker assembles a trace into a turn 60 seconds after its last span once its span without a parent has arrived, or 10 minutes after its first span arrived. Spans that arrive later update the turn, but its release, start time, and agent stay as first assembled. A turn records its release, environment, session, surface, prompt version, models, LLM calls and errors, tool calls, errors, retries, loops, tokens, cost, output length, and latency. A session is the conversation a turn belongs to. Statistics are computed over sessions, so a few long conversations cannot outweigh many short ones. See Sessions.

​

Releases

When a turn carries a release identifier that the agent has not seen before, Apgar registers a new release, records when it was first seen, and sends a New release message. See Releases and baselines.

​

Evaluations

The worker evaluates a release every 5 minutes for the first 2 hours and every 15 minutes after that. Each evaluation:

1. Picks the baseline: by default the most recent earlier release that did not fail, over its own traffic in the 7 days before the new release appeared.
2. Computes each metric on both sides, with an interval from a bootstrap over sessions.
3. Fails a metric when the whole interval lies past its tolerance on the bad side, and passes it when the interval lies inside the tolerance.
4. Breaks every metric down by tool, surface, session depth, response model, and prompt version, to say where a failing change is concentrated.

A result becomes a verdict only after it repeats on two evaluations at least 10 minutes apart. See Verdicts.

​

Messages

Apgar sends a message when a release is detected, when a regression is confirmed, when a release is confirmed healthy, and when monitoring ends without a decision because the monitoring window ran out or someone stopped it. Messages go to Slack, to signed webhooks, or both. See Slack and Webhooks.

​

End of monitoring

Monitoring of a release ends when a newer release of the same agent records its first turn, when 72 hours have passed since the release was first seen, or when someone stops it. A last check then decides the release. After that, only recording expected changes can change the result.

​

Labels

The UI labels a release by its state: First release, Collecting, Possible regression, Looks healthy, Not enough data, Regression, Healthy, and Unconfirmed. Messages use only the four titles that matter in a channel: New release, Regression, Healthy, and Unconfirmed. The state table in Verdicts maps every label to its engine state.