# Apgar > Release health for AI agents. > ## Agent Instructions > Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev > Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md # Use with AI agents Source: https://docs.useapgar.dev/ai-agents Give your coding agent the Apgar skill and the docs MCP server. Coding agents such as Claude Code, Cursor, and Codex set up Apgar more reliably with two things: the Apgar skill and the docs MCP server. The skill tells the agent what Apgar is, which pages to read for each task, and which mistakes leave a release unevaluated without an error. The MCP server lets the agent search and read these docs. Install both. ## Install the skill ```bash theme={null} npx skills add https://docs.useapgar.dev ``` The [skills CLI](https://www.npmjs.com/package/skills) finds the skill on this site, asks which agents to install it for, and installs it into the current project. Add `-g` to install it for every project. The skill checks on each use whether a newer version is published here, and uses that one when it is. To update the installed copy: ```bash theme={null} npx skills update apgar ``` ## Connect the docs MCP server The server is at `https://docs.useapgar.dev/mcp` and needs no authentication. It can search these docs and read any page. **Claude Code** ```bash theme={null} claude mcp add --transport http apgar-docs https://docs.useapgar.dev/mcp ``` Run `/mcp` in a Claude Code session to confirm the connection. **Codex** ```bash theme={null} codex mcp add apgar-docs --url https://docs.useapgar.dev/mcp ``` **Cursor** [Install in Cursor](https://cursor.com/install-mcp?name=apgar-docs\&config=eyJ1cmwiOiJodHRwczovL2RvY3MudXNlYXBnYXIuZGV2L21jcCJ9), or add the server to `mcp.json`: ```json theme={null} { "mcpServers": { "apgar-docs": { "url": "https://docs.useapgar.dev/mcp" } } } ``` **VS Code** ```bash theme={null} code --add-mcp '{"name":"apgar-docs","type":"http","url":"https://docs.useapgar.dev/mcp"}' ``` Other clients connect to the same URL as a streamable HTTP server. ## Ask your agent Paste this into your coding agent after installing the skill: ```text theme={null} Use the Apgar skill. Look at this codebase and tell me what has to change so that every release of this agent can be compared in Apgar: the traces, the release attribute, and anything the instrumentation leaves out. Do not change code until I agree. ``` ## Markdown versions Every page is also served as Markdown: append `.md` to its URL. [llms.txt](https://docs.useapgar.dev/llms.txt) lists every page, and [llms-full.txt](https://docs.useapgar.dev/llms-full.txt) contains all of them in one file. The menu at the top of each page copies the page as Markdown or opens it in ChatGPT or Claude. # Metrics Source: https://docs.useapgar.dev/concepts/metrics What Apgar measures, how much change it allows, and where a change is concentrated. Apgar compares each release with the previous one on eleven metrics, computed from turns and tool calls. Guardrail metrics decide the verdict. Informational metrics are shown on the release page and never decide it. Every metric is a guardrail by default except **Turns per session**, which is informational. ## Catalog The release page groups the metrics the same way. ### Behavior | Metric | What it measures | Worse when | Default tolerance | | - | - | - | - | | **Tool calls per turn** | Average number of tool calls per turn. A drop usually means the agent stopped using its tools | Down | 10 % | | **Turns without tools** | Share of turns that finished without calling any tool | Up | 5 pp | | **Retry rate** | Share of tool calls that repeat a failed call with the same tool and arguments in the same turn | Up | 2 pp | | **Loop rate** | Share of turns with three or more consecutive identical tool calls | Up | 1 pp | | **Output length** | Median output tokens per turn | Down | 15 % | | **Format failure rate** | Share of turns whose final response is not valid JSON. Only for agents with output format `json` | Up | 1 pp | | **Turns per session** | Average number of turns in closed sessions. A session is closed when its last turn ended more than 30 minutes ago | Up or down | 25 % | ### Reliability | Metric | What it measures | Worse when | Default tolerance | | - | - | - | - | | **Tool error rate** | Share of tool calls that ended with an error | Up | 2 pp | | **LLM error rate** | Share of turns where at least one LLM call failed | Up | 1 pp | ### Cost and latency | Metric | What it measures | Worse when | Default tolerance | | - | - | - | - | | **Cost per turn** | Average cost per turn in US dollars, as reported on the chat spans | Up | 20 % | | **Latency p95** | 95th percentile of turn latency, measured on the root span | Up | 15 % | **Output length** counts tokens when at least 95 % of turns on both sides have token counts, and characters otherwise. The release page shows the unit. Rates and averages are taken over the whole release. **Tool error rate** is all failed tool calls divided by all tool calls, not an average of per turn rates. When several metrics fail, messages and the release page list them in the group order above, and inside a group by how far they are past their tolerance. ## Tolerances A tolerance is the largest change that still counts as no regression. **Worse when** says which direction counts; a change in the other direction never fails a release. Rates have a tolerance in percentage points (pp). A percentage point is the plain difference between two percentages: a tool error rate that goes from 2 % to 4 % changes by +2 pp, even though it doubled. With the default tolerance of 2 pp, that is the largest change that still counts as no regression. The other metrics have a tolerance in percent of the previous value. Latency that goes from 2.0 s to 2.4 s changes by +20 %. The **Change** column on the release page uses the same units. A guardrail fails only when Apgar is confident that the change is past the tolerance, not as soon as the measured change crosses it. See [How a metric is judged](/concepts/verdicts#how-a-metric-is-judged). ## Change the defaults Per agent, in **Agent settings, Metrics**: | Setting | Meaning | | - | - | | **Enabled** | Turns the metric on or off for this agent. **Format failure rate** stays off until the output format in **General** is `json` | | **Tolerance** | The allowed change, in pp for rates and in % for the other metrics | | **Role** | **guardrail** decides the verdict, **informational** is only shown | Making a guardrail informational keeps it visible without letting it fail a release. To leave a metric out for one release only, use [expected changes](/concepts/releases#expected-changes). ## When a metric is unavailable Some metrics need data that not every instrumentation records. When that data is missing, or the metric does not apply, the release page lists the metric as unavailable under its group, with the reason. | Metric | Unavailable when | What to do | | - | - | - | | **Tool calls per turn**, **Turns without tools**, **Retry rate**, **Tool error rate** | The previous release made no tool calls | Nothing. An agent without tools is judged on the other guardrails | | **Retry rate** | Fewer than 90 % of tool calls on one side carry their arguments | Set `gen_ai.tool.call.arguments` on tool spans. See [Tools](/send-traces/attributes#tools) | | **Loop rate** | Fewer than 90 % of turns on one side carry arguments on every tool call | Same as **Retry rate** | | **Format failure rate** | Fewer than 90 % of turns on one side have output content | Capture output content. See [Content](/send-traces/attributes#content) | | **Output length** | Fewer than 95 % of turns on one side have token counts, and fewer than 95 % have output content | Record output token counts, or capture output content. See [Tokens](/send-traces/attributes#tokens) | | **Cost per turn** | Fewer than 90 % of turns on one side have a cost | Set a cost on each chat span. See [Cost](/send-traces/attributes#cost) | | **Turns per session** | Fewer than 10 closed sessions on one side, or more than half of the turns have no session id | Send a session id. See [Sessions](/concepts/sessions) | | Any metric | It is turned off for this agent | Turn it on in **Agent settings, Metrics** | | Any metric | It cannot be compared: one side has nothing to measure, the previous value is 0 for a metric with a % tolerance, or the values are too sparse | Wait for more traffic, or make the metric informational | An unavailable guardrail does not hold the release back; the verdict uses the other guardrails. The exception is a guardrail whose previous value is 0 or whose values are too sparse: it keeps the release in **Not enough data** until it can be compared or is made informational. An agent with no available guardrail never reaches **Healthy**. ## Slices A slice is the part of a release's traffic that shares one value of a dimension: the turns that called the `web_search` tool, the turns that came from Slack, or the turns answered by one model. Apgar computes every available metric per slice, for the previous release and this one, to show where a change is concentrated. Select a metric on the release page to see its slices, one table per dimension: | Dimension | Values | | - | - | | **Tool** | Each tool the agent called. A turn that calls two tools counts in both. See [Tools](/send-traces/attributes#tools) | | **Surface** | Each surface, such as Slack or a web chat. See [Surface](/send-traces/attributes#surface) | | **Session depth** | `short` for turns 1 to 5 of a session, `medium` for turns 6 to 20, `long` after that. Turns without a session id have no depth | | **Response model** | Each model that answered. See [Models](/send-traces/attributes#models) | | **Prompt version** | Each prompt version. See [Prompt version](/send-traces/attributes#prompt-version) | Slices never decide the verdict. The whole release is judged; slices only say where the change is. A slice table counts the slices that regressed, and the message for a failing metric sums them up, for example "Concentrated in web\_search." See [Attribution](/concepts/verdicts#attribution). A slice is compared only when both sides have at least **Minimum turns per slice** turns, 100 by default, or that many tool calls for **Tool error rate** and **Retry rate**. Values that are not compared are listed below the tables with the reason: | Reason | Meaning | | - | - | | Too few turns, or too few tool calls | One side is below **Minimum turns per slice** | | Previous value is 0, so a relative change is undefined | The metric has a % tolerance and the previous value in this slice is 0 | | Too sparse for a stable comparison | The slice has too little data for a reliable comparison | | A reason from [When a metric is unavailable](#when-a-metric-is-unavailable) | The metric is unavailable inside this slice | **Turns per session** is sliced by the first turn of each session, except that a session counts in a tool slice when any of its turns called the tool. It has no session depth slice. # Releases and baselines Source: https://docs.useapgar.dev/concepts/releases How Apgar detects a release, picks what to compare it with, and when monitoring ends. A release is one version of an agent, identified by the release attribute on its traces. Apgar compares every release, the candidate, with an earlier release of the same agent, the baseline. ## Release detection A release is created by its first turn. The turn must: * carry a release identifier, so it is not `untagged`, * not be late, and * come from the agent's environment, when the agent has one. The release is first seen at the start of that turn. If an earlier turn of the same release arrives later, the first seen time moves back to it. Apgar then queues an evaluation and sends a **New release** message. The first release of an agent has no baseline and shows as **First release** while its window is open, then as **Unconfirmed**. It becomes the baseline of the next release. ## Baseline selection By default the baseline is chosen automatically. The candidates are the earlier releases of the agent that did not fail, not awaiting a fail confirmation, and not ended with a failing guardrail. Among them, the most recently active release wins, by the time of its last turn. When every earlier release failed, the most recently active one is used anyway and the baseline is marked unverified. A baseline is verified when it is **Healthy**, also while it is still monitored, or when it ended as **Unconfirmed** because a newer release replaced it or the monitoring window ran out, with no failing guardrail. Any other baseline is unverified, including one that is still monitored without a healthy result and one that someone stopped. ## Baseline window The baseline is the baseline release's own traffic in the 7 days before the candidate was first seen. Both sides exclude: * `untagged` turns, * late turns, * turns from another environment, when the agent has one, and * mixed turns, the turns of a session that come after it already ran on another release, unless the agent allows mixed sessions. See [Sessions](/concepts/sessions). While the candidate has been live for less than 24 hours, baseline turns are restricted to the UTC hours of day that the candidate window has covered. A release that ran overnight is not compared with a baseline that ran mostly during the day. From 24 hours on, every hour is covered and no filter applies. The filter is also skipped when less than 24 hours separate the first and last turns of the baseline release in its window, because most hours would have no baseline turns. ## Change the baseline Choose another release in the **Baseline** selector on the release page. Only releases first seen before the candidate are allowed, because a later release has no traffic in the baseline window. The baseline can change only while monitoring runs. A manual baseline is compared over the same window. Changing it queues a check, and confirmation starts over. ## Expected changes Some changes are intended, such as shorter answers after a prompt rewrite. Select **Expected changes** on the release page and pick the metrics. Those metrics show as **expected** in the metric table and no longer decide the verdict of that release. Saving queues a check, and confirmation starts over. Changing the baseline or recording expected changes after a **Regression** lifts it, and the release is evaluated again with the new settings. While the release is monitored, confirmation starts over. Once monitoring has ended, the baseline can no longer change, but expected changes can still be recorded, and the next check decides right away. There is no acknowledge or override action: record an intended regression as an expected change, and fix a wrong comparison by choosing another baseline while the release is monitored. Expected changes can no longer be recorded once the trace data of the release has left [retention](/self-hosting/configuration#retention). ## End of monitoring Monitoring of a release ends at the earliest of: | What happens | The release page shows | | - | - | | A newer release of the same agent receives its first turn | **Replaced by** and the newer release | | The monitoring window runs out | **Monitoring window ended** | | Someone selects **Stop monitoring** on the release page | **Monitoring stopped** | The monitoring window starts when Apgar first sees the release. Set its length with **Monitor each release for up to** in **Agent settings, Verdict rules**: 72 hours by default, from 1 to 168. One turn of a newer release is enough. A newer release is one the agent has not seen before. Rolling back to a release Apgar already knows does not end monitoring, but rolling back under a new release name does. When the agent has an environment, only turns from that environment create releases, so traffic in another environment never ends monitoring. An agent without an environment registers releases from every environment, and they replace each other. When monitoring ends, one last check runs and decides on its own, without waiting for a second confirmation. A failing guardrail ends the release as **Regression**, and all guardrails passing end it as **Healthy**, with the same sample gates as during monitoring: the baseline needs the full minimum sample, and **Healthy** also needs it on the release. Anything else ends the release as **Unconfirmed**, except that a **Regression** holds and a release that is already **Healthy** stays **Healthy** unless the last check fails. When monitoring is stopped or the window runs out, the last check counts turns up to that moment. When a newer release takes over, the last check runs within 5 minutes of the newer release's first turn, so a few late turns of the old release are still counted. After that, only recording expected changes can change the result. # Sessions Source: https://docs.useapgar.dev/concepts/sessions Why a conversation should stay on one release, and how Apgar treats sessions that do not. A session is an ordered sequence of turns: a Slack thread, a chat, a web conversation. Apgar resamples whole sessions to compute intervals, and several signals depend on the position of a turn in its session. ## Send a session id Put a stable conversation id in `apgar.session.id`, `gen_ai.conversation.id`, `session.id`, or `langfuse.session.id` on the root span of every turn. On Slack, key the session by thread. On a messenger, key it by chat. Any channel with a stable conversation id works the same way. Without a session id, every trace is its own session and the session is marked inferred. Verdicts still work, but **Turns per session** is unavailable when more than half of the turns have an inferred session, and inferred turns fall into no **Session depth** slice. ## Keep a session on one release If the first turns of a conversation run on one release and later turns on another, the two sides are not the same kind of traffic: the new release sees mostly first turns, and the comparison is confounded by session depth. Keep a whole session on one release where you can. ## Mixed sessions A session with turns from more than one release is mixed. This happens naturally around a deploy, when a conversation that started before it continues on the new release. A turn is mixed when an earlier turn of its session ran on a different release. Mixed turns are left out of both sides by default. The candidate then consists mostly of sessions that started on it, which is the early sample bias described in [Verdicts](/concepts/verdicts#minimum-sample). Turn on **Include mixed sessions** in **Agent settings, Verdict rules** to count mixed turns on both sides. Do this only when you accept the confound and can say why. A high mixed share shrinks both sides, so the release stays in **Collecting** longer, but it does not change the rules. A share that stays high long after a deploy usually means the session id outlives a conversation, such as a user id or a constant, or the service picks a release per request instead of per session. An `untagged` turn counts as a release of its own, so tagged turns that follow an untagged turn in the same session are mixed. A session id that changes on every turn never makes turns mixed: it makes every session one turn long. ## Turn index and depth Turn index, depth, mixed status, and whether a session is closed are computed when Apgar reads the turns, over the 7 days before the comparison window. A turn of the same session older than that counts as the start of a new session. | Depth | Turn index | | - | - | | `short` | 1 to 5 | | `medium` | 6 to 20 | | `long` | above 20 | A session is closed when its last turn ended more than 30 minutes ago. ## Sessions and the end of monitoring A session that is still open when a newer release records its first turn does not keep the older release in monitoring. Monitoring ends on that first turn, or at the cap. See [End of monitoring](/concepts/releases#end-of-monitoring). # Verdicts Source: https://docs.useapgar.dev/concepts/verdicts How an evaluation becomes a verdict, and what each label means. Every evaluation compares the candidate release with its baseline on each applicable metric. A verdict is the state of the release after its latest evaluation. ## How a metric is judged For each metric, Apgar computes the change from baseline to candidate and an interval around it. The interval comes from a cluster bootstrap: sessions are resampled in both arms with Poisson weights, 10000 times by default. Resampling whole sessions keeps one long conversation from counting as many independent turns. A guardrail metric then gets a status, shown in the **Status** column of the metric table on the release page: | Status | Rule | | - | - | | **regressed** | The whole interval lies past the tolerance on the degradation side. Until the regression is confirmed, the status is **possible** | | **within tolerance** | The whole interval stays within the tolerance. For a metric checked in both directions, within the tolerance on both sides | | **not enough data** | The interval crosses the tolerance | | **expected** | The metric is listed as an expected change for this release | | Listed as unavailable under its group | The metric is not applicable to this agent or sample. See [When a metric is unavailable](/concepts/metrics#when-a-metric-is-unavailable) | Informational metrics show **informational**. The significance level per guardrail is 0.05 / N, where N is the number of applicable guardrails (Bonferroni). Informational metrics are computed and shown, and never decide the verdict. ## States The release page and the messages show the label. [Webhooks](/notifications/webhooks) also send the state and the reason behind it, as `verdict` and `reason`. | Label | Meaning | Webhook `verdict` | Webhook `reason` | | - | - | - | - | | **First release** | No earlier release to compare with | `PENDING` | `no_baseline` | | **Collecting** | The release is below the minimum sample and no guardrail fails | `PENDING` | `insufficient_sample` | | **Collecting** | The baseline has fewer turns or sessions than the minimum sample | `PENDING` | `insufficient_baseline` | | **Not enough data** | At least one interval is wider than its tolerance | `INCONCLUSIVE` | `insufficient_power` | | **Possible regression**, or **Possible regression, early** | A guardrail failed; the next check confirms or clears it | `INCONCLUSIVE` | `fail_candidate` | | **Looks healthy** | Every guardrail passed; the next check confirms it | `INCONCLUSIVE` | `pass_candidate` | | **Regression** | A guardrail failure was confirmed | `FAIL` | `confirmed` | | **Healthy** | Every guardrail passed, confirmed | `PASS` | `confirmed` | | **Unconfirmed** | Monitoring ended without a decision: a newer release replaced it, the monitoring window ran out, or someone stopped it | `INCONCLUSIVE` | `superseded`, `window_elapsed`, `stopped` | Only **Healthy** means every guardrail was confirmed within tolerance. **Looks healthy**, **Not enough data**, and **Unconfirmed** mean no regression was found, but the sample could not yet rule out a change as large as the tolerance. ## Minimum sample By default each side needs 200 turns (**Minimum turns**) and 30 sessions (**Minimum sessions**). The two possible claims have different gates, because a small sample already widens the interval: 1. Metrics are computed once the release has 20 turns and 5 sessions (or the minimum sample, when that is lower) and the baseline has the full minimum sample. Below that the release stays **Collecting**. 2. A failing guardrail can move the release to **Possible regression** from that early floor, and then to **Regression**. Such a result is marked early, and the message title reads **Regression, early**. 3. **Looks healthy**, **Healthy**, and **Not enough data** need the full minimum sample. The first turns after a deploy are not always representative: cold caches, internal testers, and mostly first turns of new sessions. For small changes this is a bias, but the interval on a small sample rejects small changes anyway. The early path exists for large jumps, where the bias does not matter. ## Confirmation A result becomes a verdict after it repeats on consecutive checks: **FAIL confirmations** and **PASS confirmations** both default to 2, and the checks must span at least 10 minutes from the first to the last. Confirmation counts the newest run of consecutive checks with the same outcome against the same baseline release. The run starts over when Apgar picks another baseline, or when someone changes the baseline or the expected changes. Set a confirmation count to 1 to decide on the first check. The 10 minute gap is fixed. The last check, when monitoring ends, decides without confirmation. ## After a decision A **Regression** holds. A later check that would pass does not clear it. Only changing the baseline or recording expected changes after the decision lifts it. While the release is monitored, confirmation then starts over. Once monitoring has ended, the baseline can no longer change, but expected changes can still be recorded, and the next check decides right away. A **Healthy** release stays **Healthy** while later checks show only **Not enough data** or **Looks healthy**. A later failing check can still move it to **Possible regression** and then to **Regression**, which sends a second message. ## Schedule The worker checks a release every 5 minutes for the first 2 hours after it was first seen (`APGAR_VERDICT_EARLY_INTERVAL_SECONDS`) and every 15 minutes after that (`APGAR_VERDICT_INTERVAL_SECONDS`). The release page shows the last and the next check. A release with no turns in the last 24 hours is not checked again until its monitoring window ends. Once monitoring has ended, a release is checked again only when someone records expected changes. ## Attribution Every applicable metric is also computed per [slice](/concepts/metrics#slices). Slices are tested with a Benjamini-Hochberg threshold over the same resamples. They never decide the verdict; for a failing metric they say where the change is: * concentrated in the failing slices of a dimension, when that dimension also has a slice with enough turns that is not flagged, * uniform across a dimension, when every slice of it with enough turns fails, * "No single slice stands out.", when no slice is flagged, or * "Attribution pending (N turns).", when no slice has enough turns yet. Concentrated and uniform dimensions combine in one sentence, for example "Concentrated in web\_search; uniform across surfaces." Tool slices are named bare; the other dimensions are prefixed with `surface`, `depth`, `model`, or `prompt`. A failing metric reads as one sentence with its values, the allowed change, and the attribution: ```text theme={null} Tool error rate rose from 2.6% to 27.1% (allowed: up to +2 pp). Concentrated in web_search and fetch_url. ``` Several failing metrics are ordered by group (behavior, reliability, cost and latency), then by how far the interval lies past the tolerance, measured in interval half widths. ## Verdict settings Per agent, in **Agent settings, Verdict rules**: | Setting | Default | Meaning | | - | - | - | | **Minimum turns** | 200 | Turns the release needs for a pass, and the baseline needs for any verdict | | **Minimum sessions** | 30 | Sessions the release needs for a pass, and the baseline needs for any verdict | | **Minimum turns per slice** | 100 | Turns needed on each side of a slice | | **FAIL confirmations** | 2 | Failing checks in a row needed for **Regression** | | **PASS confirmations** | 2 | Passing checks in a row needed for **Healthy** | | **Monitor each release for up to** | 72 hours | Longest monitoring window, 1 to 168 hours | | **Send the healthy message** | On | Send a message when a release becomes healthy | | **Include mixed sessions** | Off | Count turns of sessions that span releases | # How it works Source: https://docs.useapgar.dev/how-it-works From traces to turns, releases, verdicts, and messages. Apgar turns the traces of an agent into turns, groups turns by release, and compares each new release with a baseline release until it can say whether the new release is worse. ![Traces arrive over OTLP/HTTP at POST /v1/traces or are pulled from Langfuse by the worker. Apgar normalizes them into spans, assembles spans into turns and tool calls, discovers releases, runs evaluations, reaches verdicts, and sends messages to Slack and webhooks.](https://mintcdn.com/apgar/CQ6o-L_OizzzJ47E/images/how-it-works-light.svg?fit=max\&auto=format\&n=CQ6o-L_OizzzJ47E\&q=85\&s=9e348629eab4882bad2d983ed5eede13) ## Ingest Traces arrive in one of two ways: * An OpenTelemetry SDK or collector pushes OTLP/HTTP to `/v1/traces` with an ingest token. See [OpenTelemetry](/send-traces/opentelemetry). * The worker pulls observations from a Langfuse project. See [Langfuse](/send-traces/langfuse). Both sources go through the same normalizer. It classifies each span (workflow, agent, chat, tool, retrieval, or other), keeps a small set of attributes, and drops message content after reducing it to a character count and a JSON validity flag. Tool arguments are replaced by a keyed hash. See [Data handling](/privacy/data-handling). ## Turns A turn is one trace: one request to the agent and everything it did to answer. The worker assembles a trace into a turn 60 seconds after its last span once its span without a parent has arrived, or 10 minutes after its first span arrived. Spans that arrive later update the turn, but its release, start time, and agent stay as first assembled. A turn records its release, environment, session, surface, prompt version, models, LLM calls and errors, tool calls, errors, retries, loops, tokens, cost, output length, and latency. A session is the conversation a turn belongs to. Statistics are computed over sessions, so a few long conversations cannot outweigh many short ones. See [Sessions](/concepts/sessions). ## Releases When a turn carries a release identifier that the agent has not seen before, Apgar registers a new release, records when it was first seen, and sends a **New release** message. See [Releases and baselines](/concepts/releases). ## Evaluations The worker evaluates a release every 5 minutes for the first 2 hours and every 15 minutes after that. Each evaluation: 1. Picks the baseline: by default the most recent earlier release that did not fail, over its own traffic in the 7 days before the new release appeared. 2. Computes each [metric](/concepts/metrics) on both sides, with an interval from a bootstrap over sessions. 3. Fails a metric when the whole interval lies past its tolerance on the bad side, and passes it when the interval lies inside the tolerance. 4. Breaks every metric down by tool, surface, session depth, response model, and prompt version, to say where a failing change is concentrated. A result becomes a verdict only after it repeats on two evaluations at least 10 minutes apart. See [Verdicts](/concepts/verdicts). ## Messages Apgar sends a message when a release is detected, when a regression is confirmed, when a release is confirmed healthy, and when monitoring ends without a decision because the monitoring window ran out or someone stopped it. Messages go to Slack, to signed webhooks, or both. See [Slack](/notifications/slack) and [Webhooks](/notifications/webhooks). ## End of monitoring Monitoring of a release ends when a newer release of the same agent records its first turn, when 72 hours have passed since the release was first seen, or when someone stops it. A last check then decides the release. After that, only recording expected changes can change the result. ## Labels The UI labels a release by its state: **First release**, **Collecting**, **Possible regression**, **Looks healthy**, **Not enough data**, **Regression**, **Healthy**, and **Unconfirmed**. Messages use only the four titles that matter in a channel: **New release**, **Regression**, **Healthy**, and **Unconfirmed**. The state table in [Verdicts](/concepts/verdicts#states) maps every label to its engine state. # Introduction Source: https://docs.useapgar.dev/index Release health for AI agents. Apgar watches the first hours of every release of an AI agent and reports whether the agent started behaving worse than it did on the previous release. It reads the traces your team already collects, compares the new release with the one before it on all traffic, and posts a message when a release appears and when it is decided, with a link to one page per release. Apgar needs no LLM judge, no dataset, and no stored prompt or completion text. It runs on your own infrastructure. ## What Apgar checks Each release is compared with a baseline release on these signals: | Group | Signals | | - | - | | Behavior | Fewer tool calls per turn, more turns without a tool call, more tool retries, more loops, shorter answers, more invalid JSON output | | Reliability | More tool errors, more LLM errors | | Cost and latency | Higher cost per turn, higher p95 latency | A signal fails only when the change is larger than its tolerance and the evidence holds across two checks at least 10 minutes apart. When something fails, Apgar says where the change is concentrated: in a tool, a surface, a model, a prompt version, or a session depth. ## What Apgar is not Apgar is not a dashboard, an evaluation harness, or a rollout controller. It compares a release with the previous one, says where the change is concentrated, and leaves the decision to your team. > **[Quickstart](/quickstart)** > > Run Apgar with Docker Compose and look at the demo. > **[How it works](/how-it-works)** > > From traces to turns, releases, and verdicts. > **[OpenTelemetry](/send-traces/opentelemetry)** > > Send traces over OTLP/HTTP. > **[Langfuse](/send-traces/langfuse)** > > Pull traces from a Langfuse project. # Slack Source: https://docs.useapgar.dev/notifications/slack Post one message per release event to a Slack channel. Apgar posts to Slack through an incoming webhook. Each agent can send to one Slack connection. ## Set up **1. Create an incoming webhook** In Slack, create an app with an incoming webhook for the channel that should receive release messages, and copy the webhook URL. It starts with `https://hooks.slack.com/`. **2. Add the connection** In **Settings, Connections**, add a Slack connection with the webhook URL. Apgar stores the URL encrypted. Select **Test** to post a test message to the channel. **3. Choose it for the agent** In **Add agent**, or later in **Agent settings, General**, set **Slack connection** to the new connection. A connection that an agent uses cannot be deleted until the agent is changed or deleted. ## Messages | Event | Title | When | | - | - | - | | `release_detected` | New release | The first turn of a new release arrives | | `regression` | Regression, or Regression, early | A guardrail failure is confirmed. Early when the sample was below the minimum | | `healthy` | Healthy | Every guardrail passed and the pass is confirmed. Only when **Send the healthy message** is on, which is the default | | `ended` | Unconfirmed | Monitoring ended without a decision because the monitoring window ran out or someone stopped it | Open states such as **Collecting** or **Possible regression** never send a message. A release that is replaced by a newer one sends no message of its own; the newer release sends **New release**. A channel usually sees one detection and one decision per release. A release that was **Healthy** and fails later sends a second decision, and so does a release that is evaluated again after a baseline change or an expected change. Each message has a bold title with the agent and the release, and a colored bar: red for a regression, green for healthy, amber for unconfirmed, and blue for a new release. A **New release** message names the baseline, or says that there is nothing to compare yet, and notes when the baseline was not itself confirmed healthy. A regression lists each failing metric as one sentence with its values, the allowed change, and where the change is concentrated, followed by the sample sizes. The attachment ends with a link to the release page at `APGAR_PUBLIC_URL/agents/AGENT/releases/RELEASE`. ```text theme={null} Regression · billing-copilot · release fp-8e2f0c6a1b93 • Tool error rate rose from 0.40% to 9.1% (allowed: up to +2 pp). Concentrated in sql_query. Based on 1,204 turns against 2,950 from fp-3c1e9a7b2d44. Details and actions ``` The plain text fallback, which Slack shows in notifications and which is also the `text` field of a webhook body, carries the link only for **New release** and **Regression**. ## Delivery Messages are queued and sent by the worker. Slack must answer `200`. A network error, a `429`, or a 5xx response is retried after 1, 5, 15, and 60 minutes. After the fifth failed attempt the message is marked failed. Any other response fails at once. **Settings, Notifications** lists every delivery with its status and last error. Select **Retry** on a failed message to queue it again. > [!NOTE] > > Set `APGAR_PUBLIC_URL` to the address people use to open Apgar, or the links in messages will point to `localhost`. # Webhooks Source: https://docs.useapgar.dev/notifications/webhooks Receive signed JSON for every release event. A webhook connection receives the same events as Slack, as signed JSON. Use it to feed a deploy pipeline, an incident tool, or your own bot. ## Set up In **Settings, Connections**, add a webhook connection with: | Field | Rule | | - | - | | `url` | An `http` or `https` URL | | `secret` | 16 to 256 characters. Used to sign each request. Stored encrypted, shown masked afterwards | Every webhook connection receives the events of every agent, demo agents included. Select **Test** to send a signed test request. A 2xx response marks the connection `ok`; any other response or a network error marks it `error`. ## Events | `event` | When | | - | - | | `release_detected` | The first turn of a new release arrives | | `regression` | A guardrail failure is confirmed | | `healthy` | A pass is confirmed and the agent has **Send the healthy message** on | | `ended` | Monitoring ended without a decision because the monitoring window ran out or someone stopped it | | `test` | The connection test | ## Body The body is JSON with sorted keys, no extra whitespace, and non-ASCII characters escaped. | Field | Meaning | | - | - | | `event` | One of the events above | | `idempotency_key` | Stable key of this notification. Retries send the same value | | `agent` | Agent name | | `release` | Release identifier | | `demo` | `true` for demo agents | | `previous_verdict` | State before this transition. `null` for `release_detected` and `test` | | `verdict` | Summary of the release, see below | | `text` | Plain text version of the Slack message. Contains the release link only for `release_detected` and `regression` | | `sent_at` | UTC time the notification was queued, with millisecond precision and a `Z` suffix. A retry resends the identical body, including this value | `verdict` holds: | Field | Meaning | | - | - | | `verdict`, `reason`, `final` | Engine state, reason code, and whether monitoring has ended | | `baseline` | `release`, `mode` (`previous_release` or `manual`), and `unverified` | | `sample` | `candidate_turns`, `baseline_turns`, `candidate_sessions`, `baseline_sessions` | | `window` | `start`, `end`, and `max_end` of the candidate window | | `evaluated_at` | Time of the evaluation | | `early` | `true` when the regression was found below the minimum sample | | `ended` | How monitoring ended, or `null` | | `detectability` | One sentence on the smallest change the sample could detect, or `null` | | `failed_metrics` | One entry per failing guardrail: `id`, `baseline`, `candidate`, `delta`, `ci`, `tolerance`, `tolerance_mode`, `direction`, `unit`, `worst_slice`, `headline` | | `report_url` | Link to the release page | For `release_detected` the summary is the pending state of the release with the baseline Apgar chose. The test request has `event` set to `test`, `idempotency_key` set to `test:` followed by `sent_at`, `agent`, `release`, `previous_verdict`, and `verdict` set to `null`, `demo` set to `false`, and `text` set to `Apgar connection test`. ### Failed metrics Each entry of `failed_metrics` names its metric by `id`. `direction` is the direction that counts as worse: `up`, `down`, or `both`. `delta`, `ci`, and `tolerance` are fractions: with `tolerance_mode` `abs` they are absolute differences, so 0.02 is 2 pp; with `rel` they are relative to the baseline value, so 0.10 is 10 %. | `id` | Metric | | - | - | | `tool_calls_per_turn` | Tool calls per turn | | `zero_tool_turn_rate` | Turns without tools | | `tool_retry_rate` | Retry rate | | `loop_rate` | Loop rate | | `output_tokens_p50` | Output length | | `format_failure_rate` | Format failure rate | | `turns_per_session` | Turns per session | | `tool_error_rate` | Tool error rate | | `llm_error_rate` | LLM error rate | | `cost_per_turn` | Cost per turn | | `latency_p95` | Latency p95 | See [Metrics](/concepts/metrics) for what each one measures. ## Headers | Header | Value | | - | - | | `Content-Type` | `application/json` | | `X-Apgar-Timestamp` | Unix seconds at the time of this attempt | | `X-Apgar-Signature` | Lowercase hex HMAC-SHA256 of `timestamp + "." + body` with the connection secret | | `X-Apgar-Event` | The `event` field | | `User-Agent` | `apgar/VERSION` | ## Verify a request A receiver should: 1. Reject a request whose timestamp differs from its own clock by more than 5 minutes. 2. Compute the HMAC-SHA256 of `X-Apgar-Timestamp`, a period, and the raw body bytes with the connection secret, and compare it with `X-Apgar-Signature` in constant time. Parsing and serializing the JSON again changes the bytes and breaks the signature. 3. Store `idempotency_key` and ignore a delivery whose key it has already accepted. A retry after a timeout can arrive although the first attempt was handled. ```python Python theme={null} import hashlib import hmac import time def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool: if abs(time.time() - int(timestamp)) > 300: return False message = timestamp.encode() + b"." + body expected = hmac.new(secret.encode(), message, hashlib.sha256).hexdigest() return hmac.compare_digest(expected, signature) ``` ```javascript Node.js theme={null} import { createHmac, timingSafeEqual } from "node:crypto"; export function verify(secret, timestamp, body, signature) { if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) { return false; } const expected = createHmac("sha256", secret) .update(`${timestamp}.`) .update(body) .digest("hex"); const left = Buffer.from(expected); const right = Buffer.from(signature); return left.length === right.length && timingSafeEqual(left, right); } ``` ## Idempotency keys | Notification | Key | | - | - | | Release detected | `AGENT:RELEASE:release_detected` | | Transition | `AGENT:RELEASE:PREVIOUS->NEW:EVALUATED_AT` | | Second event of the same transition | The transition key followed by `:EVENT` | Every webhook connection receives the same key for the same notification. ## Retries A network error, a `429`, or a 5xx response is retried after 1, 5, 15, and 60 minutes. After the fifth failed attempt the notification is marked failed. Any other status fails at once. **Settings, Notifications** lists failed deliveries and can queue them again. # Data handling Source: https://docs.useapgar.dev/privacy/data-handling What Apgar stores, what it drops, and where it connects to. Apgar is self hosted. Traces stay in the ClickHouse volume on your host. It needs no account with the project and uploads no trace data. **Prompt text, completion text, and tool arguments are never stored.** ## What is stored A turn stores identifiers, timestamps, counts, and a small set of fields: agent, release, environment, session id, surface, prompt version, models, token counts, cost, output length, and whether the output parsed as JSON. A tool call stores the tool name, its status, an error type, and the argument hash described below. A span keeps its name, kind, timing, status, and an error type of up to 128 characters taken from `error.type`, `exception.type`, or the status message. Spans keep an allowlist of attributes: the ones listed in the [attribute reference](/send-traces/attributes), plus the attributes that agents name in **Release attribute** and **Surface attribute**. For Langfuse, every trace and observation metadata key, the trace tags, and the user id (`langfuse.user.id`) are stored as well. Content attributes are dropped even when named in an agent setting: * `gen_ai.input.messages`, `gen_ai.output.messages` * `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result` * OpenInference `input.value`, `output.value`, `llm.input_messages`, `llm.output_messages`, `llm.prompts`, `llm.completions` * Langfuse observation input and output Attribute values are truncated to 256 characters. ## Tool arguments Retry and loop detection compare tool arguments without keeping them. `args_hash` is an HMAC-SHA256 of the arguments, of their canonical JSON when they parse as JSON and of the raw string otherwise, under a key derived from `APGAR_SECRET_KEY`. Without the key, a hash cannot be checked against guessed arguments. Hashes are only compared within one turn. ## Secrets * **Connection secrets** (Langfuse keys, Slack webhook URLs, webhook secrets) are encrypted with AES-256-GCM before they are written, under a key derived from `APGAR_SECRET_KEY` with HKDF. The API returns Langfuse secret keys and webhook secrets masked. Slack webhook URLs are returned in full to signed in users. * **Ingest tokens** are shown once. Apgar stores a SHA-256 hash and a short prefix to identify them. * **The admin token** is read from the environment and never stored. ## Keys `APGAR_SECRET_KEY` is the root secret. Losing it makes stored connection secrets unreadable. To rotate it with Docker Compose: 1. In `.env`, set `APGAR_SECRET_KEY_PREVIOUS` to the current key and `APGAR_SECRET_KEY` to a new one. 2. Run `docker compose up -d`, which restarts the API and the worker. The worker decrypts every connection with the previous key and encrypts it again with the new one. 3. Remove `APGAR_SECRET_KEY_PREVIOUS` from `.env` and run `docker compose up -d` again. With the Helm chart and `existingSecret`, pass the chart version you run now, from `helm list -n apgar`, so that the rotation does not also upgrade Apgar: 1. Copy the current key into a second Secret, and put a new key into the chart's Secret: ```bash theme={null} kubectl -n apgar create secret generic apgar-previous-key \ --from-literal=APGAR_SECRET_KEY_PREVIOUS="$(kubectl -n apgar get secret apgar -o jsonpath='{.data.APGAR_SECRET_KEY}' | base64 -d)" kubectl -n apgar patch secret apgar --type merge \ -p "{\"stringData\":{\"APGAR_SECRET_KEY\":\"$(openssl rand -hex 32)\"}}" ``` 2. Upgrade with the second Secret as an extra variable source. This restarts the API and the worker, and the worker logs `re-encrypted N connections`: ```bash theme={null} helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar \ --reset-then-reuse-values --set 'envFrom[0].secretRef.name=apgar-previous-key' ``` 3. Remove the previous key: ```bash theme={null} helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar \ --reset-then-reuse-values --set-json 'envFrom=[]' kubectl -n apgar delete secret apgar-previous-key ``` If you already set `envFrom`, add the Secret to your list and remove it from there instead. Without `existingSecret`, the chart's Secret is `apgar-secrets`: read the current key from it in step 1, and set the new key with `--set secret.secretKey=...` in step 2 instead of patching the Secret. Rotation signs out every UI session. Tool argument hashes written under the old key do not match new ones, which matters only for a turn that is in flight during the restart. ## Outbound connections The worker connects only to what you configured: * Langfuse, at the host of each Langfuse connection. * Slack, at the webhook URL of each Slack connection. * The URL of each webhook connection. * PostHog, at the telemetry endpoint, once a day, unless telemetry is off. Ingest makes no outbound calls. The API calls out only to test a connection and, when `APGAR_WORKER_METRICS_URL` is set, to read the worker metrics for its own `/metrics` endpoint. ## Deletion * **Delete agent** hides the agent and revokes its ingest tokens at once. The retention job, which runs at worker startup and then daily, deletes its spans, turns, tool calls, releases, evaluations, and notifications. * `apgar demo purge` deletes the rows of the demo agents the same way and hides the agents. * Retention deletes trace data after `trace_data_days`, and evaluations and notifications after `verdict_history_days`. Release rows and release events are kept. See [Retention](/self-hosting/configuration#retention). # Quickstart Source: https://docs.useapgar.dev/quickstart Run Apgar with Docker Compose, explore the demo, and add your first agent. This guide runs the full stack on one machine: ClickHouse, the API, the worker, and the UI. > [!TIP] > > Setting up Apgar with a coding agent? Give it the Apgar skill and the docs MCP server first. See [Use with AI agents](/ai-agents). ## Requirements * Docker with Compose v2 * `curl` and `openssl`, present on macOS and most Linux distributions To run Apgar on Kubernetes instead, see [Kubernetes](/self-hosting/kubernetes). ## Run Apgar **1. Download the Compose file** ```bash theme={null} mkdir apgar && cd apgar curl -fsSLO https://github.com/oimo-ai/apgar/releases/latest/download/docker-compose.yml ``` **2. Create the environment file** Generate the three secrets into `.env`: ```bash theme={null} cat > .env <` attribute. See [Langfuse](/send-traces/langfuse#release-environment-and-session). ## Environment `apgar.environment`, `deployment.environment.name`, `deployment.environment`, `langfuse.environment`. Read on the root span, then on the resource. Missing means `default`. An agent with an environment counts only turns from that environment. Leave the agent environment empty to accept every environment. Do not send production and staging traffic to one agent without an environment: both would mix in the same comparison, and a staging release would end the monitoring of the production release. ## Session `apgar.session.id`, `gen_ai.conversation.id`, `session.id`, `langfuse.session.id`, on the root span. Session ids change per conversation, so they are not read from the resource. When none is present, the trace id is the session and the session is marked inferred. A turn is then its own session. See [Sessions](/concepts/sessions). ## Surface `apgar.surface` on the root span. Otherwise the agent's **Surface attribute** (`surface_mapping.from`) decides: | Value | Reads | | - | - | | An attribute name | That attribute on the root span. Resource attributes are not consulted | | `metadata.` | Langfuse trace metadata `` | | `tag:` | The rest of the first Langfuse tag that starts with `` | The surface is truncated to 64 characters. Use it for the channel a turn came from, such as `slack`, `web`, or `api`. ## Prompt version `langfuse.prompt.name` with `langfuse.prompt.version` as `name@version`, then `apgar.prompt.version`, then `gen_ai.prompt.name`. Read on the root span, then on the first chat span. ## Models On chat spans. Request model: `gen_ai.request.model`, `llm.request.model_name`, `llm.model_name`, `langfuse.observation.model.name`. Response model: `gen_ai.response.model`, `llm.response.model_name`, `llm.model_name`, `langfuse.observation.model.name`. The response model of the last chat span is the primary model of the turn. ## Tools On tool spans. Tool name: `gen_ai.tool.name`, `tool.name`, else the span name. Tool arguments: `gen_ai.tool.call.arguments`, `input.value`, `langfuse.observation.input`. Apgar stores an HMAC of the arguments, never the arguments: of their canonical JSON when they are an object, a list, or a JSON string, else of the raw string. A tool call is a retry when an earlier call in the same turn had the same tool and arguments and failed. A turn has a loop when three consecutive calls have the same tool and arguments. A call without arguments has an unknown retry status; a turn is marked loop as soon as three hashed calls match, and has an unknown loop status only when no loop was found and some call lacks arguments. ## Errors A span is an error when its OTLP status code is `ERROR` or its Langfuse level is `ERROR`. The error type comes from `error.type`, `exception.type`, the OTLP status message, or the Langfuse status message, truncated to 128 characters. ## Tokens Input: `gen_ai.usage.input_tokens`, `llm.token_count.prompt`, Langfuse usage details `input`. Output: `gen_ai.usage.output_tokens`, `llm.token_count.completion`, Langfuse usage details `output`. A turn sums its chat spans. ## Cost `apgar.cost_usd`, `gen_ai.usage.cost`, `llm.cost.total`, Langfuse cost details `total` or the observation `totalCost`. US dollars. Apgar does not estimate cost from token counts. The GenAI semantic conventions define no cost attribute, so most OTLP instrumentations need the application or a span processor to set `apgar.cost_usd` on each chat span. Langfuse fills cost itself when it knows the model price. The cost of a turn is the sum over its chat spans. If one chat span has token counts but no cost, the turn has no cost, because a partial sum would read as a lower cost. ## Content Output content is read from `gen_ai.output.messages`, OpenInference `output.value`, and the Langfuse observation output, on the root span or else on the last chat span. When the value is a message list, the last `assistant` message counts; a plain string counts whole. Apgar keeps the character count and whether the text parses as JSON, then drops the content. Input and output content attributes are never stored. OTLP logs and span events are not read. An instrumentation that records content as events, or not at all, leaves the character fallback of **Output length** and **Format failure rate** unavailable. ## Late spans A span that started more than `APGAR_LATE_SPAN_DAYS` (default 7) days before it arrived is marked late. Late turns are stored and excluded from evaluations. # Langfuse Source: https://docs.useapgar.dev/send-traces/langfuse Pull traces from a Langfuse project. When your agent already reports to Langfuse, Apgar can read the traces from there. The worker pulls observations from the Langfuse public API, so nothing changes in the agent. ## Supported servers The connector works with Langfuse Cloud and with self hosted Langfuse v4. ## Add a connection In **Settings, Connections**, add a Langfuse connection: | Field | Meaning | | - | - | | `host` | Base URL of the Langfuse server, for example `https://cloud.langfuse.com` | | `public_key` | Project public key | | `secret_key` | Project secret key. Stored encrypted, shown masked afterwards | | `fetch_content` | Request observation input and output. Default `true` | Apgar authenticates with HTTP Basic (`public_key:secret_key`) and reads the project id from `GET /api/public/projects`, taking the first project the keys can see. Use **Test** to check the keys. A connection that an agent uses cannot be deleted until that agent is changed or deleted. `fetch_content` lets Apgar read observation input and output. The output is reduced to its length and JSON validity, tool inputs to a keyed hash of the arguments, and neither is stored. Set it to `false` if you do not want content to leave Langfuse. **Output length** still works from token counts, but **Format failure rate**, **Retry rate**, and **Loop rate** are then unavailable. ## Add an agent Choose **Add agent** with the Langfuse source, pick the connection, and describe which traces belong to the agent: | Filter | Match | | - | - | | Trace name | The trace name equals this value | | Tags | The trace has all of these tags | | Metadata | The trace metadata has each of these key and value pairs | A filter needs at least a trace name, a tag, or a metadata pair. One project can feed several agents. When two filters match the same trace, the agent created first owns it, and `apgar_langfuse_filter_overlap_total` counts the overlap. > [!WARNING] > > The Langfuse Python SDK v4 and JS SDK v5 export only their own spans, spans that carry a `gen_ai.*` attribute, and spans from known LLM instrumentation scopes. Tool and agent spans from other instrumentation never reach Langfuse, so Apgar cannot see their errors, retries, and loops. Pass a permissive `should_export_span` (`shouldExportSpan` in JS) to the SDK if your tool spans are filtered out. ## Release, environment, and session The connector maps Langfuse fields onto the same attributes the OTLP path uses: | Langfuse | Apgar | | - | - | | Trace release | Release identifier | | Observation environment | Environment | | Session id | Session | | Prompt name and version | Prompt version, as `name@version` | | Observation type | Span kind: `agent`, `generation`, `tool`, `retriever`, and `chain` or `span` at the root | | Observation level `ERROR` | Error status. The status message becomes the error type | | Usage details `input`, `output` | Token counts | | Cost details `total`, else the observation `totalCost` | Cost in US dollars | | Trace metadata, trace tags, user id | Stored as span attributes. See [Data handling](/privacy/data-handling) | Set the release on every trace, for example from your deploy pipeline. A trace without a release is stored as `untagged` and is never evaluated. A prompt version change inside one release shows up as a **Prompt version** [slice](/concepts/metrics#slices); it does not open a new release by itself. Set the agent's **Release attribute** to `langfuse.release` to read the trace release, or to `langfuse.trace.metadata.` to read a metadata key, for example `langfuse.trace.metadata.version` for the key `version`. Apgar reads Langfuse traces only through their Langfuse fields, so `apgar.release`, `gen_ai.agent.version`, or `service.version` leave every turn `untagged`. The agent's **Environment** works as it does for OTLP: only turns whose Langfuse environment equals it create releases and count in comparisons. **Add agent** fills in `production`. Set it to the environment your traces carry, `default` when the SDK sets none, or clear it to accept every environment. To read the surface from Langfuse, set **Surface attribute** to `metadata.` for a metadata key, or `tag:` to use the rest of the first tag that starts with the prefix. See [Attribute reference](/send-traces/attributes). ## Sync cycle The worker runs one sync cycle per connection every 60 seconds (`APGAR_LANGFUSE_SYNC_INTERVAL_SECONDS`): 1. The window runs from the watermark minus a 30 minute lookback to 2 minutes before now. The first successful cycle of a connection starts 24 hours back. It runs within a minute of saving the connection, even before an agent uses it, so an agent added later sees traces from about 30 minutes before it was added on. 2. Phase one lists the root observations in the window that match each agent filter. 3. Phase two loads the remaining observations of those traces in batches of 50 trace ids, over a window widened by one hour on each side so that late children are found. 4. The watermark moves forward only after the whole window has been read. A request that fails with 429, a 5xx status, or a network error is retried up to 8 times, waiting `min(60, 2^attempt)` seconds plus up to one second of jitter. A numeric `Retry-After` header replaces the computed wait. After the eighth failure the cycle fails and the watermark stays put, so the next cycle re-reads the same window. A failing connection does not hold back other connections. The sync status is shown on the agent page. The lookback covers observations that become visible late. Langfuse delays data from exporters that do not send `x-langfuse-ingestion-version: 4`, and from SDKs older than Python 4.7.0 or JS 5.4.0, by up to 15 minutes. Self hosted v4 can set `LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct` to remove that delay. ## Trace ids A Langfuse trace id of 32 hex characters is kept, lowercased. Any other id is replaced with the first 32 hex characters of its SHA-256, and the original stays on the span attribute `langfuse.trace.id`. Observation ids follow the same rule with 16 characters. Apgar stores no links into Langfuse; search the trace id there to open a trace. ## Export from the Langfuse SDKs instead The Langfuse Python SDK v4 and JS SDK v5 are built on OpenTelemetry, so the same tracer provider can also export to Apgar directly while it reports to Langfuse. Add an OTLP/HTTP exporter for Apgar as described in [OpenTelemetry](/send-traces/opentelemetry). The Langfuse export filter applies to the Langfuse span processor only; a second exporter on the same provider receives every span. # OpenTelemetry Source: https://docs.useapgar.dev/send-traces/opentelemetry Send agent traces to Apgar over OTLP/HTTP. Apgar accepts OpenTelemetry traces at `POST /v1/traces`. Any SDK or collector that exports OTLP over HTTP can send to it. Instrumentations that follow the OpenTelemetry GenAI semantic conventions or OpenInference work without changes. ## Create an ingest token An ingest token has the form `apg_ing_` followed by 32 characters. Apgar shows it once and stores only its hash. There are two ways to get one: * **Add agent** with the OTLP source creates a token named `AGENT (auto)` and bound to that agent. Every trace sent with it belongs to that agent, whatever its attributes say. * **Settings, Ingest tokens** creates a token that you can bind to an agent or leave unbound. With an unbound token, Apgar resolves the agent from the [agent name attributes](/send-traces/attributes#agent-name), and the first trace with a new name creates the agent. Turn off **Create agents automatically** on the token to drop traces of unknown agents instead. An agent created this way has no environment and no **Release attribute**, so it reads the release from `apgar.release`, `gen_ai.agent.version`, or `langfuse.release`; set both in **Agent settings**. One unbound token can serve several agents. Deleting a token revokes it. The API caches token lookups for 60 seconds, so a revoked token can still be accepted for up to a minute. ## Configure the exporter Set these variables in the environment of the agent service. `ORIGIN` is the address of the Apgar UI, the value of `APGAR_PUBLIC_URL`. The UI forwards `/v1` to the API. ```bash theme={null} OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=ORIGIN/v1/traces OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer apg_ing_... OTEL_RESOURCE_ATTRIBUTES=gen_ai.agent.name=AGENT,deployment.environment.name=ENVIRONMENT,apgar.release=RELEASE ``` The snippet that **Add agent** shows contains the first three lines plus the agent name and environment. Add the release yourself: `RELEASE` is the identifier of the running build, a version, a tag, or a commit hash, and comes from your deploy pipeline. Put it in the attribute you chose as **Release attribute** on the agent; `apgar.release` is the usual choice. Without it every turn is stored as `untagged` and is never evaluated. See [Release](/send-traces/attributes#release). ## What a trace should look like Apgar treats one trace as one turn of the agent. It works best when a trace has: * A root span for the turn, such as an `invoke_agent` or `invoke_workflow` span, carrying the conversation id. See [Sessions](/concepts/sessions). * One chat span per LLM call, with the model, token counts, and ideally a cost. * One tool span per tool call, with the tool name and its arguments. Span kinds are read from `gen_ai.operation.name`, `openinference.span.kind`, or `langfuse.observation.type`. Set `apgar.kind` to override the kind of a span. See [Span kinds](/send-traces/attributes#span-kinds). > [!NOTE] > > Tool arguments are needed to detect retries and loops. Apgar never stores them: it keeps only a keyed hash. Content attributes are reduced to a length and a JSON validity flag and then dropped. ### Example with the Python SDK The OTLP exporter reads the variables above. The application opens one root span per turn and puts the conversation id on it; the chat and tool spans come from the instrumentation of your LLM client or agent framework. ```bash theme={null} pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http ``` ```python theme={null} from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) tracer = trace.get_tracer("my-agent") def handle_turn(conversation_id: str, message: str) -> str: with tracer.start_as_current_span( "invoke_agent my-agent", attributes={ "gen_ai.operation.name": "invoke_agent", "gen_ai.conversation.id": conversation_id, }, ): return run_agent(message) # instrumented LLM and tool calls happen here ``` ## Endpoint | Property | Value | | - | - | | Path | `POST /v1/traces` | | Authentication | `Authorization: Bearer apg_ing_...` | | Body | OTLP protobuf (`application/x-protobuf`) or OTLP/JSON (`application/json`), optionally with `Content-Encoding: gzip` | | Size limit | 32 MiB by default (`APGAR_MAX_BODY_BYTES`), enforced on both the received body and the decompressed body | | Status | Meaning | | - | - | | `200` | The request was stored. Spans with a missing, all-zero, or wrong-length trace or span id, a parent span id of the wrong length, or no valid start time are dropped silently and counted in `apgar_ingest_spans_dropped_total`. An all-zero parent span id means no parent, so that span is a root. A request with no spans is also `200` | | `400` | The body could not be decoded, including a broken gzip stream | | `401` | Missing, unknown, or revoked ingest token | | `413` | The received or decompressed body is larger than the limit. The received size is checked before the token, the decompressed size after it | | `415` | Unsupported content type | | `503` | Storage is unavailable, also for the token lookup. Retry after the `Retry-After` header | Apgar accepts OTLP over HTTP only. An SDK that exports OTLP/gRPC switches to HTTP with `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf`. ## Send through an OpenTelemetry Collector When traces already pass through a collector, add an exporter for Apgar next to the existing one. The application keeps exporting to the collector, and the collector sends a copy to Apgar. Each exporter has its own queue and retries, so an unavailable Apgar does not hold back your existing backend. ```yaml collector.yaml theme={null} receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } http: { endpoint: 0.0.0.0:4318 } processors: batch: {} exporters: otlp_grpc/existing: endpoint: backend.example.com:4317 otlp_http/apgar: endpoint: https://apgar.example.com headers: { Authorization: "Bearer ${env:APGAR_INGEST_TOKEN}" } service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp_grpc/existing, otlp_http/apgar] ``` In an existing collector only `otlp_http/apgar` and its entry in the traces pipeline are new. Set `endpoint` to `APGAR_PUBLIC_URL`; the exporter appends `/v1/traces`. Set `APGAR_INGEST_TOKEN` in the environment of the collector. Collector releases before the component rename know the exporters as `otlphttp` and `otlp`; current releases accept both spellings. Validate the configuration before you deploy it: ```bash theme={null} docker run --rm -e APGAR_INGEST_TOKEN=dummy -v "$PWD/collector.yaml:/c.yaml" otel/opentelemetry-collector-contrib:0.163.1 validate --config /c.yaml ``` > [!WARNING] > > Put the Apgar exporter in a pipeline without sampling. A tail sampling policy that keeps only errors changes the error, retry, and loop rates that Apgar compares, and sampling by trace removes turns from sessions. ## Check that traces arrive Open the agent page. A trace becomes a turn about a minute after its last span arrives, and its release then appears in the release list, whose turn counts update every 5 minutes. When the parent of your top span belongs to a service that does not send to Apgar, a trace waits 10 minutes from its first span instead. The page warns when more than 5 % of turns have no release, which means the release attribute is missing on some traffic or the agent reads a different attribute. When nothing arrives, see [Troubleshooting](/troubleshooting). # Troubleshooting Source: https://docs.useapgar.dev/troubleshooting What to check when turns, releases, verdicts, or messages do not show up. Each section starts with what you see and lists the causes in the order they are worth checking. The agent page, **Settings, Notifications**, and the [Prometheus metrics](/self-hosting/monitoring) are the places to look. ## No turns on the agent page * **The exporter does not reach Apgar.** Check the exporter logs for the status of `POST /v1/traces`, and `apgar_ingest_requests_total` by status. `401` is a missing, mistyped, or revoked token. `415` means the exporter sends gRPC or an unexpected content type; set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf`. * **Spans arrived, but no turn is assembled yet.** A trace becomes a turn 60 seconds after its last span once its span without a parent has arrived, and otherwise 10 minutes after its first span arrived. When the parent of your top span belongs to a service that does not send to Apgar, every turn waits the 10 minutes. `apgar_assembly_lag_seconds` shows how far assembly is behind. * **The trace belongs to another agent, or to none.** A token bound to an agent sends every trace to that agent, whatever the trace names. `apgar_turns_dropped_total` with reason `agent_not_found` means the bound agent was deleted, or an unbound token with **Create agents automatically** off received a trace that names an unknown agent; with reason `no_agent`, the trace carries no agent name and the token is unbound. See [Agent name](/send-traces/attributes#agent-name). * **Spans were dropped at ingest.** `apgar_ingest_spans_dropped_total` with reason `invalid_ids` or `invalid_timestamps` means the SDK sent spans without a valid trace id and span id (all-zero ids are invalid), with a malformed parent span id, or without a start time. * **The spans are older than 7 days** (`APGAR_LATE_SPAN_DAYS`). They are stored as late, never create releases, and are never evaluated. A backfill of more recent traces creates releases and sends messages like live traffic. * **Langfuse: the sync has not run.** The worker syncs each connection every minute. Only the first successful cycle of a connection reads the last 24 hours; an agent added to a connection that has already synced sees traces from about 30 minutes before it was added on. The agent page shows the sync status, and `apgar_langfuse_requests_total` by status shows whether Langfuse answers. A `404` on every request means a self hosted v3 server. ## Turns arrive, but no release appears The agent page says "No releases yet" where the release list would be. Without any turn in the last 24 hours it would show the **Getting started** steps instead. * **The environment does not match.** Only turns whose environment equals the agent's environment create releases and count in comparisons. The environment comes from `apgar.environment`, `deployment.environment.name`, `deployment.environment`, or `langfuse.environment`, and is `default` when none is set. **Add agent** fills in `production`. Compare the agent environment with the environment on the spans, or clear it to accept every environment. The agent page shows no warning for this. * **The turns are untagged.** See the next section. ## Turns are untagged * The release attribute is missing on the spans. Add it to `OTEL_RESOURCE_ATTRIBUTES`, or set it on the turn's root span, the outermost `invoke_agent` or `invoke_workflow` span. A release on a span above that root is not read. * The release is set only on the root span of a long run. A turn whose root span arrives more than 10 minutes after its first span is assembled without the root and keeps the release it had then. Set the release on the resource instead. * The agent reads a different attribute. **Agent settings, Release attribute** names the only attribute that is read; an agent without one reads `apgar.release`, `gen_ai.agent.version`, then `langfuse.release`. Spans stored before a custom attribute was chosen do not carry it. * Langfuse: the trace has no release. Set the release on the trace in the SDK. * Langfuse: the agent reads `apgar.release`, `gen_ai.agent.version`, or `service.version`. Langfuse traces do not carry them; use `langfuse.release` or `langfuse.trace.metadata.`. ## The release stays in Collecting * The release is below the early floor of 20 turns and 5 sessions, or below the minimum sample (**Minimum turns** and **Minimum sessions**) with no failing guardrail. The release page shows the counts. * The baseline has fewer turns or sessions than the minimum sample. The previous release may have been short lived, or its traffic is older than the 7 day baseline window. * Most sessions are mixed: they started on the previous release. Mixed turns leave both sides. See [Mixed sessions](/concepts/sessions#mixed-sessions). * No turns arrived in the last 24 hours, so the release is not checked again until its window ends. ## The release stays in Not enough data * The intervals are wider than the tolerances. The release page states the smallest change the sample can detect. More traffic narrows the intervals; so does a larger tolerance. * A guardrail has a previous value of 0 and a relative tolerance, so its change is undefined. Make that metric informational or turn it off. See [When a metric is unavailable](/concepts/metrics#when-a-metric-is-unavailable). * The agent has no applicable guardrail, for example an agent without tools whose remaining guardrails lack coverage. Such a release never reaches **Healthy**. ## A metric is unavailable The release page says what is missing. See [When a metric is unavailable](/concepts/metrics#when-a-metric-is-unavailable). The usual fixes: * **Retry rate** or **Loop rate**: tool spans lack arguments. Set `gen_ai.tool.call.arguments` on tool spans. * **Cost per turn**: chat spans lack a cost. Set `apgar.cost_usd` on each chat span, or let Langfuse price the model. * **Format failure rate**, or **Output length** without token counts: output content is not captured, or the Langfuse connection has `fetch_content` off. * **Turns per session**: fewer than 10 closed sessions, or most sessions are inferred because no session id is sent. See [Sessions](/concepts/sessions). ## No Slack message * The agent has no Slack connection. **Healthy** also needs **Send the healthy message** on in **Agent settings, Verdict rules**, and a release replaced by a newer one sends no **Unconfirmed** message. * Delivery failed. **Settings, Notifications** lists every delivery with its last error. Slack must answer `200`. * Links point at `localhost`: set `APGAR_PUBLIC_URL`. ## Webhook signatures do not verify * Compute the signature over the raw body bytes. Serializing the JSON again changes them. * Sign `X-Apgar-Timestamp`, a period, and the body. `sent_at` inside the body is the queue time and differs from the header on retries. * Use the secret stored on the connection. The connection test signs with the same one, so it is the quickest check. ## A process exits at startup | Exit code | Cause | | - | - | | `1` | ClickHouse is unreachable or its certificate does not verify, or a second worker saw a live heartbeat | | `2` | A required variable is missing or too short. The message names the field | | `3` | The database schema does not match this version. Run `apgar migrate` | On Kubernetes, the worker applies the migrations in an init container. See [When the worker does not start](/self-hosting/kubernetes#when-the-worker-does-not-start).