> ## Documentation Index > Fetch the complete documentation index at: https://docs.useapgar.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev > Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md # Verdicts > How an evaluation becomes a verdict, and what each label means. Every evaluation compares the candidate release with its baseline on each applicable metric. A verdict is the state of the release after its latest evaluation. ## How a metric is judged For each metric, Apgar computes the change from baseline to candidate and an interval around it. The interval comes from a cluster bootstrap: sessions are resampled in both arms with Poisson weights, 10000 times by default. Resampling whole sessions keeps one long conversation from counting as many independent turns. A guardrail metric then gets a status, shown in the **Status** column of the metric table on the release page: | Status | Rule | | - | - | | **regressed** | The whole interval lies past the tolerance on the degradation side. Until the regression is confirmed, the status is **possible** | | **within tolerance** | The whole interval stays within the tolerance. For a metric checked in both directions, within the tolerance on both sides | | **not enough data** | The interval crosses the tolerance | | **expected** | The metric is listed as an expected change for this release | | Listed as unavailable under its group | The metric is not applicable to this agent or sample. See [When a metric is unavailable](https://docs.useapgar.dev/concepts/metrics.md#when-a-metric-is-unavailable) | Informational metrics show **informational**. The significance level per guardrail is 0.05 / N, where N is the number of applicable guardrails (Bonferroni). Informational metrics are computed and shown, and never decide the verdict. ## States The release page and the messages show the label. [Webhooks](https://docs.useapgar.dev/notifications/webhooks.md) also send the state and the reason behind it, as `verdict` and `reason`. | Label | Meaning | Webhook `verdict` | Webhook `reason` | | - | - | - | - | | **First release** | No earlier release to compare with | `PENDING` | `no_baseline` | | **Collecting** | The release is below the minimum sample and no guardrail fails | `PENDING` | `insufficient_sample` | | **Collecting** | The baseline has fewer turns or sessions than the minimum sample | `PENDING` | `insufficient_baseline` | | **Not enough data** | At least one interval is wider than its tolerance | `INCONCLUSIVE` | `insufficient_power` | | **Possible regression**, or **Possible regression, early** | A guardrail failed; the next check confirms or clears it | `INCONCLUSIVE` | `fail_candidate` | | **Looks healthy** | Every guardrail passed; the next check confirms it | `INCONCLUSIVE` | `pass_candidate` | | **Regression** | A guardrail failure was confirmed | `FAIL` | `confirmed` | | **Healthy** | Every guardrail passed, confirmed | `PASS` | `confirmed` | | **Unconfirmed** | Monitoring ended without a decision: a newer release replaced it, the monitoring window ran out, or someone stopped it | `INCONCLUSIVE` | `superseded`, `window_elapsed`, `stopped` | Only **Healthy** means every guardrail was confirmed within tolerance. **Looks healthy**, **Not enough data**, and **Unconfirmed** mean no regression was found, but the sample could not yet rule out a change as large as the tolerance. ## Minimum sample By default each side needs 200 turns (**Minimum turns**) and 30 sessions (**Minimum sessions**). The two possible claims have different gates, because a small sample already widens the interval: 1. Metrics are computed once the release has 20 turns and 5 sessions (or the minimum sample, when that is lower) and the baseline has the full minimum sample. Below that the release stays **Collecting**. 2. A failing guardrail can move the release to **Possible regression** from that early floor, and then to **Regression**. Such a result is marked early, and the message title reads **Regression, early**. 3. **Looks healthy**, **Healthy**, and **Not enough data** need the full minimum sample. The first turns after a deploy are not always representative: cold caches, internal testers, and mostly first turns of new sessions. For small changes this is a bias, but the interval on a small sample rejects small changes anyway. The early path exists for large jumps, where the bias does not matter. ## Confirmation A result becomes a verdict after it repeats on consecutive checks: **FAIL confirmations** and **PASS confirmations** both default to 2, and the checks must span at least 10 minutes from the first to the last. Confirmation counts the newest run of consecutive checks with the same outcome against the same baseline release. The run starts over when Apgar picks another baseline, or when someone changes the baseline or the expected changes. Set a confirmation count to 1 to decide on the first check. The 10 minute gap is fixed. The last check, when monitoring ends, decides without confirmation. ## After a decision A **Regression** holds. A later check that would pass does not clear it. Only changing the baseline or recording expected changes after the decision lifts it. While the release is monitored, confirmation then starts over. Once monitoring has ended, the baseline can no longer change, but expected changes can still be recorded, and the next check decides right away. A **Healthy** release stays **Healthy** while later checks show only **Not enough data** or **Looks healthy**. A later failing check can still move it to **Possible regression** and then to **Regression**, which sends a second message. ## Schedule The worker checks a release every 5 minutes for the first 2 hours after it was first seen (`APGAR_VERDICT_EARLY_INTERVAL_SECONDS`) and every 15 minutes after that (`APGAR_VERDICT_INTERVAL_SECONDS`). The release page shows the last and the next check. A release with no turns in the last 24 hours is not checked again until its monitoring window ends. Once monitoring has ended, a release is checked again only when someone records expected changes. ## Attribution Every applicable metric is also computed per [slice](https://docs.useapgar.dev/concepts/metrics.md#slices). Slices are tested with a Benjamini-Hochberg threshold over the same resamples. They never decide the verdict; for a failing metric they say where the change is: * concentrated in the failing slices of a dimension, when that dimension also has a slice with enough turns that is not flagged, * uniform across a dimension, when every slice of it with enough turns fails, * "No single slice stands out.", when no slice is flagged, or * "Attribution pending (N turns).", when no slice has enough turns yet. Concentrated and uniform dimensions combine in one sentence, for example "Concentrated in web\_search; uniform across surfaces." Tool slices are named bare; the other dimensions are prefixed with `surface`, `depth`, `model`, or `prompt`. A failing metric reads as one sentence with its values, the allowed change, and the attribution: ```text theme={null} Tool error rate rose from 2.6% to 27.1% (allowed: up to +2 pp). Concentrated in web_search and fetch_url. ``` Several failing metrics are ordered by group (behavior, reliability, cost and latency), then by how far the interval lies past the tolerance, measured in interval half widths. ## Verdict settings Per agent, in **Agent settings, Verdict rules**: | Setting | Default | Meaning | | - | - | - | | **Minimum turns** | 200 | Turns the release needs for a pass, and the baseline needs for any verdict | | **Minimum sessions** | 30 | Sessions the release needs for a pass, and the baseline needs for any verdict | | **Minimum turns per slice** | 100 | Turns needed on each side of a slice | | **FAIL confirmations** | 2 | Failing checks in a row needed for **Regression** | | **PASS confirmations** | 2 | Passing checks in a row needed for **Healthy** | | **Monitor each release for up to** | 72 hours | Longest monitoring window, 1 to 168 hours | | **Send the healthy message** | On | Send a message when a release becomes healthy | | **Include mixed sessions** | Off | Count turns of sessions that span releases | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.