Skip to main content
Every evaluation compares the candidate release with its baseline on each applicable metric. A verdict is the state of the release after its latest evaluation.

​
How a metric is judged

For each metric, Apgar computes the change from baseline to candidate and an interval around it. The interval comes from a cluster bootstrap: sessions are resampled in both arms with Poisson weights, 10000 times by default. Resampling whole sessions keeps one long conversation from counting as many independent turns. A guardrail metric then gets a status, shown in the Status column of the metric table on the release page: Informational metrics show informational. The significance level per guardrail is 0.05 / N, where N is the number of applicable guardrails (Bonferroni). Informational metrics are computed and shown, and never decide the verdict.

​
States

The release page and the messages show the label. Webhooks also send the state and the reason behind it, as verdict and reason. Only Healthy means every guardrail was confirmed within tolerance. Looks healthy, Not enough data, and Unconfirmed mean no regression was found, but the sample could not yet rule out a change as large as the tolerance.

​
Minimum sample

By default each side needs 200 turns (Minimum turns) and 30 sessions (Minimum sessions). The two possible claims have different gates, because a small sample already widens the interval:
  1. Metrics are computed once the release has 20 turns and 5 sessions (or the minimum sample, when that is lower) and the baseline has the full minimum sample. Below that the release stays Collecting.
  2. A failing guardrail can move the release to Possible regression from that early floor, and then to Regression. Such a result is marked early, and the message title reads Regression, early.
  3. Looks healthy, Healthy, and Not enough data need the full minimum sample.
The first turns after a deploy are not always representative: cold caches, internal testers, and mostly first turns of new sessions. For small changes this is a bias, but the interval on a small sample rejects small changes anyway. The early path exists for large jumps, where the bias does not matter.

​
Confirmation

A result becomes a verdict after it repeats on consecutive checks: FAIL confirmations and PASS confirmations both default to 2, and the checks must span at least 10 minutes from the first to the last. Confirmation counts the newest run of consecutive checks with the same outcome against the same baseline release. The run starts over when Apgar picks another baseline, or when someone changes the baseline or the expected changes. Set a confirmation count to 1 to decide on the first check. The 10 minute gap is fixed. The last check, when monitoring ends, decides without confirmation.

​
After a decision

A Regression holds. A later check that would pass does not clear it. Only changing the baseline or recording expected changes after the decision lifts it. While the release is monitored, confirmation then starts over. Once monitoring has ended, the baseline can no longer change, but expected changes can still be recorded, and the next check decides right away. A Healthy release stays Healthy while later checks show only Not enough data or Looks healthy. A later failing check can still move it to Possible regression and then to Regression, which sends a second message.

​
Schedule

The worker checks a release every 5 minutes for the first 2 hours after it was first seen (APGAR_VERDICT_EARLY_INTERVAL_SECONDS) and every 15 minutes after that (APGAR_VERDICT_INTERVAL_SECONDS). The release page shows the last and the next check. A release with no turns in the last 24 hours is not checked again until its monitoring window ends. Once monitoring has ended, a release is checked again only when someone records expected changes.

​
Attribution

Every applicable metric is also computed per slice. Slices are tested with a Benjamini-Hochberg threshold over the same resamples. They never decide the verdict; for a failing metric they say where the change is:
  • concentrated in the failing slices of a dimension, when that dimension also has a slice with enough turns that is not flagged,
  • uniform across a dimension, when every slice of it with enough turns fails,
  • “No single slice stands out.”, when no slice is flagged, or
  • “Attribution pending (N turns).”, when no slice has enough turns yet.
Concentrated and uniform dimensions combine in one sentence, for example “Concentrated in web_search; uniform across surfaces.” Tool slices are named bare; the other dimensions are prefixed with surface, depth, model, or prompt. A failing metric reads as one sentence with its values, the allowed change, and the attribution:
Several failing metrics are ordered by group (behavior, reliability, cost and latency), then by how far the interval lies past the tolerance, measured in interval half widths.

​
Verdict settings

Per agent, in Agent settings, Verdict rules: