Skip to main content
A release is one version of an agent, identified by the release attribute on its traces. Apgar compares every release, the candidate, with an earlier release of the same agent, the baseline.

​
Release detection

A release is created by its first turn. The turn must:
  • carry a release identifier, so it is not untagged,
  • not be late, and
  • come from the agent’s environment, when the agent has one.
The release is first seen at the start of that turn. If an earlier turn of the same release arrives later, the first seen time moves back to it. Apgar then queues an evaluation and sends a New release message. The first release of an agent has no baseline and shows as First release while its window is open, then as Unconfirmed. It becomes the baseline of the next release.

​
Baseline selection

By default the baseline is chosen automatically. The candidates are the earlier releases of the agent that did not fail, not awaiting a fail confirmation, and not ended with a failing guardrail. Among them, the most recently active release wins, by the time of its last turn. When every earlier release failed, the most recently active one is used anyway and the baseline is marked unverified. A baseline is verified when it is Healthy, also while it is still monitored, or when it ended as Unconfirmed because a newer release replaced it or the monitoring window ran out, with no failing guardrail. Any other baseline is unverified, including one that is still monitored without a healthy result and one that someone stopped.

​
Baseline window

The baseline is the baseline release’s own traffic in the 7 days before the candidate was first seen. Both sides exclude:
  • untagged turns,
  • late turns,
  • turns from another environment, when the agent has one, and
  • mixed turns, the turns of a session that come after it already ran on another release, unless the agent allows mixed sessions. See Sessions.
While the candidate has been live for less than 24 hours, baseline turns are restricted to the UTC hours of day that the candidate window has covered. A release that ran overnight is not compared with a baseline that ran mostly during the day. From 24 hours on, every hour is covered and no filter applies. The filter is also skipped when less than 24 hours separate the first and last turns of the baseline release in its window, because most hours would have no baseline turns.

​
Change the baseline

Choose another release in the Baseline selector on the release page. Only releases first seen before the candidate are allowed, because a later release has no traffic in the baseline window. The baseline can change only while monitoring runs. A manual baseline is compared over the same window. Changing it queues a check, and confirmation starts over.

​
Expected changes

Some changes are intended, such as shorter answers after a prompt rewrite. Select Expected changes on the release page and pick the metrics. Those metrics show as expected in the metric table and no longer decide the verdict of that release. Saving queues a check, and confirmation starts over. Changing the baseline or recording expected changes after a Regression lifts it, and the release is evaluated again with the new settings. While the release is monitored, confirmation starts over. Once monitoring has ended, the baseline can no longer change, but expected changes can still be recorded, and the next check decides right away. There is no acknowledge or override action: record an intended regression as an expected change, and fix a wrong comparison by choosing another baseline while the release is monitored. Expected changes can no longer be recorded once the trace data of the release has left retention.

​
End of monitoring

Monitoring of a release ends at the earliest of: The monitoring window starts when Apgar first sees the release. Set its length with Monitor each release for up to in Agent settings, Verdict rules: 72 hours by default, from 1 to 168. One turn of a newer release is enough. A newer release is one the agent has not seen before. Rolling back to a release Apgar already knows does not end monitoring, but rolling back under a new release name does. When the agent has an environment, only turns from that environment create releases, so traffic in another environment never ends monitoring. An agent without an environment registers releases from every environment, and they replace each other. When monitoring ends, one last check runs and decides on its own, without waiting for a second confirmation. A failing guardrail ends the release as Regression, and all guardrails passing end it as Healthy, with the same sample gates as during monitoring: the baseline needs the full minimum sample, and Healthy also needs it on the release. Anything else ends the release as Unconfirmed, except that a Regression holds and a release that is already Healthy stays Healthy unless the last check fails. When monitoring is stopped or the window runs out, the last check counts turns up to that moment. When a newer release takes over, the last check runs within 5 minutes of the newer release’s first turn, so a few late turns of the old release are still counted. After that, only recording expected changes can change the result.