Metrics

Apgar compares each release with the previous one on eleven metrics, computed from turns and tool calls. Guardrail metrics decide the verdict. Informational metrics are shown on the release page and never decide it. Every metric is a guardrail by default except Turns per session, which is informational.

Catalog

The release page groups the metrics the same way.

Behavior

Metric	What it measures	Worse when	Default tolerance
Tool calls per turn	Average number of tool calls per turn. A drop usually means the agent stopped using its tools	Down	10 %
Turns without tools	Share of turns that finished without calling any tool	Up	5 pp
Retry rate	Share of tool calls that repeat a failed call with the same tool and arguments in the same turn	Up	2 pp
Loop rate	Share of turns with three or more consecutive identical tool calls	Up	1 pp
Output length	Median output tokens per turn	Down	15 %
Format failure rate	Share of turns whose final response is not valid JSON. Only for agents with output format `json`	Up	1 pp
Turns per session	Average number of turns in closed sessions. A session is closed when its last turn ended more than 30 minutes ago	Up or down	25 %

Reliability

Metric	What it measures	Worse when	Default tolerance
Tool error rate	Share of tool calls that ended with an error	Up	2 pp
LLM error rate	Share of turns where at least one LLM call failed	Up	1 pp

Cost and latency

Metric	What it measures	Worse when	Default tolerance
Cost per turn	Average cost per turn in US dollars, as reported on the chat spans	Up	20 %
Latency p95	95th percentile of turn latency, measured on the root span	Up	15 %

Output length counts tokens when at least 95 % of turns on both sides have token counts, and characters otherwise. The release page shows the unit. Rates and averages are taken over the whole release. Tool error rate is all failed tool calls divided by all tool calls, not an average of per turn rates. When several metrics fail, messages and the release page list them in the group order above, and inside a group by how far they are past their tolerance.

Tolerances

A tolerance is the largest change that still counts as no regression. Worse when says which direction counts; a change in the other direction never fails a release. Rates have a tolerance in percentage points (pp). A percentage point is the plain difference between two percentages: a tool error rate that goes from 2 % to 4 % changes by +2 pp, even though it doubled. With the default tolerance of 2 pp, that is the largest change that still counts as no regression. The other metrics have a tolerance in percent of the previous value. Latency that goes from 2.0 s to 2.4 s changes by +20 %. The Change column on the release page uses the same units. A guardrail fails only when Apgar is confident that the change is past the tolerance, not as soon as the measured change crosses it. See How a metric is judged.

Change the defaults

Per agent, in Agent settings, Metrics:

Setting	Meaning
Enabled	Turns the metric on or off for this agent. Format failure rate stays off until the output format in General is `json`
Tolerance	The allowed change, in pp for rates and in % for the other metrics
Role	guardrail decides the verdict, informational is only shown

Making a guardrail informational keeps it visible without letting it fail a release. To leave a metric out for one release only, use expected changes.

When a metric is unavailable

Some metrics need data that not every instrumentation records. When that data is missing, or the metric does not apply, the release page lists the metric as unavailable under its group, with the reason.

Metric	Unavailable when	What to do
Tool calls per turn, Turns without tools, Retry rate, Tool error rate	The previous release made no tool calls	Nothing. An agent without tools is judged on the other guardrails
Retry rate	Fewer than 90 % of tool calls on one side carry their arguments	Set `gen_ai.tool.call.arguments` on tool spans. See Tools
Loop rate	Fewer than 90 % of turns on one side carry arguments on every tool call	Same as Retry rate
Format failure rate	Fewer than 90 % of turns on one side have output content	Capture output content. See Content
Output length	Fewer than 95 % of turns on one side have token counts, and fewer than 95 % have output content	Record output token counts, or capture output content. See Tokens
Cost per turn	Fewer than 90 % of turns on one side have a cost	Set a cost on each chat span. See Cost
Turns per session	Fewer than 10 closed sessions on one side, or more than half of the turns have no session id	Send a session id. See Sessions
Any metric	It is turned off for this agent	Turn it on in Agent settings, Metrics
Any metric	It cannot be compared: one side has nothing to measure, the previous value is 0 for a metric with a % tolerance, or the values are too sparse	Wait for more traffic, or make the metric informational

An unavailable guardrail does not hold the release back; the verdict uses the other guardrails. The exception is a guardrail whose previous value is 0 or whose values are too sparse: it keeps the release in Not enough data until it can be compared or is made informational. An agent with no available guardrail never reaches Healthy.

Slices

A slice is the part of a release’s traffic that shares one value of a dimension: the turns that called the web_search tool, the turns that came from Slack, or the turns answered by one model. Apgar computes every available metric per slice, for the previous release and this one, to show where a change is concentrated. Select a metric on the release page to see its slices, one table per dimension:

Dimension	Values
Tool	Each tool the agent called. A turn that calls two tools counts in both. See Tools
Surface	Each surface, such as Slack or a web chat. See Surface
Session depth	`short` for turns 1 to 5 of a session, `medium` for turns 6 to 20, `long` after that. Turns without a session id have no depth
Response model	Each model that answered. See Models
Prompt version	Each prompt version. See Prompt version

Slices never decide the verdict. The whole release is judged; slices only say where the change is. A slice table counts the slices that regressed, and the message for a failing metric sums them up, for example “Concentrated in web_search.” See Attribution. A slice is compared only when both sides have at least Minimum turns per slice turns, 100 by default, or that many tool calls for Tool error rate and Retry rate. Values that are not compared are listed below the tables with the reason:

Reason	Meaning
Too few turns, or too few tool calls	One side is below Minimum turns per slice
Previous value is 0, so a relative change is undefined	The metric has a % tolerance and the previous value in this slice is 0
Too sparse for a stable comparison	The slice has too little data for a reliable comparison
A reason from When a metric is unavailable	The metric is unavailable inside this slice

Turns per session is sliced by the first turn of each session, except that a session counts in a tool slice when any of its turns called the tool. It has no session depth slice.

Get started

Send traces

Concepts

Notifications

Self hosting

Data and privacy

Catalog

Behavior

Reliability

Cost and latency

Tolerances

Change the defaults

When a metric is unavailable

Slices

​Catalog

​Behavior

​Reliability

​Cost and latency

​Tolerances

​Change the defaults

​When a metric is unavailable

​Slices

Catalog

Behavior

Reliability

Cost and latency

Tolerances

Change the defaults

When a metric is unavailable

Slices