> ## Documentation Index
> Fetch the complete documentation index at: https://docs.useapgar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev
> Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md

# Metrics

> What Apgar measures, how much change it allows, and where a change is concentrated.

Apgar compares each release with the previous one on eleven metrics, computed from turns and tool calls. Guardrail metrics decide the verdict. Informational metrics are shown on the release page and never decide it. Every metric is a guardrail by default except **Turns per session**, which is informational.

## Catalog

The release page groups the metrics the same way.

### Behavior

| Metric | What it measures | Worse when | Default tolerance |
| - | - | - | - |
| **Tool calls per turn** | Average number of tool calls per turn. A drop usually means the agent stopped using its tools | Down | 10 % |
| **Turns without tools** | Share of turns that finished without calling any tool | Up | 5 pp |
| **Retry rate** | Share of tool calls that repeat a failed call with the same tool and arguments in the same turn | Up | 2 pp |
| **Loop rate** | Share of turns with three or more consecutive identical tool calls | Up | 1 pp |
| **Output length** | Median output tokens per turn | Down | 15 % |
| **Format failure rate** | Share of turns whose final response is not valid JSON. Only for agents with output format `json` | Up | 1 pp |
| **Turns per session** | Average number of turns in closed sessions. A session is closed when its last turn ended more than 30 minutes ago | Up or down | 25 % |

### Reliability

| Metric | What it measures | Worse when | Default tolerance |
| - | - | - | - |
| **Tool error rate** | Share of tool calls that ended with an error | Up | 2 pp |
| **LLM error rate** | Share of turns where at least one LLM call failed | Up | 1 pp |

### Cost and latency

| Metric | What it measures | Worse when | Default tolerance |
| - | - | - | - |
| **Cost per turn** | Average cost per turn in US dollars, as reported on the chat spans | Up | 20 % |
| **Latency p95** | 95th percentile of turn latency, measured on the root span | Up | 15 % |

**Output length** counts tokens when at least 95 % of turns on both sides have token counts, and characters otherwise. The release page shows the unit.

Rates and averages are taken over the whole release. **Tool error rate** is all failed tool calls divided by all tool calls, not an average of per turn rates.

When several metrics fail, messages and the release page list them in the group order above, and inside a group by how far they are past their tolerance.

## Tolerances

A tolerance is the largest change that still counts as no regression. **Worse when** says which direction counts; a change in the other direction never fails a release.

Rates have a tolerance in percentage points (pp). A percentage point is the plain difference between two percentages: a tool error rate that goes from 2 % to 4 % changes by +2 pp, even though it doubled. With the default tolerance of 2 pp, that is the largest change that still counts as no regression.

The other metrics have a tolerance in percent of the previous value. Latency that goes from 2.0 s to 2.4 s changes by +20 %.

The **Change** column on the release page uses the same units. A guardrail fails only when Apgar is confident that the change is past the tolerance, not as soon as the measured change crosses it. See [How a metric is judged](https://docs.useapgar.dev/concepts/verdicts.md#how-a-metric-is-judged).

## Change the defaults

Per agent, in **Agent settings, Metrics**:

| Setting | Meaning |
| - | - |
| **Enabled** | Turns the metric on or off for this agent. **Format failure rate** stays off until the output format in **General** is `json` |
| **Tolerance** | The allowed change, in pp for rates and in % for the other metrics |
| **Role** | **guardrail** decides the verdict, **informational** is only shown |

Making a guardrail informational keeps it visible without letting it fail a release. To leave a metric out for one release only, use [expected changes](https://docs.useapgar.dev/concepts/releases.md#expected-changes).

## When a metric is unavailable

Some metrics need data that not every instrumentation records. When that data is missing, or the metric does not apply, the release page lists the metric as unavailable under its group, with the reason.

| Metric | Unavailable when | What to do |
| - | - | - |
| **Tool calls per turn**, **Turns without tools**, **Retry rate**, **Tool error rate** | The previous release made no tool calls | Nothing. An agent without tools is judged on the other guardrails |
| **Retry rate** | Fewer than 90 % of tool calls on one side carry their arguments | Set `gen_ai.tool.call.arguments` on tool spans. See [Tools](https://docs.useapgar.dev/send-traces/attributes.md#tools) |
| **Loop rate** | Fewer than 90 % of turns on one side carry arguments on every tool call | Same as **Retry rate** |
| **Format failure rate** | Fewer than 90 % of turns on one side have output content | Capture output content. See [Content](https://docs.useapgar.dev/send-traces/attributes.md#content) |
| **Output length** | Fewer than 95 % of turns on one side have token counts, and fewer than 95 % have output content | Record output token counts, or capture output content. See [Tokens](https://docs.useapgar.dev/send-traces/attributes.md#tokens) |
| **Cost per turn** | Fewer than 90 % of turns on one side have a cost | Set a cost on each chat span. See [Cost](https://docs.useapgar.dev/send-traces/attributes.md#cost) |
| **Turns per session** | Fewer than 10 closed sessions on one side, or more than half of the turns have no session id | Send a session id. See [Sessions](https://docs.useapgar.dev/concepts/sessions.md) |
| Any metric | It is turned off for this agent | Turn it on in **Agent settings, Metrics** |
| Any metric | It cannot be compared: one side has nothing to measure, the previous value is 0 for a metric with a % tolerance, or the values are too sparse | Wait for more traffic, or make the metric informational |

An unavailable guardrail does not hold the release back; the verdict uses the other guardrails. The exception is a guardrail whose previous value is 0 or whose values are too sparse: it keeps the release in **Not enough data** until it can be compared or is made informational. An agent with no available guardrail never reaches **Healthy**.

## Slices

A slice is the part of a release's traffic that shares one value of a dimension: the turns that called the `web_search` tool, the turns that came from Slack, or the turns answered by one model. Apgar computes every available metric per slice, for the previous release and this one, to show where a change is concentrated.

Select a metric on the release page to see its slices, one table per dimension:

| Dimension | Values |
| - | - |
| **Tool** | Each tool the agent called. A turn that calls two tools counts in both. See [Tools](https://docs.useapgar.dev/send-traces/attributes.md#tools) |
| **Surface** | Each surface, such as Slack or a web chat. See [Surface](https://docs.useapgar.dev/send-traces/attributes.md#surface) |
| **Session depth** | `short` for turns 1 to 5 of a session, `medium` for turns 6 to 20, `long` after that. Turns without a session id have no depth |
| **Response model** | Each model that answered. See [Models](https://docs.useapgar.dev/send-traces/attributes.md#models) |
| **Prompt version** | Each prompt version. See [Prompt version](https://docs.useapgar.dev/send-traces/attributes.md#prompt-version) |

Slices never decide the verdict. The whole release is judged; slices only say where the change is. A slice table counts the slices that regressed, and the message for a failing metric sums them up, for example "Concentrated in web\_search." See [Attribution](https://docs.useapgar.dev/concepts/verdicts.md#attribution).

A slice is compared only when both sides have at least **Minimum turns per slice** turns, 100 by default, or that many tool calls for **Tool error rate** and **Retry rate**. Values that are not compared are listed below the tables with the reason:

| Reason | Meaning |
| - | - |
| Too few turns, or too few tool calls | One side is below **Minimum turns per slice** |
| Previous value is 0, so a relative change is undefined | The metric has a % tolerance and the previous value in this slice is 0 |
| Too sparse for a stable comparison | The slice has too little data for a reliable comparison |
| A reason from [When a metric is unavailable](#when-a-metric-is-unavailable) | The metric is unavailable inside this slice |

**Turns per session** is sliced by the first turn of each session, except that a session counts in a tool slice when any of its turns called the tool. It has no session depth slice.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.