> ## Documentation Index
> Fetch the complete documentation index at: https://docs.useapgar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev
> Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md

# Webhooks

> Receive signed JSON for every release event.

A webhook connection receives the same events as Slack, as signed JSON. Use it to feed a deploy pipeline, an incident tool, or your own bot.

## Set up

In **Settings, Connections**, add a webhook connection with:

| Field | Rule |
| - | - |
| `url` | An `http` or `https` URL |
| `secret` | 16 to 256 characters. Used to sign each request. Stored encrypted, shown masked afterwards |

Every webhook connection receives the events of every agent, demo agents included. Select **Test** to send a signed test request. A 2xx response marks the connection `ok`; any other response or a network error marks it `error`.

## Events

| `event` | When |
| - | - |
| `release_detected` | The first turn of a new release arrives |
| `regression` | A guardrail failure is confirmed |
| `healthy` | A pass is confirmed and the agent has **Send the healthy message** on |
| `ended` | Monitoring ended without a decision because the monitoring window ran out or someone stopped it |
| `test` | The connection test |

## Body

The body is JSON with sorted keys, no extra whitespace, and non-ASCII characters escaped.

| Field | Meaning |
| - | - |
| `event` | One of the events above |
| `idempotency_key` | Stable key of this notification. Retries send the same value |
| `agent` | Agent name |
| `release` | Release identifier |
| `demo` | `true` for demo agents |
| `previous_verdict` | State before this transition. `null` for `release_detected` and `test` |
| `verdict` | Summary of the release, see below |
| `text` | Plain text version of the Slack message. Contains the release link only for `release_detected` and `regression` |
| `sent_at` | UTC time the notification was queued, with millisecond precision and a `Z` suffix. A retry resends the identical body, including this value |

`verdict` holds:

| Field | Meaning |
| - | - |
| `verdict`, `reason`, `final` | Engine state, reason code, and whether monitoring has ended |
| `baseline` | `release`, `mode` (`previous_release` or `manual`), and `unverified` |
| `sample` | `candidate_turns`, `baseline_turns`, `candidate_sessions`, `baseline_sessions` |
| `window` | `start`, `end`, and `max_end` of the candidate window |
| `evaluated_at` | Time of the evaluation |
| `early` | `true` when the regression was found below the minimum sample |
| `ended` | How monitoring ended, or `null` |
| `detectability` | One sentence on the smallest change the sample could detect, or `null` |
| `failed_metrics` | One entry per failing guardrail: `id`, `baseline`, `candidate`, `delta`, `ci`, `tolerance`, `tolerance_mode`, `direction`, `unit`, `worst_slice`, `headline` |
| `report_url` | Link to the release page |

For `release_detected` the summary is the pending state of the release with the baseline Apgar chose. The test request has `event` set to `test`, `idempotency_key` set to `test:` followed by `sent_at`, `agent`, `release`, `previous_verdict`, and `verdict` set to `null`, `demo` set to `false`, and `text` set to `Apgar connection test`.

### Failed metrics

Each entry of `failed_metrics` names its metric by `id`. `direction` is the direction that counts as worse: `up`, `down`, or `both`. `delta`, `ci`, and `tolerance` are fractions: with `tolerance_mode` `abs` they are absolute differences, so 0.02 is 2 pp; with `rel` they are relative to the baseline value, so 0.10 is 10 %.

| `id` | Metric |
| - | - |
| `tool_calls_per_turn` | Tool calls per turn |
| `zero_tool_turn_rate` | Turns without tools |
| `tool_retry_rate` | Retry rate |
| `loop_rate` | Loop rate |
| `output_tokens_p50` | Output length |
| `format_failure_rate` | Format failure rate |
| `turns_per_session` | Turns per session |
| `tool_error_rate` | Tool error rate |
| `llm_error_rate` | LLM error rate |
| `cost_per_turn` | Cost per turn |
| `latency_p95` | Latency p95 |

See [Metrics](https://docs.useapgar.dev/concepts/metrics.md) for what each one measures.

## Headers

| Header | Value |
| - | - |
| `Content-Type` | `application/json` |
| `X-Apgar-Timestamp` | Unix seconds at the time of this attempt |
| `X-Apgar-Signature` | Lowercase hex HMAC-SHA256 of `timestamp + "." + body` with the connection secret |
| `X-Apgar-Event` | The `event` field |
| `User-Agent` | `apgar/VERSION` |

## Verify a request

A receiver should:

1. Reject a request whose timestamp differs from its own clock by more than 5 minutes.
2. Compute the HMAC-SHA256 of `X-Apgar-Timestamp`, a period, and the raw body bytes with the connection secret, and compare it with `X-Apgar-Signature` in constant time. Parsing and serializing the JSON again changes the bytes and breaks the signature.
3. Store `idempotency_key` and ignore a delivery whose key it has already accepted. A retry after a timeout can arrive although the first attempt was handled.

```python Python theme={null}
import hashlib
import hmac
import time


def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:
    if abs(time.time() - int(timestamp)) > 300:
        return False
    message = timestamp.encode() + b"." + body
    expected = hmac.new(secret.encode(), message, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)
```

```javascript Node.js theme={null}
import { createHmac, timingSafeEqual } from "node:crypto";

export function verify(secret, timestamp, body, signature) {
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) {
    return false;
  }
  const expected = createHmac("sha256", secret)
    .update(`${timestamp}.`)
    .update(body)
    .digest("hex");
  const left = Buffer.from(expected);
  const right = Buffer.from(signature);
  return left.length === right.length && timingSafeEqual(left, right);
}
```

## Idempotency keys

| Notification | Key |
| - | - |
| Release detected | `AGENT:RELEASE:release_detected` |
| Transition | `AGENT:RELEASE:PREVIOUS->NEW:EVALUATED_AT` |
| Second event of the same transition | The transition key followed by `:EVENT` |

Every webhook connection receives the same key for the same notification.

## Retries

A network error, a `429`, or a 5xx response is retried after 1, 5, 15, and 60 minutes. After the fifth failed attempt the notification is marked failed. Any other status fails at once. **Settings, Notifications** lists failed deliveries and can queue them again.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.