Webhooks

A webhook connection receives the same events as Slack, as signed JSON. Use it to feed a deploy pipeline, an incident tool, or your own bot.

Set up

In Settings, Connections, add a webhook connection with:

Field	Rule
`url`	An `http` or `https` URL
`secret`	16 to 256 characters. Used to sign each request. Stored encrypted, shown masked afterwards

Every webhook connection receives the events of every agent, demo agents included. Select Test to send a signed test request. A 2xx response marks the connection ok; any other response or a network error marks it error.

Events

`event`	When
`release_detected`	The first turn of a new release arrives
`regression`	A guardrail failure is confirmed
`healthy`	A pass is confirmed and the agent has Send the healthy message on
`ended`	Monitoring ended without a decision because the monitoring window ran out or someone stopped it
`test`	The connection test

Body

The body is JSON with sorted keys, no extra whitespace, and non-ASCII characters escaped.

Field	Meaning
`event`	One of the events above
`idempotency_key`	Stable key of this notification. Retries send the same value
`agent`	Agent name
`release`	Release identifier
`demo`	`true` for demo agents
`previous_verdict`	State before this transition. `null` for `release_detected` and `test`
`verdict`	Summary of the release, see below
`text`	Plain text version of the Slack message. Contains the release link only for `release_detected` and `regression`
`sent_at`	UTC time the notification was queued, with millisecond precision and a `Z` suffix. A retry resends the identical body, including this value

verdict holds:

Field	Meaning
`verdict`, `reason`, `final`	Engine state, reason code, and whether monitoring has ended
`baseline`	`release`, `mode` (`previous_release` or `manual`), and `unverified`
`sample`	`candidate_turns`, `baseline_turns`, `candidate_sessions`, `baseline_sessions`
`window`	`start`, `end`, and `max_end` of the candidate window
`evaluated_at`	Time of the evaluation
`early`	`true` when the regression was found below the minimum sample
`ended`	How monitoring ended, or `null`
`detectability`	One sentence on the smallest change the sample could detect, or `null`
`failed_metrics`	One entry per failing guardrail: `id`, `baseline`, `candidate`, `delta`, `ci`, `tolerance`, `tolerance_mode`, `direction`, `unit`, `worst_slice`, `headline`
`report_url`	Link to the release page

For release_detected the summary is the pending state of the release with the baseline Apgar chose. The test request has event set to test, idempotency_key set to test: followed by sent_at, agent, release, previous_verdict, and verdict set to null, demo set to false, and text set to Apgar connection test.

Failed metrics

Each entry of failed_metrics names its metric by id. direction is the direction that counts as worse: up, down, or both. delta, ci, and tolerance are fractions: with tolerance_mode abs they are absolute differences, so 0.02 is 2 pp; with rel they are relative to the baseline value, so 0.10 is 10 %.

`id`	Metric
`tool_calls_per_turn`	Tool calls per turn
`zero_tool_turn_rate`	Turns without tools
`tool_retry_rate`	Retry rate
`loop_rate`	Loop rate
`output_tokens_p50`	Output length
`format_failure_rate`	Format failure rate
`turns_per_session`	Turns per session
`tool_error_rate`	Tool error rate
`llm_error_rate`	LLM error rate
`cost_per_turn`	Cost per turn
`latency_p95`	Latency p95

See Metrics for what each one measures.

Headers

Header	Value
`Content-Type`	`application/json`
`X-Apgar-Timestamp`	Unix seconds at the time of this attempt
`X-Apgar-Signature`	Lowercase hex HMAC-SHA256 of `timestamp + "." + body` with the connection secret
`X-Apgar-Event`	The `event` field
`User-Agent`	`apgar/VERSION`

Verify a request

A receiver should:

Reject a request whose timestamp differs from its own clock by more than 5 minutes.
Compute the HMAC-SHA256 of X-Apgar-Timestamp, a period, and the raw body bytes with the connection secret, and compare it with X-Apgar-Signature in constant time. Parsing and serializing the JSON again changes the bytes and breaks the signature.
Store idempotency_key and ignore a delivery whose key it has already accepted. A retry after a timeout can arrive although the first attempt was handled.

import hashlib
import hmac
import time


def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:
    if abs(time.time() - int(timestamp)) > 300:
        return False
    message = timestamp.encode() + b"." + body
    expected = hmac.new(secret.encode(), message, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

import { createHmac, timingSafeEqual } from "node:crypto";

export function verify(secret, timestamp, body, signature) {
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) {
    return false;
  }
  const expected = createHmac("sha256", secret)
    .update(`${timestamp}.`)
    .update(body)
    .digest("hex");
  const left = Buffer.from(expected);
  const right = Buffer.from(signature);
  return left.length === right.length && timingSafeEqual(left, right);
}

Idempotency keys

Notification	Key
Release detected	`AGENT:RELEASE:release_detected`
Transition	`AGENT:RELEASE:PREVIOUS->NEW:EVALUATED_AT`
Second event of the same transition	The transition key followed by `:EVENT`

Every webhook connection receives the same key for the same notification.

Retries

A network error, a 429, or a 5xx response is retried after 1, 5, 15, and 60 minutes. After the fifth failed attempt the notification is marked failed. Any other status fails at once. Settings, Notifications lists failed deliveries and can queue them again.

Get started

Send traces

Concepts

Notifications

Self hosting

Data and privacy

Set up

Events

Body

Failed metrics

Headers

Verify a request

Idempotency keys

Retries

​Set up

​Events

​Body

​Failed metrics

​Headers

​Verify a request

​Idempotency keys

​Retries

Set up

Events

Body

Failed metrics

Headers

Verify a request

Idempotency keys

Retries