> ## Documentation Index
> Fetch the complete documentation index at: https://docs.useapgar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev
> Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md

# Monitoring

> Health checks, logs, and Prometheus metrics.

## Health checks

| Endpoint | Returns |
| - | - |
| `GET /healthz` | 200 while the API process is up |
| `GET /readyz` | 200 with the schema version when ClickHouse is reachable and the schema matches. 503 with `storage_unavailable` or `schema_mismatch` otherwise |
| `GET /api/v1/status` | Version, storage state, schema version, worker heartbeat age, demo state, and the public URL. No authentication |

`/healthz` and `/readyz` are also reachable through the UI on port 8080. The worker writes a heartbeat every 10 seconds and removes it when it stops cleanly, so a heartbeat age of `null` or above 30 seconds means the worker is not running. The UI shows a banner when storage is unavailable.

## Logs

Every log line is one JSON object with `ts`, `level`, `logger`, and `msg`, plus fields for the event and `exc` for a traceback. Tokens, secrets, message content, and full attribute maps are never logged. Set the level with `APGAR_LOG_LEVEL`.

## Prometheus metrics

The API serves `GET /metrics` on port 8000. The worker serves `GET /metrics` on port 8001. When `APGAR_WORKER_METRICS_URL` is set, as in the compose file, the API also includes the worker's turn, assembly, and rollup metrics. Scrape both processes to get everything.

The Compose file publishes neither port. Run Prometheus on the Compose network, or publish the ports in `docker-compose.override.yml`, bound to an address that only Prometheus reaches:

```yaml docker-compose.override.yml theme={null}
services:
  apgar-api:
    ports: ["127.0.0.1:8000:8000"]
  apgar-worker:
    ports: ["127.0.0.1:8001:8001"]
```

On Kubernetes with the Prometheus Operator, enable the chart's ServiceMonitor. It scrapes the API and the worker directly, and the chart leaves `APGAR_WORKER_METRICS_URL` unset. Add the labels your Prometheus selects ServiceMonitors by, for example:

```yaml values.yaml theme={null}
metrics:
  serviceMonitor:
    enabled: true
    labels:
      release: prometheus
  prometheusRule:
    enabled: true
```

`metrics.prometheusRule` adds alerts for a process that stops answering on `/metrics`, assembly lag above 30 minutes, and ingest requests that fail to write to ClickHouse. With `networkPolicy.enabled`, allow Prometheus to reach the worker, or the worker is not scraped:

```yaml values.yaml theme={null}
networkPolicy:
  enabled: true
  metricsFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
```

Labels are low cardinality. Counter names end in `_total`.

### Ingest

| Metric | Labels | Meaning |
| - | - | - |
| `apgar_ingest_requests_total` | `source_id`, `status` | Ingest requests by result |
| `apgar_ingest_spans_received_total` | `source_id` | Spans in received payloads |
| `apgar_ingest_spans_accepted_total` | `source_id` | Spans stored |
| `apgar_ingest_spans_dropped_total` | `source_id`, `reason` | Spans dropped before storage: `invalid_ids` or `invalid_timestamps` |
| `apgar_ingest_bytes_received_total` | `source_id` | Request body bytes |
| `apgar_ingest_storage_errors_total` | | Ingest writes that failed |

### Assembly

| Metric | Labels | Meaning |
| - | - | - |
| `apgar_assembly_lag_seconds` | | Age of the oldest trace due for assembly |
| `apgar_assembly_errors_total` | | Traces the assembler failed on |
| `apgar_turns_assembled_total` | `agent` | Turns written |
| `apgar_turns_partial_total` | `agent` | Partial turns written, from traces where every span has a parent |
| `apgar_turns_late_total` | `agent` | Late turns written |
| `apgar_turns_dropped_total` | `reason` | Traces dropped: `no_agent` or `agent_not_found` |
| `apgar_rollup_lag_seconds` | | Age of the rollup watermark |

### Verdicts and jobs

| Metric | Labels | Meaning |
| - | - | - |
| `apgar_releases_registered_total` | `agent` | Releases detected |
| `apgar_evaluations_total` | `agent`, `state` | Scheduled evaluations completed. Evaluations triggered from the UI, and the last check when monitoring ends, are not counted |
| `apgar_evaluation_duration_seconds` | `agent` | Scheduled evaluation duration |
| `apgar_evaluation_errors_total` | `agent` | Evaluations that failed |
| `apgar_job_duration_seconds` | `job` | Background job duration |
| `apgar_job_errors_total` | `job` | Background job failures |
| `apgar_job_last_success_timestamp_seconds` | `job` | Unix time of the last success |
| `apgar_job_requests_failed_total` | `kind` | Queued requests whose handler failed |
| `apgar_retention_runs_total` | | Retention job runs |

### Langfuse

| Metric | Labels | Meaning |
| - | - | - |
| `apgar_langfuse_requests_total` | `connection`, `status` | Requests to Langfuse |
| `apgar_langfuse_observations_fetched_total` | `connection` | Observations fetched |
| `apgar_langfuse_filter_overlap_total` | `connection` | Traces matched by more than one agent filter |
| `apgar_connector_lag_seconds` | `connection` | How far the sync is behind |

### Notifications and telemetry

| Metric | Labels | Meaning |
| - | - | - |
| `apgar_notifications_total` | `channel`, `result` | Delivery attempts |
| `apgar_notifications_failed_rows` | | Notifications currently failed |
| `apgar_telemetry_attempts_total` | `result` | Usage report posts, `sent` or `error` |

## What to alert on

* `/readyz` not 200.
* Worker heartbeat age `null` or above 60 seconds.
* `apgar_assembly_lag_seconds` well above `APGAR_ASSEMBLY_MAX_AGE_SECONDS`.
* `apgar_job_last_success_timestamp_seconds` for `verdict` or `notifications` older than a few intervals.
* `apgar_notifications_failed_rows` above 0.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.