Monitoring

Health checks

Endpoint	Returns
`GET /healthz`	200 while the API process is up
`GET /readyz`	200 with the schema version when ClickHouse is reachable and the schema matches. 503 with `storage_unavailable` or `schema_mismatch` otherwise
`GET /api/v1/status`	Version, storage state, schema version, worker heartbeat age, demo state, and the public URL. No authentication

/healthz and /readyz are also reachable through the UI on port 8080. The worker writes a heartbeat every 10 seconds and removes it when it stops cleanly, so a heartbeat age of null or above 30 seconds means the worker is not running. The UI shows a banner when storage is unavailable.

Logs

Every log line is one JSON object with ts, level, logger, and msg, plus fields for the event and exc for a traceback. Tokens, secrets, message content, and full attribute maps are never logged. Set the level with APGAR_LOG_LEVEL.

Prometheus metrics

The API serves GET /metrics on port 8000. The worker serves GET /metrics on port 8001. When APGAR_WORKER_METRICS_URL is set, as in the compose file, the API also includes the worker’s turn, assembly, and rollup metrics. Scrape both processes to get everything. The Compose file publishes neither port. Run Prometheus on the Compose network, or publish the ports in docker-compose.override.yml, bound to an address that only Prometheus reaches:

docker-compose.override.yml

services:
  apgar-api:
    ports: ["127.0.0.1:8000:8000"]
  apgar-worker:
    ports: ["127.0.0.1:8001:8001"]

On Kubernetes with the Prometheus Operator, enable the chart’s ServiceMonitor. It scrapes the API and the worker directly, and the chart leaves APGAR_WORKER_METRICS_URL unset. Add the labels your Prometheus selects ServiceMonitors by, for example:

values.yaml

metrics:
  serviceMonitor:
    enabled: true
    labels:
      release: prometheus
  prometheusRule:
    enabled: true

metrics.prometheusRule adds alerts for a process that stops answering on /metrics, assembly lag above 30 minutes, and ingest requests that fail to write to ClickHouse. With networkPolicy.enabled, allow Prometheus to reach the worker, or the worker is not scraped:

values.yaml

networkPolicy:
  enabled: true
  metricsFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring

Labels are low cardinality. Counter names end in _total.

Ingest

Metric	Labels	Meaning
`apgar_ingest_requests_total`	`source_id`, `status`	Ingest requests by result
`apgar_ingest_spans_received_total`	`source_id`	Spans in received payloads
`apgar_ingest_spans_accepted_total`	`source_id`	Spans stored
`apgar_ingest_spans_dropped_total`	`source_id`, `reason`	Spans dropped before storage: `invalid_ids` or `invalid_timestamps`
`apgar_ingest_bytes_received_total`	`source_id`	Request body bytes
`apgar_ingest_storage_errors_total`		Ingest writes that failed

Assembly

Metric	Labels	Meaning
`apgar_assembly_lag_seconds`		Age of the oldest trace due for assembly
`apgar_assembly_errors_total`		Traces the assembler failed on
`apgar_turns_assembled_total`	`agent`	Turns written
`apgar_turns_partial_total`	`agent`	Partial turns written, from traces where every span has a parent
`apgar_turns_late_total`	`agent`	Late turns written
`apgar_turns_dropped_total`	`reason`	Traces dropped: `no_agent` or `agent_not_found`
`apgar_rollup_lag_seconds`		Age of the rollup watermark

Verdicts and jobs

Metric	Labels	Meaning
`apgar_releases_registered_total`	`agent`	Releases detected
`apgar_evaluations_total`	`agent`, `state`	Scheduled evaluations completed. Evaluations triggered from the UI, and the last check when monitoring ends, are not counted
`apgar_evaluation_duration_seconds`	`agent`	Scheduled evaluation duration
`apgar_evaluation_errors_total`	`agent`	Evaluations that failed
`apgar_job_duration_seconds`	`job`	Background job duration
`apgar_job_errors_total`	`job`	Background job failures
`apgar_job_last_success_timestamp_seconds`	`job`	Unix time of the last success
`apgar_job_requests_failed_total`	`kind`	Queued requests whose handler failed
`apgar_retention_runs_total`		Retention job runs

Langfuse

Metric	Labels	Meaning
`apgar_langfuse_requests_total`	`connection`, `status`	Requests to Langfuse
`apgar_langfuse_observations_fetched_total`	`connection`	Observations fetched
`apgar_langfuse_filter_overlap_total`	`connection`	Traces matched by more than one agent filter
`apgar_connector_lag_seconds`	`connection`	How far the sync is behind

Notifications and telemetry

Metric	Labels	Meaning
`apgar_notifications_total`	`channel`, `result`	Delivery attempts
`apgar_notifications_failed_rows`		Notifications currently failed
`apgar_telemetry_attempts_total`	`result`	Usage report posts, `sent` or `error`

What to alert on

/readyz not 200.
Worker heartbeat age null or above 60 seconds.
apgar_assembly_lag_seconds well above APGAR_ASSEMBLY_MAX_AGE_SECONDS.
apgar_job_last_success_timestamp_seconds for verdict or notifications older than a few intervals.
apgar_notifications_failed_rows above 0.

Get started

Send traces

Concepts

Notifications

Self hosting

Data and privacy

Health checks

Logs

Prometheus metrics

Ingest

Assembly

Verdicts and jobs

Langfuse

Notifications and telemetry

What to alert on

​Health checks

​Logs

​Prometheus metrics

​Ingest

​Assembly

​Verdicts and jobs

​Langfuse

​Notifications and telemetry

​What to alert on

Health checks

Logs

Prometheus metrics

Ingest

Assembly

Verdicts and jobs

Langfuse

Notifications and telemetry

What to alert on