> ## Documentation Index
> Fetch the complete documentation index at: https://docs.useapgar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev
> Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md

# Data handling

> What Apgar stores, what it drops, and where it connects to.

Apgar is self hosted. Traces stay in the ClickHouse volume on your host. It needs no account with the project and uploads no trace data.

**Prompt text, completion text, and tool arguments are never stored.**

## What is stored

A turn stores identifiers, timestamps, counts, and a small set of fields: agent, release, environment, session id, surface, prompt version, models, token counts, cost, output length, and whether the output parsed as JSON. A tool call stores the tool name, its status, an error type, and the argument hash described below. A span keeps its name, kind, timing, status, and an error type of up to 128 characters taken from `error.type`, `exception.type`, or the status message.

Spans keep an allowlist of attributes: the ones listed in the [attribute reference](https://docs.useapgar.dev/send-traces/attributes.md), plus the attributes that agents name in **Release attribute** and **Surface attribute**. For Langfuse, every trace and observation metadata key, the trace tags, and the user id (`langfuse.user.id`) are stored as well. Content attributes are dropped even when named in an agent setting:

* `gen_ai.input.messages`, `gen_ai.output.messages`
* `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`
* OpenInference `input.value`, `output.value`, `llm.input_messages`, `llm.output_messages`, `llm.prompts`, `llm.completions`
* Langfuse observation input and output

Attribute values are truncated to 256 characters.

## Tool arguments

Retry and loop detection compare tool arguments without keeping them. `args_hash` is an HMAC-SHA256 of the arguments, of their canonical JSON when they parse as JSON and of the raw string otherwise, under a key derived from `APGAR_SECRET_KEY`. Without the key, a hash cannot be checked against guessed arguments. Hashes are only compared within one turn.

## Secrets

* **Connection secrets** (Langfuse keys, Slack webhook URLs, webhook secrets) are encrypted with AES-256-GCM before they are written, under a key derived from `APGAR_SECRET_KEY` with HKDF. The API returns Langfuse secret keys and webhook secrets masked. Slack webhook URLs are returned in full to signed in users.
* **Ingest tokens** are shown once. Apgar stores a SHA-256 hash and a short prefix to identify them.
* **The admin token** is read from the environment and never stored.

## Keys

`APGAR_SECRET_KEY` is the root secret. Losing it makes stored connection secrets unreadable. To rotate it with Docker Compose:

1. In `.env`, set `APGAR_SECRET_KEY_PREVIOUS` to the current key and `APGAR_SECRET_KEY` to a new one.
2. Run `docker compose up -d`, which restarts the API and the worker. The worker decrypts every connection with the previous key and encrypts it again with the new one.
3. Remove `APGAR_SECRET_KEY_PREVIOUS` from `.env` and run `docker compose up -d` again.

With the Helm chart and `existingSecret`, pass the chart version you run now, from `helm list -n apgar`, so that the rotation does not also upgrade Apgar:

1. Copy the current key into a second Secret, and put a new key into the chart's Secret:

   ```bash theme={null}
   kubectl -n apgar create secret generic apgar-previous-key \
     --from-literal=APGAR_SECRET_KEY_PREVIOUS="$(kubectl -n apgar get secret apgar -o jsonpath='{.data.APGAR_SECRET_KEY}' | base64 -d)"
   kubectl -n apgar patch secret apgar --type merge \
     -p "{\"stringData\":{\"APGAR_SECRET_KEY\":\"$(openssl rand -hex 32)\"}}"
   ```

2. Upgrade with the second Secret as an extra variable source. This restarts the API and the worker, and the worker logs `re-encrypted N connections`:

   ```bash theme={null}
   helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar \
     --reset-then-reuse-values --set 'envFrom[0].secretRef.name=apgar-previous-key'
   ```

3. Remove the previous key:

   ```bash theme={null}
   helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar \
     --reset-then-reuse-values --set-json 'envFrom=[]'
   kubectl -n apgar delete secret apgar-previous-key
   ```

If you already set `envFrom`, add the Secret to your list and remove it from there instead. Without `existingSecret`, the chart's Secret is `apgar-secrets`: read the current key from it in step 1, and set the new key with `--set secret.secretKey=...` in step 2 instead of patching the Secret.

Rotation signs out every UI session. Tool argument hashes written under the old key do not match new ones, which matters only for a turn that is in flight during the restart.

## Outbound connections

The worker connects only to what you configured:

* Langfuse, at the host of each Langfuse connection.
* Slack, at the webhook URL of each Slack connection.
* The URL of each webhook connection.
* PostHog, at the telemetry endpoint, once a day, unless telemetry is off.

Ingest makes no outbound calls. The API calls out only to test a connection and, when `APGAR_WORKER_METRICS_URL` is set, to read the worker metrics for its own `/metrics` endpoint.

## Deletion

* **Delete agent** hides the agent and revokes its ingest tokens at once. The retention job, which runs at worker startup and then daily, deletes its spans, turns, tool calls, releases, evaluations, and notifications.
* `apgar demo purge` deletes the rows of the demo agents the same way and hides the agents.
* Retention deletes trace data after `trace_data_days`, and evaluations and notifications after `verdict_history_days`. Release rows and release events are kept. See [Retention](https://docs.useapgar.dev/self-hosting/configuration.md#retention).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.