Data handling

Apgar is self hosted. Traces stay in the ClickHouse volume on your host. It needs no account with the project and uploads no trace data. Prompt text, completion text, and tool arguments are never stored.

​

What is stored

A turn stores identifiers, timestamps, counts, and a small set of fields: agent, release, environment, session id, surface, prompt version, models, token counts, cost, output length, and whether the output parsed as JSON. A tool call stores the tool name, its status, an error type, and the argument hash described below. A span keeps its name, kind, timing, status, and an error type of up to 128 characters taken from error.type, exception.type, or the status message. Spans keep an allowlist of attributes: the ones listed in the attribute reference, plus the attributes that agents name in Release attribute and Surface attribute. For Langfuse, every trace and observation metadata key, the trace tags, and the user id (langfuse.user.id) are stored as well. Content attributes are dropped even when named in an agent setting:

• gen_ai.input.messages, gen_ai.output.messages
• gen_ai.tool.call.arguments, gen_ai.tool.call.result
• OpenInference input.value, output.value, llm.input_messages, llm.output_messages, llm.prompts, llm.completions
• Langfuse observation input and output

Attribute values are truncated to 256 characters.

​

Tool arguments

Retry and loop detection compare tool arguments without keeping them. args_hash is an HMAC-SHA256 of the arguments, of their canonical JSON when they parse as JSON and of the raw string otherwise, under a key derived from APGAR_SECRET_KEY. Without the key, a hash cannot be checked against guessed arguments. Hashes are only compared within one turn.

​

Secrets

• Connection secrets (Langfuse keys, Slack webhook URLs, webhook secrets) are encrypted with AES-256-GCM before they are written, under a key derived from APGAR_SECRET_KEY with HKDF. The API returns Langfuse secret keys and webhook secrets masked. Slack webhook URLs are returned in full to signed in users.
• Ingest tokens are shown once. Apgar stores a SHA-256 hash and a short prefix to identify them.
• The admin token is read from the environment and never stored.

​

Keys

APGAR_SECRET_KEY is the root secret. Losing it makes stored connection secrets unreadable. To rotate it with Docker Compose:

1. In .env, set APGAR_SECRET_KEY_PREVIOUS to the current key and APGAR_SECRET_KEY to a new one.
2. Run docker compose up -d, which restarts the API and the worker. The worker decrypts every connection with the previous key and encrypts it again with the new one.
3. Remove APGAR_SECRET_KEY_PREVIOUS from .env and run docker compose up -d again.

With the Helm chart and existingSecret, pass the chart version you run now, from helm list -n apgar, so that the rotation does not also upgrade Apgar:

1. Copy the current key into a second Secret, and put a new key into the chart’s Secret:
   kubectl -n apgar create secret generic apgar-previous-key \
     --from-literal=APGAR_SECRET_KEY_PREVIOUS="$(kubectl -n apgar get secret apgar -o jsonpath='{.data.APGAR_SECRET_KEY}' | base64 -d)"
   kubectl -n apgar patch secret apgar --type merge \
     -p "{\"stringData\":{\"APGAR_SECRET_KEY\":\"$(openssl rand -hex 32)\"}}"
   
2. Upgrade with the second Secret as an extra variable source. This restarts the API and the worker, and the worker logs re-encrypted N connections:
   helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar \
     --reset-then-reuse-values --set 'envFrom[0].secretRef.name=apgar-previous-key'
   
3. Remove the previous key:
   helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar \
     --reset-then-reuse-values --set-json 'envFrom=[]'
   kubectl -n apgar delete secret apgar-previous-key
   

If you already set envFrom, add the Secret to your list and remove it from there instead. Without existingSecret, the chart’s Secret is apgar-secrets: read the current key from it in step 1, and set the new key with --set secret.secretKey=... in step 2 instead of patching the Secret. Rotation signs out every UI session. Tool argument hashes written under the old key do not match new ones, which matters only for a turn that is in flight during the restart.

​

Outbound connections

The worker connects only to what you configured:

• Langfuse, at the host of each Langfuse connection.
• Slack, at the webhook URL of each Slack connection.
• The URL of each webhook connection.
• PostHog, at the telemetry endpoint, once a day, unless telemetry is off.

Ingest makes no outbound calls. The API calls out only to test a connection and, when APGAR_WORKER_METRICS_URL is set, to read the worker metrics for its own /metrics endpoint.

​

Deletion

• Delete agent hides the agent and revokes its ingest tokens at once. The retention job, which runs at worker startup and then daily, deletes its spans, turns, tool calls, releases, evaluations, and notifications.
• apgar demo purge deletes the rows of the demo agents the same way and hides the agents.
• Retention deletes trace data after trace_data_days, and evaluations and notifications after verdict_history_days. Release rows and release events are kept. See Retention.