> ## Documentation Index
> Fetch the complete documentation index at: https://docs.useapgar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev
> Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md

# Kubernetes

> Run Apgar on Kubernetes with the Helm chart.

The Helm chart runs the same services as the Compose file: the API, exactly one worker, the UI, and ClickHouse. ClickHouse runs as one node inside the release, or you point the chart at your own. Every Apgar release publishes the chart at `oci://ghcr.io/oimo-ai/charts/apgar`, with the same version as the images.

## Requirements

* A Kubernetes cluster and `kubectl` access to one namespace.
* Helm 3.14 or later.
* A default StorageClass for the ClickHouse volume, unless you use an [external ClickHouse](#external-clickhouse).
* Free capacity for the requests below, plus 250m CPU for rolling updates and `helm test`.

| Component | Kind | CPU request | Memory request | Memory limit |
| - | - | - | - | - |
| `apgar-clickhouse` | StatefulSet, 1 pod, 50 GiB volume | 500m | 2 GiB | 4 GiB |
| `apgar-worker` | StatefulSet, 1 pod | 1 | 1 GiB | 2 GiB |
| `apgar-api` | Deployment | 250m | 256 MiB | 1 GiB |
| `apgar-ui` | Deployment | 50m | 64 MiB | 256 MiB |

On a node with 2 vCPU, these requests leave no room for a rolling update of the API, so the upgrade waits. Lower the worker request with `--set worker.resources.requests.cpu=500m`. The worker starts one evaluator process per CPU of its limit, minus one. Set `clickhouse.persistence.size` to fit your trace volume and [retention](https://docs.useapgar.dev/self-hosting/configuration.md#retention).

## Install

**1. Create the Secret**

The chart never generates secrets. A key that changed on an upgrade would sign everyone out and make stored connection secrets unreadable.

```bash theme={null}
kubectl create namespace apgar
kubectl -n apgar create secret generic apgar \
  --from-literal=APGAR_SECRET_KEY=$(openssl rand -hex 32) \
  --from-literal=APGAR_ADMIN_TOKEN=$(openssl rand -hex 16) \
  --from-literal=APGAR_CLICKHOUSE_PASSWORD=$(openssl rand -hex 16)
```

Keep a copy of `APGAR_SECRET_KEY` outside the cluster. See [Data handling](https://docs.useapgar.dev/privacy/data-handling.md#keys).

**2. Install the chart**

```bash theme={null}
helm install apgar oci://ghcr.io/oimo-ai/charts/apgar -n apgar \
  --set existingSecret=apgar \
  --set publicUrl=http://localhost:8080
```

`publicUrl` is the address people and exporters use. Message links and the exporter snippet use it. Here it is the address of the port forward in the last step. In a cluster that people and exporters reach, set it to that address and [expose Apgar](#expose-apgar) there instead. Add `--set env.APGAR_TELEMETRY=0` to turn off the anonymous usage report.

**3. Wait for the rollout**

```bash theme={null}
kubectl -n apgar rollout status statefulset/apgar-worker
kubectl -n apgar rollout status deployment/apgar-api
helm test apgar -n apgar
```

The worker applies the migrations in an init container once ClickHouse accepts connections. The API waits in its own init container until the schema matches, then becomes ready. A first install takes a few minutes. `helm test` checks that the UI reaches the API and that the schema matches.

**4. Sign in**

Without an ingress, forward the UI to your machine:

```bash theme={null}
kubectl -n apgar port-forward service/apgar-ui 8080:80
```

Open [http://localhost:8080](http://localhost:8080) and sign in with the admin token:

```bash theme={null}
kubectl -n apgar get secret apgar -o jsonpath='{.data.APGAR_ADMIN_TOKEN}' | base64 -d
```

The names above assume the release is called `apgar`. Another release name prefixes every resource with it, for example `team-apgar-api`.

While no agent exists, the Agents page offers **Seed demo data**. To seed it or remove it from the command line:

```bash theme={null}
kubectl -n apgar exec deploy/apgar-api -- apgar demo seed --purge-first
kubectl -n apgar exec deploy/apgar-api -- apgar demo purge
```

## Expose Apgar

People and exporters reach Apgar through the UI Service, whose nginx forwards `/api/` and `/v1/` to the API. Expose that one Service on the host of `publicUrl`.

### Ingress

```yaml values.yaml theme={null}
publicUrl: https://apgar.example.com
existingSecret: apgar
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
  tls:
    secretName: apgar-tls
```

```bash theme={null}
helm install apgar oci://ghcr.io/oimo-ai/charts/apgar -n apgar -f values.yaml
```

The ingress uses the host of `publicUrl` unless `ingress.host` is set. `tls.secretName` names the Secret with the certificate; with the cert-manager annotation, cert-manager writes it. OTLP requests can carry up to 64 MiB. The chart sets the matching ingress-nginx annotations; with another controller, allow bodies of that size on `/v1/traces`.

### Gateway API

```yaml values.yaml theme={null}
httpRoute:
  enabled: true
  parentRefs:
    - name: gateway
      namespace: gateway-system
```

The route uses the host of `publicUrl` unless `httpRoute.hostnames` is set. TLS is configured on the Gateway.

### Exporters in the cluster

Services in the same cluster can send to the API directly, without the ingress:

```bash theme={null}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://apgar-api.apgar.svc:8000/v1/traces
```

### Proxies and the login rate limit

The login rate limit counts failed sign ins per client address. It reads that address from `X-Forwarded-For`, counting the proxies that append to it from the right. With an ingress or a route, the chart counts two: the ingress controller and the UI's nginx. When a load balancer in front of the ingress controller terminates the client connection, the ingress controller sees only the load balancer, and every sign in counts against its address. Then have the ingress controller keep the address the load balancer passes in `X-Forwarded-For` (for ingress-nginx, `use-forwarded-headers`), and set `api.trustedProxyHops` to 3.

Keep the UI Service of type `ClusterIP` when an ingress or route is enabled. The chart refuses `LoadBalancer` and `NodePort` there, because a client that bypasses the ingress could choose the address the rate limit counts.

## External ClickHouse

Set `clickhouse.enabled=false` to use ClickHouse Cloud or a ClickHouse server you run. Apgar connects over the HTTP interface, port 8123 or 8443 with TLS, not the native port 9000. Point it at one node or service: the tables use single node engines, and a self-managed cluster of replicas is not supported.

**1. Create the database and the user**

```sql theme={null}
CREATE DATABASE apgar;
CREATE USER apgar IDENTIFIED WITH sha256_password BY 'PASSWORD';
GRANT ALL ON apgar.* TO apgar;
```

These grants are enough. A user that may create databases can also skip `CREATE DATABASE`: the worker creates it on the first start.

**2. Create the Secret**

Put the password of that user into `APGAR_CLICKHOUSE_PASSWORD`:

```bash theme={null}
kubectl -n apgar create secret generic apgar \
  --from-literal=APGAR_SECRET_KEY=$(openssl rand -hex 32) \
  --from-literal=APGAR_ADMIN_TOKEN=$(openssl rand -hex 16) \
  --from-literal=APGAR_CLICKHOUSE_PASSWORD='PASSWORD'
```

When the server certificate is signed by a private CA, store the CA too:

```bash theme={null}
kubectl -n apgar create secret generic clickhouse-ca --from-file=ca.crt=ca.crt
```

**3. Install the chart**

```yaml values.yaml theme={null}
publicUrl: https://apgar.example.com
existingSecret: apgar
clickhouse:
  enabled: false
  user: apgar
  database: apgar
  external:
    url: https://abc.clickhouse.cloud:8443
    # Only for a certificate signed by a private CA.
    caSecret: clickhouse-ca
```

```bash theme={null}
helm install apgar oci://ghcr.io/oimo-ai/charts/apgar -n apgar -f values.yaml
```

Then wait for the rollout as in [Install](#install).

`clickhouse.external.verify: false` turns off certificate verification. Use it only for a test.

### When the worker does not start

The worker applies the migrations in an init container. While it cannot connect, it retries for up to `worker.migrateTimeoutSeconds` (600 by default) and logs every attempt. Any other error stops it at once. Read its log:

```bash theme={null}
kubectl -n apgar logs apgar-worker-0 -c migrate
```

| Log | Cause |
| - | - |
| `waiting for clickhouse` with `Connect call failed` | Wrong URL or port, or a network policy or firewall blocks the cluster |
| The container exits with `CERTIFICATE_VERIFY_FAILED` | The certificate is signed by a CA the pod does not trust. Set `clickhouse.external.caSecret` |
| The container exits with an authentication error | Wrong user or password |

Correct the values with `helm upgrade`. The worker restarts with the new values, and the API follows once the migrations are applied.

## Upgrade

Read the [release notes](https://github.com/oimo-ai/apgar/releases) of every version between yours and the new one, and take a [backup](#backups). Then:

```bash theme={null}
helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar --reset-then-reuse-values
```

`--reset-then-reuse-values` keeps the values you set and applies the defaults of the new chart. Or keep your values in a file and pass `-f values.yaml` on every upgrade. The worker applies the new migrations before it starts. New API pods wait for the schema, and the old ones serve requests until then.

Migrations run forward only. `helm rollback` to a version before a migration leaves the worker exiting with code 3. To go back, restore the backup and install the version that wrote it.

## Backups

With the bundled ClickHouse, all durable state lives in the volume `data-apgar-clickhouse-0`. Use ClickHouse `BACKUP` to object storage you control, or the volume snapshots of your storage provider. With an external ClickHouse, back up the `apgar` database there.

## Argo CD and Flux

The chart uses no Helm hooks and generates no values, so `helm template` renders the same resources as `helm install`. Argo CD and Flux apply it like any other chart. Keep the secrets in `existingSecret` and manage that Secret with your secret tooling.

## Uninstall

```bash theme={null}
helm uninstall apgar -n apgar
```

The ClickHouse volume and the Secret you created stay. Delete them to remove the data:

```bash theme={null}
kubectl -n apgar delete pvc data-apgar-clickhouse-0
kubectl -n apgar delete secret apgar
```

## Values

| Value | Default | Description |
| - | - | - |
| `publicUrl` | none, required | Address people and exporters use |
| `existingSecret` | empty | Secret with `APGAR_SECRET_KEY`, `APGAR_ADMIN_TOKEN`, and `APGAR_CLICKHOUSE_PASSWORD` |
| `secret.secretKey`, `secret.adminToken`, `secret.clickhousePassword` | empty | Used only without `existingSecret`. The chart creates the Secret from them |
| `clickhouse.enabled` | `true` | Run one ClickHouse node in the release |
| `clickhouse.persistence.size` | `50Gi` | Size of the ClickHouse volume |
| `clickhouse.external.url` | empty | HTTP interface of an external ClickHouse |
| `clickhouse.external.caSecret` | empty | Secret with `ca.crt` for a private CA |
| `ingress.enabled`, `httpRoute.enabled` | `false` | Expose the UI. See [Expose Apgar](#expose-apgar) |
| `api.replicas` | `1` | API pods. `api.autoscaling.enabled` adds a HorizontalPodAutoscaler instead |
| `api.trustedProxyHops` | 1, or 2 with an ingress or route | See [Proxies and the login rate limit](#proxies-and-the-login-rate-limit) |
| `worker.resources` | see [Requirements](#requirements) | The CPU limit sets the number of evaluator processes |
| `env` | empty | Extra `APGAR_` variables, for example `APGAR_TELEMETRY: "0"`. See [Configuration](https://docs.useapgar.dev/self-hosting/configuration.md) |
| `metrics.serviceMonitor.enabled` | `false` | Scrape the API and the worker with the Prometheus Operator |
| `networkPolicy.enabled` | `false` | Allow ClickHouse only from Apgar, and the worker metrics only from `networkPolicy.metricsFrom` |

`helm show values oci://ghcr.io/oimo-ai/charts/apgar` prints every value with its comment.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.