Kubernetes

The Helm chart runs the same services as the Compose file: the API, exactly one worker, the UI, and ClickHouse. ClickHouse runs as one node inside the release, or you point the chart at your own. Every Apgar release publishes the chart at oci://ghcr.io/oimo-ai/charts/apgar, with the same version as the images.

Requirements

A Kubernetes cluster and kubectl access to one namespace.
Helm 3.14 or later.
A default StorageClass for the ClickHouse volume, unless you use an external ClickHouse.
Free capacity for the requests below, plus 250m CPU for rolling updates and helm test.

Component	Kind	CPU request	Memory request	Memory limit
`apgar-clickhouse`	StatefulSet, 1 pod, 50 GiB volume	500m	2 GiB	4 GiB
`apgar-worker`	StatefulSet, 1 pod	1	1 GiB	2 GiB
`apgar-api`	Deployment	250m	256 MiB	1 GiB
`apgar-ui`	Deployment	50m	64 MiB	256 MiB

On a node with 2 vCPU, these requests leave no room for a rolling update of the API, so the upgrade waits. Lower the worker request with --set worker.resources.requests.cpu=500m. The worker starts one evaluator process per CPU of its limit, minus one. Set clickhouse.persistence.size to fit your trace volume and retention.

Install

Create the Secret

The chart never generates secrets. A key that changed on an upgrade would sign everyone out and make stored connection secrets unreadable.

kubectl create namespace apgar
kubectl -n apgar create secret generic apgar \
  --from-literal=APGAR_SECRET_KEY=$(openssl rand -hex 32) \
  --from-literal=APGAR_ADMIN_TOKEN=$(openssl rand -hex 16) \
  --from-literal=APGAR_CLICKHOUSE_PASSWORD=$(openssl rand -hex 16)

Keep a copy of APGAR_SECRET_KEY outside the cluster. See Data handling.

Install the chart

helm install apgar oci://ghcr.io/oimo-ai/charts/apgar -n apgar \
  --set existingSecret=apgar \
  --set publicUrl=http://localhost:8080

publicUrl is the address people and exporters use. Message links and the exporter snippet use it. Here it is the address of the port forward in the last step. In a cluster that people and exporters reach, set it to that address and expose Apgar there instead. Add --set env.APGAR_TELEMETRY=0 to turn off the anonymous usage report.

Wait for the rollout

kubectl -n apgar rollout status statefulset/apgar-worker
kubectl -n apgar rollout status deployment/apgar-api
helm test apgar -n apgar

The worker applies the migrations in an init container once ClickHouse accepts connections. The API waits in its own init container until the schema matches, then becomes ready. A first install takes a few minutes. helm test checks that the UI reaches the API and that the schema matches.

Without an ingress, forward the UI to your machine:

kubectl -n apgar port-forward service/apgar-ui 8080:80

Open http://localhost:8080 and sign in with the admin token:

kubectl -n apgar get secret apgar -o jsonpath='{.data.APGAR_ADMIN_TOKEN}' | base64 -d

The names above assume the release is called apgar. Another release name prefixes every resource with it, for example team-apgar-api. While no agent exists, the Agents page offers Seed demo data. To seed it or remove it from the command line:

kubectl -n apgar exec deploy/apgar-api -- apgar demo seed --purge-first
kubectl -n apgar exec deploy/apgar-api -- apgar demo purge

Expose Apgar

People and exporters reach Apgar through the UI Service, whose nginx forwards /api/ and /v1/ to the API. Expose that one Service on the host of publicUrl.

Ingress

values.yaml

publicUrl: https://apgar.example.com
existingSecret: apgar
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
  tls:
    secretName: apgar-tls

helm install apgar oci://ghcr.io/oimo-ai/charts/apgar -n apgar -f values.yaml

The ingress uses the host of publicUrl unless ingress.host is set. tls.secretName names the Secret with the certificate; with the cert-manager annotation, cert-manager writes it. OTLP requests can carry up to 64 MiB. The chart sets the matching ingress-nginx annotations; with another controller, allow bodies of that size on /v1/traces.

Gateway API

values.yaml

httpRoute:
  enabled: true
  parentRefs:
    - name: gateway
      namespace: gateway-system

The route uses the host of publicUrl unless httpRoute.hostnames is set. TLS is configured on the Gateway.

Exporters in the cluster

Services in the same cluster can send to the API directly, without the ingress:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://apgar-api.apgar.svc:8000/v1/traces

The login rate limit counts failed sign ins per client address. It reads that address from X-Forwarded-For, counting the proxies that append to it from the right. With an ingress or a route, the chart counts two: the ingress controller and the UI’s nginx. When a load balancer in front of the ingress controller terminates the client connection, the ingress controller sees only the load balancer, and every sign in counts against its address. Then have the ingress controller keep the address the load balancer passes in X-Forwarded-For (for ingress-nginx, use-forwarded-headers), and set api.trustedProxyHops to 3. Keep the UI Service of type ClusterIP when an ingress or route is enabled. The chart refuses LoadBalancer and NodePort there, because a client that bypasses the ingress could choose the address the rate limit counts.

External ClickHouse

Set clickhouse.enabled=false to use ClickHouse Cloud or a ClickHouse server you run. Apgar connects over the HTTP interface, port 8123 or 8443 with TLS, not the native port 9000. Point it at one node or service: the tables use single node engines, and a self-managed cluster of replicas is not supported.

Create the database and the user

CREATE DATABASE apgar;
CREATE USER apgar IDENTIFIED WITH sha256_password BY 'PASSWORD';
GRANT ALL ON apgar.* TO apgar;

These grants are enough. A user that may create databases can also skip CREATE DATABASE: the worker creates it on the first start.

Create the Secret

Put the password of that user into APGAR_CLICKHOUSE_PASSWORD:

kubectl -n apgar create secret generic apgar \
  --from-literal=APGAR_SECRET_KEY=$(openssl rand -hex 32) \
  --from-literal=APGAR_ADMIN_TOKEN=$(openssl rand -hex 16) \
  --from-literal=APGAR_CLICKHOUSE_PASSWORD='PASSWORD'

When the server certificate is signed by a private CA, store the CA too:

kubectl -n apgar create secret generic clickhouse-ca --from-file=ca.crt=ca.crt

Install the chart

values.yaml

publicUrl: https://apgar.example.com
existingSecret: apgar
clickhouse:
  enabled: false
  user: apgar
  database: apgar
  external:
    url: https://abc.clickhouse.cloud:8443
    # Only for a certificate signed by a private CA.
    caSecret: clickhouse-ca

helm install apgar oci://ghcr.io/oimo-ai/charts/apgar -n apgar -f values.yaml

Then wait for the rollout as in Install.

clickhouse.external.verify: false turns off certificate verification. Use it only for a test.

When the worker does not start

The worker applies the migrations in an init container. While it cannot connect, it retries for up to worker.migrateTimeoutSeconds (600 by default) and logs every attempt. Any other error stops it at once. Read its log:

kubectl -n apgar logs apgar-worker-0 -c migrate

Log	Cause
`waiting for clickhouse` with `Connect call failed`	Wrong URL or port, or a network policy or firewall blocks the cluster
The container exits with `CERTIFICATE_VERIFY_FAILED`	The certificate is signed by a CA the pod does not trust. Set `clickhouse.external.caSecret`
The container exits with an authentication error	Wrong user or password

Correct the values with helm upgrade. The worker restarts with the new values, and the API follows once the migrations are applied.

Upgrade

Read the release notes of every version between yours and the new one, and take a backup. Then:

helm upgrade apgar oci://ghcr.io/oimo-ai/charts/apgar --version X.Y.Z -n apgar --reset-then-reuse-values

--reset-then-reuse-values keeps the values you set and applies the defaults of the new chart. Or keep your values in a file and pass -f values.yaml on every upgrade. The worker applies the new migrations before it starts. New API pods wait for the schema, and the old ones serve requests until then. Migrations run forward only. helm rollback to a version before a migration leaves the worker exiting with code 3. To go back, restore the backup and install the version that wrote it.

Backups

With the bundled ClickHouse, all durable state lives in the volume data-apgar-clickhouse-0. Use ClickHouse BACKUP to object storage you control, or the volume snapshots of your storage provider. With an external ClickHouse, back up the apgar database there.

Argo CD and Flux

The chart uses no Helm hooks and generates no values, so helm template renders the same resources as helm install. Argo CD and Flux apply it like any other chart. Keep the secrets in existingSecret and manage that Secret with your secret tooling.

Uninstall

helm uninstall apgar -n apgar

The ClickHouse volume and the Secret you created stay. Delete them to remove the data:

kubectl -n apgar delete pvc data-apgar-clickhouse-0
kubectl -n apgar delete secret apgar

Values

Value	Default	Description
`publicUrl`	none, required	Address people and exporters use
`existingSecret`	empty	Secret with `APGAR_SECRET_KEY`, `APGAR_ADMIN_TOKEN`, and `APGAR_CLICKHOUSE_PASSWORD`
`secret.secretKey`, `secret.adminToken`, `secret.clickhousePassword`	empty	Used only without `existingSecret`. The chart creates the Secret from them
`clickhouse.enabled`	`true`	Run one ClickHouse node in the release
`clickhouse.persistence.size`	`50Gi`	Size of the ClickHouse volume
`clickhouse.external.url`	empty	HTTP interface of an external ClickHouse
`clickhouse.external.caSecret`	empty	Secret with `ca.crt` for a private CA
`ingress.enabled`, `httpRoute.enabled`	`false`	Expose the UI. See Expose Apgar
`api.replicas`	`1`	API pods. `api.autoscaling.enabled` adds a HorizontalPodAutoscaler instead
`api.trustedProxyHops`	1, or 2 with an ingress or route	See Proxies and the login rate limit
`worker.resources`	see Requirements	The CPU limit sets the number of evaluator processes
`env`	empty	Extra `APGAR_` variables, for example `APGAR_TELEMETRY: "0"`. See Configuration
`metrics.serviceMonitor.enabled`	`false`	Scrape the API and the worker with the Prometheus Operator
`networkPolicy.enabled`	`false`	Allow ClickHouse only from Apgar, and the worker metrics only from `networkPolicy.metricsFrom`

helm show values oci://ghcr.io/oimo-ai/charts/apgar prints every value with its comment.

Get started

Send traces

Concepts

Notifications

Self hosting

Data and privacy

Requirements

Install

Expose Apgar

Ingress

Gateway API

Exporters in the cluster

External ClickHouse

When the worker does not start

Upgrade

Backups

Argo CD and Flux

Uninstall

Values

​Requirements

​Install

​Expose Apgar

​Ingress

​Gateway API

​Exporters in the cluster

​Proxies and the login rate limit

​External ClickHouse

​When the worker does not start

​Upgrade

​Backups

​Argo CD and Flux

​Uninstall

​Values

Requirements

Install

Expose Apgar

Ingress

Gateway API

Exporters in the cluster

Proxies and the login rate limit

External ClickHouse

When the worker does not start

Upgrade

Backups

Argo CD and Flux

Uninstall

Values