Skip to main content
Each section starts with what you see and lists the causes in the order they are worth checking. The agent page, Settings, Notifications, and the Prometheus metrics are the places to look.

​
No turns on the agent page

  • The exporter does not reach Apgar. Check the exporter logs for the status of POST /v1/traces, and apgar_ingest_requests_total by status. 401 is a missing, mistyped, or revoked token. 415 means the exporter sends gRPC or an unexpected content type; set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf.
  • Spans arrived, but no turn is assembled yet. A trace becomes a turn 60 seconds after its last span once its span without a parent has arrived, and otherwise 10 minutes after its first span arrived. When the parent of your top span belongs to a service that does not send to Apgar, every turn waits the 10 minutes. apgar_assembly_lag_seconds shows how far assembly is behind.
  • The trace belongs to another agent, or to none. A token bound to an agent sends every trace to that agent, whatever the trace names. apgar_turns_dropped_total with reason agent_not_found means the bound agent was deleted, or an unbound token with Create agents automatically off received a trace that names an unknown agent; with reason no_agent, the trace carries no agent name and the token is unbound. See Agent name.
  • Spans were dropped at ingest. apgar_ingest_spans_dropped_total with reason invalid_ids or invalid_timestamps means the SDK sent spans without a valid trace id and span id (all-zero ids are invalid), with a malformed parent span id, or without a start time.
  • The spans are older than 7 days (APGAR_LATE_SPAN_DAYS). They are stored as late, never create releases, and are never evaluated. A backfill of more recent traces creates releases and sends messages like live traffic.
  • Langfuse: the sync has not run. The worker syncs each connection every minute. Only the first successful cycle of a connection reads the last 24 hours; an agent added to a connection that has already synced sees traces from about 30 minutes before it was added on. The agent page shows the sync status, and apgar_langfuse_requests_total by status shows whether Langfuse answers. A 404 on every request means a self hosted v3 server.

​
Turns arrive, but no release appears

The agent page says “No releases yet” where the release list would be. Without any turn in the last 24 hours it would show the Getting started steps instead.
  • The environment does not match. Only turns whose environment equals the agent’s environment create releases and count in comparisons. The environment comes from apgar.environment, deployment.environment.name, deployment.environment, or langfuse.environment, and is default when none is set. Add agent fills in production. Compare the agent environment with the environment on the spans, or clear it to accept every environment. The agent page shows no warning for this.
  • The turns are untagged. See the next section.

​
Turns are untagged

  • The release attribute is missing on the spans. Add it to OTEL_RESOURCE_ATTRIBUTES, or set it on the turn’s root span, the outermost invoke_agent or invoke_workflow span. A release on a span above that root is not read.
  • The release is set only on the root span of a long run. A turn whose root span arrives more than 10 minutes after its first span is assembled without the root and keeps the release it had then. Set the release on the resource instead.
  • The agent reads a different attribute. Agent settings, Release attribute names the only attribute that is read; an agent without one reads apgar.release, gen_ai.agent.version, then langfuse.release. Spans stored before a custom attribute was chosen do not carry it.
  • Langfuse: the trace has no release. Set the release on the trace in the SDK.
  • Langfuse: the agent reads apgar.release, gen_ai.agent.version, or service.version. Langfuse traces do not carry them; use langfuse.release or langfuse.trace.metadata.<key>.

​
The release stays in Collecting

  • The release is below the early floor of 20 turns and 5 sessions, or below the minimum sample (Minimum turns and Minimum sessions) with no failing guardrail. The release page shows the counts.
  • The baseline has fewer turns or sessions than the minimum sample. The previous release may have been short lived, or its traffic is older than the 7 day baseline window.
  • Most sessions are mixed: they started on the previous release. Mixed turns leave both sides. See Mixed sessions.
  • No turns arrived in the last 24 hours, so the release is not checked again until its window ends.

​
The release stays in Not enough data

  • The intervals are wider than the tolerances. The release page states the smallest change the sample can detect. More traffic narrows the intervals; so does a larger tolerance.
  • A guardrail has a previous value of 0 and a relative tolerance, so its change is undefined. Make that metric informational or turn it off. See When a metric is unavailable.
  • The agent has no applicable guardrail, for example an agent without tools whose remaining guardrails lack coverage. Such a release never reaches Healthy.

​
A metric is unavailable

The release page says what is missing. See When a metric is unavailable. The usual fixes:
  • Retry rate or Loop rate: tool spans lack arguments. Set gen_ai.tool.call.arguments on tool spans.
  • Cost per turn: chat spans lack a cost. Set apgar.cost_usd on each chat span, or let Langfuse price the model.
  • Format failure rate, or Output length without token counts: output content is not captured, or the Langfuse connection has fetch_content off.
  • Turns per session: fewer than 10 closed sessions, or most sessions are inferred because no session id is sent. See Sessions.

​
No Slack message

  • The agent has no Slack connection. Healthy also needs Send the healthy message on in Agent settings, Verdict rules, and a release replaced by a newer one sends no Unconfirmed message.
  • Delivery failed. Settings, Notifications lists every delivery with its last error. Slack must answer 200.
  • Links point at localhost: set APGAR_PUBLIC_URL.

​
Webhook signatures do not verify

  • Compute the signature over the raw body bytes. Serializing the JSON again changes them.
  • Sign X-Apgar-Timestamp, a period, and the body. sent_at inside the body is the queue time and differs from the header on retries.
  • Use the secret stored on the connection. The connection test signs with the same one, so it is the quickest check.

​
A process exits at startup

On Kubernetes, the worker applies the migrations in an init container. See When the worker does not start.