OpenTelemetry

Apgar accepts OpenTelemetry traces at POST /v1/traces. Any SDK or collector that exports OTLP over HTTP can send to it. Instrumentations that follow the OpenTelemetry GenAI semantic conventions or OpenInference work without changes.

Create an ingest token

An ingest token has the form apg_ing_ followed by 32 characters. Apgar shows it once and stores only its hash. There are two ways to get one:

Add agent with the OTLP source creates a token named AGENT (auto) and bound to that agent. Every trace sent with it belongs to that agent, whatever its attributes say.
Settings, Ingest tokens creates a token that you can bind to an agent or leave unbound. With an unbound token, Apgar resolves the agent from the agent name attributes, and the first trace with a new name creates the agent. Turn off Create agents automatically on the token to drop traces of unknown agents instead. An agent created this way has no environment and no Release attribute, so it reads the release from apgar.release, gen_ai.agent.version, or langfuse.release; set both in Agent settings. One unbound token can serve several agents.

Deleting a token revokes it. The API caches token lookups for 60 seconds, so a revoked token can still be accepted for up to a minute.

Configure the exporter

Set these variables in the environment of the agent service. ORIGIN is the address of the Apgar UI, the value of APGAR_PUBLIC_URL. The UI forwards /v1 to the API.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=ORIGIN/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer apg_ing_...
OTEL_RESOURCE_ATTRIBUTES=gen_ai.agent.name=AGENT,deployment.environment.name=ENVIRONMENT,apgar.release=RELEASE

The snippet that Add agent shows contains the first three lines plus the agent name and environment. Add the release yourself: RELEASE is the identifier of the running build, a version, a tag, or a commit hash, and comes from your deploy pipeline. Put it in the attribute you chose as Release attribute on the agent; apgar.release is the usual choice. Without it every turn is stored as untagged and is never evaluated. See Release.

What a trace should look like

Apgar treats one trace as one turn of the agent. It works best when a trace has:

A root span for the turn, such as an invoke_agent or invoke_workflow span, carrying the conversation id. See Sessions.
One chat span per LLM call, with the model, token counts, and ideally a cost.
One tool span per tool call, with the tool name and its arguments.

Span kinds are read from gen_ai.operation.name, openinference.span.kind, or langfuse.observation.type. Set apgar.kind to override the kind of a span. See Span kinds.

Tool arguments are needed to detect retries and loops. Apgar never stores them: it keeps only a keyed hash. Content attributes are reduced to a length and a JSON validity flag and then dropped.

Example with the Python SDK

The OTLP exporter reads the variables above. The application opens one root span per turn and puts the conversation id on it; the chat and tool spans come from the instrumentation of your LLM client or agent framework.

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my-agent")


def handle_turn(conversation_id: str, message: str) -> str:
    with tracer.start_as_current_span(
        "invoke_agent my-agent",
        attributes={
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.conversation.id": conversation_id,
        },
    ):
        return run_agent(message)  # instrumented LLM and tool calls happen here

Endpoint

Property	Value
Path	`POST /v1/traces`
Authentication	`Authorization: Bearer apg_ing_...`
Body	OTLP protobuf (`application/x-protobuf`) or OTLP/JSON (`application/json`), optionally with `Content-Encoding: gzip`
Size limit	32 MiB by default (`APGAR_MAX_BODY_BYTES`), enforced on both the received body and the decompressed body

Status	Meaning
`200`	The request was stored. Spans with a missing, all-zero, or wrong-length trace or span id, a parent span id of the wrong length, or no valid start time are dropped silently and counted in `apgar_ingest_spans_dropped_total`. An all-zero parent span id means no parent, so that span is a root. A request with no spans is also `200`
`400`	The body could not be decoded, including a broken gzip stream
`401`	Missing, unknown, or revoked ingest token
`413`	The received or decompressed body is larger than the limit. The received size is checked before the token, the decompressed size after it
`415`	Unsupported content type
`503`	Storage is unavailable, also for the token lookup. Retry after the `Retry-After` header

Apgar accepts OTLP over HTTP only. An SDK that exports OTLP/gRPC switches to HTTP with OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf.

Send through an OpenTelemetry Collector

When traces already pass through a collector, add an exporter for Apgar next to the existing one. The application keeps exporting to the collector, and the collector sends a copy to Apgar. Each exporter has its own queue and retries, so an unavailable Apgar does not hold back your existing backend.

collector.yaml

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch: {}
exporters:
  otlp_grpc/existing:
    endpoint: backend.example.com:4317
  otlp_http/apgar:
    endpoint: https://apgar.example.com
    headers: { Authorization: "Bearer ${env:APGAR_INGEST_TOKEN}" }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_grpc/existing, otlp_http/apgar]

In an existing collector only otlp_http/apgar and its entry in the traces pipeline are new. Set endpoint to APGAR_PUBLIC_URL; the exporter appends /v1/traces. Set APGAR_INGEST_TOKEN in the environment of the collector. Collector releases before the component rename know the exporters as otlphttp and otlp; current releases accept both spellings. Validate the configuration before you deploy it:

docker run --rm -e APGAR_INGEST_TOKEN=dummy -v "$PWD/collector.yaml:/c.yaml" otel/opentelemetry-collector-contrib:0.163.1 validate --config /c.yaml

Put the Apgar exporter in a pipeline without sampling. A tail sampling policy that keeps only errors changes the error, retry, and loop rates that Apgar compares, and sampling by trace removes turns from sessions.

Check that traces arrive

Open the agent page. A trace becomes a turn about a minute after its last span arrives, and its release then appears in the release list, whose turn counts update every 5 minutes. When the parent of your top span belongs to a service that does not send to Apgar, a trace waits 10 minutes from its first span instead. The page warns when more than 5 % of turns have no release, which means the release attribute is missing on some traffic or the agent reads a different attribute. When nothing arrives, see Troubleshooting.

Get started

Send traces

Concepts

Notifications

Self hosting

Data and privacy

OpenTelemetry

Create an ingest token

Configure the exporter

What a trace should look like

Example with the Python SDK

Endpoint

Send through an OpenTelemetry Collector

Check that traces arrive

​Create an ingest token

​Configure the exporter

​What a trace should look like

​Example with the Python SDK

​Endpoint

​Send through an OpenTelemetry Collector

​Check that traces arrive

Create an ingest token

Configure the exporter

What a trace should look like

Example with the Python SDK

Endpoint

Send through an OpenTelemetry Collector

Check that traces arrive