> ## Documentation Index
> Fetch the complete documentation index at: https://docs.useapgar.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Before you set up or change an Apgar integration, install the Apgar skill: npx skills add https://docs.useapgar.dev
> Search these docs with the MCP server at https://docs.useapgar.dev/mcp. Setup for each agent: https://docs.useapgar.dev/ai-agents.md

# OpenTelemetry

> Send agent traces to Apgar over OTLP/HTTP.

Apgar accepts OpenTelemetry traces at `POST /v1/traces`. Any SDK or collector that exports OTLP over HTTP can send to it. Instrumentations that follow the OpenTelemetry GenAI semantic conventions or OpenInference work without changes.

## Create an ingest token

An ingest token has the form `apg_ing_` followed by 32 characters. Apgar shows it once and stores only its hash. There are two ways to get one:

* **Add agent** with the OTLP source creates a token named `AGENT (auto)` and bound to that agent. Every trace sent with it belongs to that agent, whatever its attributes say.
* **Settings, Ingest tokens** creates a token that you can bind to an agent or leave unbound. With an unbound token, Apgar resolves the agent from the [agent name attributes](https://docs.useapgar.dev/send-traces/attributes.md#agent-name), and the first trace with a new name creates the agent. Turn off **Create agents automatically** on the token to drop traces of unknown agents instead. An agent created this way has no environment and no **Release attribute**, so it reads the release from `apgar.release`, `gen_ai.agent.version`, or `langfuse.release`; set both in **Agent settings**. One unbound token can serve several agents.

Deleting a token revokes it. The API caches token lookups for 60 seconds, so a revoked token can still be accepted for up to a minute.

## Configure the exporter

Set these variables in the environment of the agent service. `ORIGIN` is the address of the Apgar UI, the value of `APGAR_PUBLIC_URL`. The UI forwards `/v1` to the API.

```bash theme={null}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=ORIGIN/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer apg_ing_...
OTEL_RESOURCE_ATTRIBUTES=gen_ai.agent.name=AGENT,deployment.environment.name=ENVIRONMENT,apgar.release=RELEASE
```

The snippet that **Add agent** shows contains the first three lines plus the agent name and environment. Add the release yourself: `RELEASE` is the identifier of the running build, a version, a tag, or a commit hash, and comes from your deploy pipeline. Put it in the attribute you chose as **Release attribute** on the agent; `apgar.release` is the usual choice. Without it every turn is stored as `untagged` and is never evaluated. See [Release](https://docs.useapgar.dev/send-traces/attributes.md#release).

## What a trace should look like

Apgar treats one trace as one turn of the agent. It works best when a trace has:

* A root span for the turn, such as an `invoke_agent` or `invoke_workflow` span, carrying the conversation id. See [Sessions](https://docs.useapgar.dev/concepts/sessions.md).
* One chat span per LLM call, with the model, token counts, and ideally a cost.
* One tool span per tool call, with the tool name and its arguments.

Span kinds are read from `gen_ai.operation.name`, `openinference.span.kind`, or `langfuse.observation.type`. Set `apgar.kind` to override the kind of a span. See [Span kinds](https://docs.useapgar.dev/send-traces/attributes.md#span-kinds).

> [!NOTE]
>
> Tool arguments are needed to detect retries and loops. Apgar never stores them: it keeps only a keyed hash. Content attributes are reduced to a length and a JSON validity flag and then dropped.

### Example with the Python SDK

The OTLP exporter reads the variables above. The application opens one root span per turn and puts the conversation id on it; the chat and tool spans come from the instrumentation of your LLM client or agent framework.

```bash theme={null}
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

```python theme={null}
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my-agent")


def handle_turn(conversation_id: str, message: str) -> str:
    with tracer.start_as_current_span(
        "invoke_agent my-agent",
        attributes={
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.conversation.id": conversation_id,
        },
    ):
        return run_agent(message)  # instrumented LLM and tool calls happen here
```

## Endpoint

| Property | Value |
| - | - |
| Path | `POST /v1/traces` |
| Authentication | `Authorization: Bearer apg_ing_...` |
| Body | OTLP protobuf (`application/x-protobuf`) or OTLP/JSON (`application/json`), optionally with `Content-Encoding: gzip` |
| Size limit | 32 MiB by default (`APGAR_MAX_BODY_BYTES`), enforced on both the received body and the decompressed body |

| Status | Meaning |
| - | - |
| `200` | The request was stored. Spans with a missing, all-zero, or wrong-length trace or span id, a parent span id of the wrong length, or no valid start time are dropped silently and counted in `apgar_ingest_spans_dropped_total`. An all-zero parent span id means no parent, so that span is a root. A request with no spans is also `200` |
| `400` | The body could not be decoded, including a broken gzip stream |
| `401` | Missing, unknown, or revoked ingest token |
| `413` | The received or decompressed body is larger than the limit. The received size is checked before the token, the decompressed size after it |
| `415` | Unsupported content type |
| `503` | Storage is unavailable, also for the token lookup. Retry after the `Retry-After` header |

Apgar accepts OTLP over HTTP only. An SDK that exports OTLP/gRPC switches to HTTP with `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf`.

## Send through an OpenTelemetry Collector

When traces already pass through a collector, add an exporter for Apgar next to the existing one. The application keeps exporting to the collector, and the collector sends a copy to Apgar. Each exporter has its own queue and retries, so an unavailable Apgar does not hold back your existing backend.

```yaml collector.yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch: {}
exporters:
  otlp_grpc/existing:
    endpoint: backend.example.com:4317
  otlp_http/apgar:
    endpoint: https://apgar.example.com
    headers: { Authorization: "Bearer ${env:APGAR_INGEST_TOKEN}" }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_grpc/existing, otlp_http/apgar]
```

In an existing collector only `otlp_http/apgar` and its entry in the traces pipeline are new. Set `endpoint` to `APGAR_PUBLIC_URL`; the exporter appends `/v1/traces`. Set `APGAR_INGEST_TOKEN` in the environment of the collector. Collector releases before the component rename know the exporters as `otlphttp` and `otlp`; current releases accept both spellings.

Validate the configuration before you deploy it:

```bash theme={null}
docker run --rm -e APGAR_INGEST_TOKEN=dummy -v "$PWD/collector.yaml:/c.yaml" otel/opentelemetry-collector-contrib:0.163.1 validate --config /c.yaml
```

> [!WARNING]
>
> Put the Apgar exporter in a pipeline without sampling. A tail sampling policy that keeps only errors changes the error, retry, and loop rates that Apgar compares, and sampling by trace removes turns from sessions.

## Check that traces arrive

Open the agent page. A trace becomes a turn about a minute after its last span arrives, and its release then appears in the release list, whose turn counts update every 5 minutes. When the parent of your top span belongs to a service that does not send to Apgar, a trace waits 10 minutes from its first span instead. The page warns when more than 5 % of turns have no release, which means the release attribute is missing on some traffic or the agent reads a different attribute. When nothing arrives, see [Troubleshooting](https://docs.useapgar.dev/troubleshooting.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.