A single user message can cross a Go API, a Temporal workflow, a Python AI service, and several model providers. When each layer logs on its own, finding where a reply disappeared takes too long.
OpenTelemetry connects those steps. Trace context follows HTTP and gRPC calls. Temporal tracing interceptors carry it across workflow and activity boundaries.
A trace ID helps once you have it. Support usually starts with a user or conversation ID. Selected business identifiers travel in OpenTelemetry baggage and get copied into span attributes. That makes it possible to start from the information in a support ticket instead of a raw trace ID.
Go and Python should share the same allowed keys. Limit values to 128 characters and exclude secrets and request bodies. Otherwise, metadata grows on every downstream call.
Give each signal a job
| Signal | Role |
|---|---|
| Traces | Follow a request and inspect tool calls and model responses. |
| Metrics | Track request rates, failures, latency, and worker queues. |
| Error tracking | Group unexpected failures that need investigation. |
| Profiles | Find the functions consuming CPU when requests get slower. |
User and conversation IDs belong in traces and logs. Metric labels stay bounded: route pattern, status, provider, or tool type. A label for every user would create too many time series.
For each model call, record the provider, model, latency, token usage, and estimated cost. Also record fallback use and whether response checks passed, changed, or blocked the output. A successful HTTP response alone tells you little about the assistant's behavior.
Keep the useful data
Filter routine spans that add volume without helping diagnosis. Expected business errors should not flood error tracking. Alerts need a concrete action and a runbook.
For Temporal, watch how long activities wait for a worker, along with retries and failures. That separates a slow dependency from a queue that has no worker capacity.
The highest-leverage piece is the shared context contract. Every tool becomes more useful once the same request is easy to find in each of them.