Skip to main content

​
Overview

Upsonic instruments every agent run with OpenTelemetry spans following the GenAI semantic conventions. Every LLM call, pipeline step, tool execution, and agent run is traced automatically — giving you complete visibility into what your agent did, how long it took, and how much it cost.
Install the optional dependencies:
pip install upsonic[otel]
# or
uv sync --extra otel

​
What You Get

When tracing is enabled, each agent run produces a span hierarchy: 
agent.run (SERVER)
  └─ pipeline.execute (INTERNAL)
       ├─ pipeline.step.ModelExecutionStep (INTERNAL)
       │    └─ chat (CLIENT) — LLM call with GenAI attributes
       ├─ pipeline.step.ToolExecutionStep (INTERNAL)
       │    └─ tool.execute (INTERNAL) — per tool call
       └─ pipeline.step.MemorySaveStep (INTERNAL)
Attributes on spans include:
  • gen_ai.request.model, gen_ai.response.model — model used
  • gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — token counts
  • gen_ai.request.temperature, gen_ai.request.max_tokens — request parameters
  • gen_ai.response.id, gen_ai.response.finish_reasons — response metadata
  • upsonic.total_cost, upsonic.execution_time — cost and duration
  • upsonic.model_execution_time, upsonic.framework_overhead_time — timing breakdown
  • upsonic.time_to_first_token — streaming latency
  • upsonic.tool_call_count — number of tool calls
  • upsonic.tool.name, upsonic.tool.execution_time, upsonic.tool.success — per-tool details
  • upsonic.input, upsonic.output — task input/output
  • upsonic.run_id, upsonic.status — run identification and status
  • upsonic.agent.name, upsonic.agent.model — agent metadata
  • upsonic.usage.total_tokens, upsonic.usage.cache_read_tokens, upsonic.usage.cache_write_tokens — detailed token usage
  • upsonic.pipeline.total_steps, upsonic.pipeline.executed_steps — pipeline progress
  • upsonic.step.name, upsonic.step.status, upsonic.step.execution_time — per-step details
  • user.id, session.id — user and session tracking

​
Usage

​
Environment Variable — Zero Code Change

Set UPSONIC_OTEL_ENABLED=true and every Agent is automatically instrumented — no code changes needed. 
export UPSONIC_OTEL_ENABLED=true
export UPSONIC_OTEL_ENDPOINT=http://localhost:4317

from upsonic import Agent

# Automatically instrumented — no instrument= parameter needed
agent = Agent("anthropic/claude-sonnet-4-6")
agent.print_do("What is 2 + 2?")
When Upsonic is imported, it detects UPSONIC_OTEL_ENABLED and calls Agent.instrument_all() with a DefaultTracingProvider under the hood. All agents created in the process are instrumented automatically.
UPSONIC_OTEL_ENDPOINT must point to a running OTLP collector (Jaeger, Grafana Tempo, etc.). If no collector is reachable, you’ll see a ConnectionError in stderr at process exit when the exporter tries to flush spans. This is harmless — the agent runs normally, spans are simply dropped. See Troubleshooting below.

​
Quick Start — instrument=True

Creates a DefaultTracingProvider from environment variables automatically. 
from upsonic import Agent

agent = Agent("anthropic/claude-sonnet-4-6", instrument=True)
agent.print_do("What is 2 + 2?")

​
DefaultTracingProvider — Any OTLP Backend

Send traces to Jaeger, Grafana Tempo, Datadog, or any OTLP-compatible collector. 
from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")
agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)
agent.print_do("Summarize this report.")

​
DefaultTracingProvider — Full Configuration

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(
    endpoint="http://localhost:4317",
    service_name="payment-agent",
    sample_rate=0.25,
    include_content=False,
)

agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)
agent.print_do("What is 2 + 2?")

​
Global Instrumentation — Agent.instrument_all()

Instruments every Agent created after the call, without passing instrument each time. 
from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")
Agent.instrument_all(provider)

agent1 = Agent("anthropic/claude-sonnet-4-6")   # instrumented
agent2 = Agent("anthropic/claude-sonnet-4-6")   # also instrumented

agent1.print_do("What is 2 + 2?")
agent2.print_do("What is 3 + 3?")

Agent.instrument_all(False)        # disable

​
Session & User Tracking

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")

agent = Agent(
    "anthropic/claude-sonnet-4-6",
    instrument=provider,
    session_id="conversation-123",
    user_id="user@example.com",
)
agent.print_do("What is 2 + 2?")

​
Streaming

Tracing works identically with streaming — no extra setup. 
from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")
agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)

for chunk in agent.stream("Count to 5", events=False):
    print(chunk, end="")

​
Hide Sensitive Content

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317", include_content=False)
agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)
agent.print_do("Process SSN: 123-45-6789")
# Prompts and responses will NOT appear in traces

​
Advanced — Raw InstrumentationSettings

For testing or custom setups, pass InstrumentationSettings directly. 
from upsonic import Agent
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from upsonic.models.instrumented import InstrumentationSettings

exporter = InMemorySpanExporter()
tp = TracerProvider()
tp.add_span_processor(SimpleSpanProcessor(exporter))

settings = InstrumentationSettings(
    tracer_provider=tp,
    meter_provider=MeterProvider(),
)

agent = Agent("anthropic/claude-sonnet-4-6", instrument=settings)
agent.print_do("Hello!")

for span in exporter.get_finished_spans():
    print(span.name, span.attributes)

​
Environment Variables

Env VarDescriptionDefault
UPSONIC_OTEL_ENABLEDSet true to auto-instrument all agents on import—
UPSONIC_OTEL_ENDPOINTOTLP collector endpointhttp://localhost:4317
UPSONIC_OTEL_SERVICE_NAMEservice.name resource attributeupsonic
UPSONIC_OTEL_SAMPLE_RATESampling rate (0.0–1.0)1.0
UPSONIC_OTEL_HEADERSComma-separated key=value headers—
Setting UPSONIC_OTEL_ENABLED=true triggers automatic OpenTelemetry setup on import and calls Agent.instrument_all().

​
Lifecycle

All providers set flush_on_exit=True by default — pending spans are flushed automatically when the process exits. You do not need to call shutdown() manually in normal usage. Call shutdown() explicitly only if you need to flush spans mid-process (e.g., in tests or before a graceful restart).

​
Troubleshooting

​
ConnectionError at process exit

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4317):
Max retries exceeded ... Connection refused
This is harmless. It means the OTLP exporter tried to send spans but no collector is running at the configured endpoint. Your agent executed normally — only the trace export failed. To fix it, either:
  1. Run a collector — start Jaeger, Grafana Tempo, or any OTLP-compatible backend:
docker run -d --name jaeger -p 4317:4317 -p 16686:16686 jaegertracing/all-in-one:latest
  1. Use Langfuse instead — no local collector needed, traces go to the cloud:
from upsonic.integrations.langfuse import Langfuse
agent = Agent("anthropic/claude-sonnet-4-6", instrument=Langfuse())
  1. Don’t use tracing — simply don’t set instrument or UPSONIC_OTEL_ENABLED.

Langfuse Integration

Send traces to Langfuse for LLM-specific dashboards, cost tracking, and prompt management.

PromptLayer Integration

Log agent runs and evaluations to PromptLayer for prompt management, versioning, and observability.