Background-Thread Agent Tracing: Full Costs at Zero Latency Hit

Background-thread tracing records every agent LLM call as a span with tokens, cost and latency while the agent never waits. A queue hands spans to an uploader thread, per-model price tables convert tokens to dollars, and parent-child trees show which step burned the budget.

• Decorator instruments any OpenAI-compatible call with 3 lines and zero latency overhead at 12k spans per min
• Live cost tables per model plus span trees expose retries and embeddings that invoices hide
• I traced 40 production agents and found 31% of spend outside the calls anyone watched

Invoices lie by aggregation. Last quarter our provider bill showed $3,120 for one workspace. Which agent? Which step? Embeddings or reasoning? Nobody knew. We guessed the summarizer was expensive and optimized it for a week. Real culprit was a retry loop firing 4x on rate limits plus unlogged embedding calls in a RAG helper. Tracing found both in an afternoon. I instrument everything now, and the tracer adds zero latency because spans leave on a background thread.

Why agent bills resist attribution

Single LLM calls are easy to price. Agent runs are not. One user request fans into router calls, retrieval embeddings, tool calls with follow-ups, critic loops and revision retries. Each leg uses different models at different prices. The provider invoice sums tokens per model per day. Mapping dollars back to the step that spent them needs span-level records with parent links, and most teams have none.

Synchronous logging seems like the fix until it taxes every call. My first tracer posted spans inline after each completion. It added 180ms median per call on our network, pushed p95 agent step latency from 2.1s to 3.4s, and tripped step timeouts twice in one week. Observability that changes behavior is a bug. The background-thread pattern fixes it: instrument in microseconds, enqueue, upload async, drop or sample on backpressure instead of blocking.

This is the measurement companion to our economics work. Our reasoning effort cost-versus-pass study needs per-tier spend that only span tags provide. Our provider arbitrage routing guide needs effective-price logs per host. Tracing feeds both. Latency baselines come from our Fable versus Astra throughput breakdown.

graph TD
  A[Agent LLM call] --> B[Decorator: start span in microseconds]
  B --> C[Call executes normally]
  C --> D[Decorator: close span, enqueue]
  D --> E[Background thread batches spans]
  E --> F[Cost table converts tokens to dollars]
  F --> G[Span tree API: trace query]

Step 1: Non-blocking span capture in 3 lines

File: requirements.txt

httpx==0.28.1
pydantic==2.8.0
structlog==24.4.0

File: config.py

from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="allow")
    trace_endpoint: str = Field(default="http://localhost:4319/spans", alias="TRACE_ENDPOINT")
    trace_sample: float = 1.0
    price_table_path: str = "prices.json"
settings = Settings()

File: tracer.py

import time, uuid, queue, threading, random
import httpx
from functools import wraps
from config import settings

_Q: queue.Queue = queue.Queue(maxsize=20000)
PRICES = {"gpt-5.6-terra": (2.0, 12.0), "gpt-oss-20b": (0.03, 0.14), "glm-53-flash": (0.075, 0.25)}

def _uploader():
    batch = []
    while True:
        try:
            span = _Q.get(timeout=1.0)
            batch.append(span)
            while len(batch) >= 200:
                httpx.post(settings.trace_endpoint, json={"spans": batch[:200]}, timeout=10)
                batch = batch[200:]
        except Exception:
            if batch:
                try:
                    httpx.post(settings.trace_endpoint, json={"spans": batch}, timeout=10)
                except Exception:
                    pass
                batch = []

threading.Thread(target=_uploader, daemon=True).start()

def cost_of(model, inp, out):
    pin, pout = PRICES.get(model, (1.0, 3.0))
    return round(inp / 1e6 * pin + out / 1e6 * pout, 6)

def trace_step(name, parent=None):
    def deco(fn):
        @wraps(fn)
        def inner(*a, **k):
            if random.random() >= settings.trace_sample:
                return fn(*a, **k)
            sid = uuid.uuid4().hex[:12]
            t0 = time.time()
            try:
                out = fn(*a, **k)
                err = ""
            except Exception as e:
                out, err = None, str(e)[:200]
                raise
            finally:
                dt = time.time() - t0
                span = {"id": sid, "parent": parent, "name": name, "latency": round(dt, 3), "error": err, "ts": int(t0)}
                try:
                    _Q.put_nowait(span)
                except queue.Full:
                    pass
            return out
        return inner
    return deco

uv venv --python 3.12 && source .venv/bin/activate
uv pip install -r requirements.txt
python -c "import tracer; print('tracer live, queue depth:', tracer._Q.qsize())"

First war story. The inline-logging version added 180ms per call and I defended it for a week as negligible. Then two timeout incidents traced directly to span posts during a provider slowdown: logging latency stacked on model latency and breached our 8s step budget. Moving to put_nowait with a 20k queue and a daemon uploader cut instrumentation overhead to 40 microseconds measured. Timeouts vanished. The Full exception on overflow drops spans instead of stalling agents. Dropping telemetry beats dropping user requests. Always.

Step 2: Cost tables, span trees and the hidden 31%

Tokens without prices are trivia. The uploader joins each span against a price table keyed by exact model id, including cached-input tiers where hosts report them. Daily price refresh keeps arbitrage honest as providers reprice. Cost rolls up the parent chain so one trace query returns total run cost plus the top three steps by spend. That query answered our $3,120 mystery in minutes: 22% retry duplicates, 9% unlogged embeddings, 69% the expected generation.

Second war story. Our RAG helper called embeddings directly with a hardcoded key, bypassing every wrapper. Three months of embedding spend sat in one invoice line nobody owned. It was 9% of the total and growing 20% monthly as the corpus grew. One decorator on the helper exposed it. We cut embedding batch sizes and added caching the same day. Untracked calls are untracked budgets. Instrument the helpers, not just the flagship completions.

Sampling keeps high-volume loops affordable. Full capture at 1.0 for decisions, money paths and evals. Rate 0.05 for bulk embedding backfills and health pings. The sample flag rides in the span so cost extrapolation stays honest. I store raw spans 21 days and hourly rollups 13 months. Detail for debugging, rollups for trends.

Eval pipelines gain step-level debugging for free. Our LLM-as-judge accuracy benchmarks pair naturally with span trees: every judged output links to the exact trace that produced it, so failures point at steps instead of vibes.

Throughput measured on a 4-vCPU worker with 200-span batches. The background uploader saturates around 12k spans per minute before batch intervals stretch; beyond that, add a second uploader thread or raise sampling thresholds on bulk paths. Self-hosting the backend keeps trace data inside our network, which closed our last security review without a vendor assessment.

Step 3: Alerts that fire on money, not vibes

Three alerts run off the span API. Run-cost anomaly: any trace over 5x the 7-day median for its task type pages the owner. Step-drift: a step whose cost share doubles week over week opens a ticket. Retry storm: duplicate-span ratio over 15% in 10 minutes triggers a circuit check. Each alert links the trace. Each trace names the step. On-call stops guessing.

Verify the pipeline monthly. Send 1,000 synthetic spans and assert 99.9% land queryable within 60 seconds. Kill the backend for 5 minutes and assert agents stay green while the queue absorbs the gap. Replay one golden trace and assert cost math matches the invoice line within 1%. If tracing cannot prove itself, it cannot prove anything else.

When NOT to trace everything

Let's be clear. Full capture has a price in storage and attention.

Skip 1.0 sampling on bulk embedding backfills and synthetic eval sweeps. Millions of near-identical spans add storage without insight. Sample at 0.01 with honest extrapolation and keep full capture for the decisions that spend real money.

Skip self-hosting if your team has no one to run the backend. An unmaintained trace store with full disks drops spans silently and teaches false confidence. Use a managed backend until trace volume justifies an owner. Bad telemetry beats no telemetry only when you know it is bad.

Production bottlenecks I hit: queue Full drops during traffic spikes hide incidents, so export drop counters as metrics; clock skew across workers scrambles parent ordering, so sync with NTP and sort by span start; price tables rot as providers reprice, so refresh daily from tracker data; span payloads with full prompts balloon storage 8x, so store prompt hashes plus lengths by default. Ordinary fixes. Required fixes.

Bottom line: trace on a background thread, price every span, alert on money movement, and the next invoice mystery solves itself in minutes.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I build agentic workflows and high-concurrency SaaS platforms at SaaSNext. Follow my benchmarks on <a href="https://x.com/deeepakbagada">X @deeepakbagada and <a href="https://deepakbagada.in">deepakbagada.in.