Tracing a Runaway LLM Token Spike From Session to Trace to RUM

An organization implemented LLM observability within its product to manage rising costs associated with agent sessions by conducting thorough investigations into expensive turns without relying on multi-team communication. This approach involves analyzing three key signals: the LLM session, the distributed trace, and the RUM session, all linked by a shared session ID. By examining a costly session, the organization discovered that an agent was caught in a loop due to prompt caching being off, causing it to resend an entire growing context repeatedly, leading to high costs. The process, which integrates OpenObserve and RUM SDK, allows for efficient identification of the cost, cause, and user action behind such spikes. Governance teams can leverage this information to enforce prompt caching, optimize context handling, and attribute costs accurately, transforming potential mysteries into quick, accountable investigations.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.