Introducing Ray History Server: Post-Mortem Observability for Ray on Kubernetes

Ray History Server, promoted to beta in KubeRay v1.7, provides post-mortem observability for ephemeral Ray clusters on Kubernetes by preserving and reconstructing Ray Dashboard data after clusters terminate. It offers a centralized interface for discovering, filtering, and opening both live and historical sessions, proxying live dashboards while rebuilding terminated ones from logs, structured events, and endpoint snapshots stored in object storage. A collector sidecar on Ray head and worker pods records telemetry using disk-first event collection, log uploads, metadata capture, and safeguards for disk pressure, while the stateless history server lazily retrieves and decompresses session data and uses an LRU cache to limit memory consumption. The storage-agnostic design supports Google Cloud Storage, Amazon S3 and MinIO, Alibaba Cloud OSS, and Azure Blob Storage, with deterministic paths that organize data by cluster and workload ownership. Benchmarks indicate that gzip compression and at least 1.5 CPU cores can reduce event storage by about 91%, speed cold session loads by roughly 2.4 times, and allow sessions of up to 50,000 tasks to open within a practical UI time budget. Planned improvements include automated collector injection through the KubeRay operator and replayable historical metrics such as CPU, GPU, memory, and actor-throughput data.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.