December 2025 Summaries

Polar Signals faced significant cross-zone traffic costs on Google Cloud Platform due to the way Thanos handled Service Level Objective (SLO) monitoring with Pyrra, leading to substantial data transfer expenses. The root cause was the evaluation process of SLOs that required fetching millions of raw samples across zones for each evaluation, significantly increasing network bandwidth consumption and CPU usage. To address this, Polar Signals implemented an optimization technique using Prometheus subqueries to pre-aggregate data into manageable chunks, which dramatically reduced the data volume transferred across zones by processing fewer pre-aggregated values instead of raw samples. This approach resulted in a 90% reduction in cross-zone traffic and improved query performance, although it introduced a minor 1% accuracy discrepancy. The trade-off between accuracy and performance was managed by offering this optimization as an opt-in feature, allowing users to balance resource efficiency with precision. The new method is currently being tested in production, with plans to integrate it into Pyrra for broader community use, emphasizing that performance should not hinder comprehensive observability.

The open-source NVIDIA CUDA profiler facilitates continuous production monitoring by allowing users to inject a shared library into application processes without altering the build process, utilizing Kubernetes init containers and a shared volume. This method enables CUDA workloads, such as PyTorch training jobs, to load the profiler library seamlessly, capturing every kernel launch, memory transfer, and synchronization event. By using an init container to copy the library to a shared volume and setting the CUDA_INJECTION64_PATH environment variable, the library is made available to the main container, allowing for transparent profiling of any CUDA application, including TensorFlow, JAX, or custom C++ code. Users can verify successful implementation through application logs or by checking the environment directly, and the profiler provides valuable insights into CUDA function execution times, aiding in optimization decisions like batch sizing and operator fusion. Future blog posts will explore more detailed use cases and upcoming features, while further resources and community support are available through Discord and other documentation.

User-level statically defined tracepoints (USDTs) offer a robust solution to the challenges posed by uprobes in tracing software, providing a stable, discoverable, and efficient method for extracting program information without the fragility of traditional hacks or the overhead of uprobes. At Polar Signals, USDTs are utilized for GPU profiling in CUDA, exemplifying their potential for deep visibility into operations with minimal system impact. These tracepoints allow binaries to declare where key data can be found, effectively bypassing the complexity of uprobes by providing a transparent interface for tracing applications. By integrating USDTs, developers can ensure their software is natively observable, reducing the need for unreliable offset extraction or disassembly and benefiting from modern Linux kernel supports like multiprobes and attach cookies. As USDTs are underutilized despite their capabilities, the call to action is for developers to embed them into their applications, frameworks, and libraries to enhance observability and debugging efficiency. Polar Signals invites further exploration and adoption of USDTs to facilitate seamless system profiling and observability.

The text discusses the modifications made to Delta Lake, an open-source table format originally developed by Databricks, which was integrated into a new database system referred to as Great Lakes. The original Delta Lake provides ACID transactions through multiversion concurrency control (MVCC) and is optimized for cloud-native databases performing stream processing. Modifications include replacing the standard Parquet file format with Vortex to reduce query times and memory use, transitioning from JSON to Vortex format for commit logs to save CPU time and memory, and introducing formal Vortex struct columns to eliminate the need for parsing JSON in query paths. Additionally, a centralized commit coordinator was implemented to avoid write contention, and virtual files were added to the log for atomic data compaction, allowing efficient data movement between tables without duplication risks. These enhancements have reportedly improved performance and efficiency, with aspirations to contribute these advancements back to the open-source community.