Company
Date Published
Author
Callum Styan
Word count
563
Language
English
Hacker News points
None

Summary

Prometheus 2.8 introduced a significant enhancement with Write-Ahead Logging (WAL) for the remote_write API, aimed at improving data handling during network disruptions by preventing data loss and excessive memory usage. Previously, the remote_write API would buffer data in memory, leading to potential memory overflow or data loss if the remote endpoint was unreachable. The new implementation writes transactions to a WAL, halting data reading until the successful transmission of data, thus offering virtually unlimited buffering contingent on disk size. This approach also optimizes the system by encoding data once and attempting resends until successful, reducing CPU and memory demands. Although the project faced challenges like code assumptions around locking and handling potential WAL corruptions, the update has been positively received, with reports of decreased CPU usage and stabilized memory usage. Feedback is encouraged through GitHub to address any emerging issues, particularly those concerning WAL corruption.