Company
Date Published
Author
JF Joly
Word count
954
Language
English
Hacker News points
None

Summary

Apache Kafka is widely used by companies like New Relic, Uber, and Square for building scalable and reliable real-time streaming systems, yet its management can be complex. Amazon Managed Streaming for Apache Kafka (MSK) simplifies this by handling Kafka management and exposing metrics in a Prometheus-compatible format, which can be integrated with New Relic's Prometheus OpenMetrics for enhanced monitoring. This integration allows users to combine MSK metrics with other data sources in New Relic One, facilitating comprehensive observability across application stacks. The article details the setup process for an Amazon MSK cluster, configuration of the New Relic integration, and examples of using metrics for monitoring Kafka nodes, such as tracking filesystem usage and producer request rates, as well as setting alerts for storage thresholds. Written by JF Joly, a New Relic product manager, the blog emphasizes the environment-specific nature of the solutions provided and directs readers to the Explorers Hub for further discussion and support.