Using Apache Kafka for Real-Time Event Processing at New Relic

New Relic, an early adopter of Apache Kafka, utilizes the platform to build scalable, high-throughput, real-time streaming systems that efficiently manage data streams through microservices. By leveraging Kafka's capabilities, New Relic's Events Pipeline team processes "event data"—fine-grained monitoring information—through a chain of services connected via Kafka topics, enabling asynchronous communication that decouples services and allows dynamic scaling based on traffic. Kafka serves as both a changelog and a "durable cache," facilitating state storage and retrieval, which aids in rebuilding application states without additional database dependencies. The system incorporates the LMAX disruptor pattern to handle high concurrency and throughput, ensuring efficient data processing across multiple handlers. This approach reduces complexity and maintains scalability as New Relic continues to evolve its data infrastructure, demonstrating the platform's flexibility and power in supporting distributed systems.