Nicholas Thomson, Jane Wang, and Jonathan Morin discuss the challenges of managing streaming data pipelines that use technologies like Kafka and RabbitMQ. The authors argue that SREs and application developers often struggle to determine if these pipelines are performing as expected due to a lack of visibility into every component of the pipeline. Datadog Data Streams Monitoring (DSM) helps address this issue by providing end-to-end latency, throughput, and consumer lag metrics for streaming data pipelines and event-driven applications. DSM enables users to monitor latency on services, pinpoint faulty producers, consumers, or queues driving latency and lag, discover hard-to-debug pipeline issues, root-cause and remediate bottlenecks, and quickly see who owns a pipeline component for immediate resolution. The tool automatically maps the architecture of your entire streaming data pipeline, including visualization of interdependencies, service ownership, and key health metrics across services and infrastructure dependencies. By using DSM, users can monitor and alert on end-to-end latency, pinpoint the root causes of bottlenecks in pipelines, identify and remediate floods of backed-up messages, enhance existing troubleshooting workflows, and take advantage of deep visibility into their streaming data pipelines.