Datadog faced significant challenges in scaling its real-time data pipeline for its Processes and Containers products due to the increasing demand for live data views, which necessitated handling millions of data points per second. Originally, the system collected data from all hosts in a tenant's infrastructure, leading to inefficiencies and high resource usage. To address this, Datadog refined its approach, focusing on enabling real-time data collection only for actively viewed hosts, thus reducing data traffic by 100x and cutting infrastructure costs by 98%. This optimization not only enhanced the system's efficiency but also improved the user experience by decreasing latency and CPU usage. By adopting a more targeted data collection strategy and leveraging the standard 10-second data intervals for sorting, Datadog was able to maintain system performance while significantly reducing resource consumption. This approach demonstrated the importance of scaling systems based on user needs rather than sheer data volume, leading to substantial performance gains and cost savings.