June 2019 Summaries

The Kubernetes Control Plane is a critical component of the cluster, responsible for managing the worker nodes and ensuring the desired state of the cluster. Datadog has introduced new integrations to monitor the Control Plane, providing visibility into its performance, including real-time data on workload and latency. The integrations cover the API Server, Controller Manager, Scheduler, and etcd, offering metrics such as request rates, goroutines, threads, and proposal counts. Monitoring these components enables rapid troubleshooting of scheduling and orchestration issues and provides insights into cluster-wide performance. The new integrations are available with Datadog's Agent version 6.12 and can be configured to collect metrics from the Control Plane.

Datadog operates over 40 Kafka and ZooKeeper clusters that process trillions of datapoints daily across multiple platforms, data centers, and regions. The company has learned valuable lessons from scaling these clusters to support diverse workloads. They share insights on coordinating changes to maximum message size, unclean leader elections, investigating data reprocessing issues on low-throughput topics, and why low-traffic topics can retain data longer than expected. Monitoring certain metrics helps ensure the durability of data and availability of clusters.

Kafka stores data across partitions in each topic, and each partition has a leader and zero or more followers that fetch and replicate new data from the leader. Unclean leader elections can lead to data loss if not managed properly. Coordinating changes to maximum message size is crucial for smooth messaging pipeline operation. Monitoring system load and other host-level resource metrics can help detect issues with certain brokers processing messages. Investigating data reprocessing issues on low-throughput topics requires adjusting the consumer offset retention period. Kafka's approach to segment-level retention can cause unexpected results if not properly configured, especially in low-throughput topics. The default configuration settings are designed for high-traffic topics, so it's essential to test and adjust settings according to specific use cases.

Datadog introduces an enhanced full-screen view for its timeseries graphs to facilitate deeper exploration of the data. Users can now apply advanced functions such as anomaly detection, trend lines, and outlier detection without creating new graphs or cloning existing ones. The mini-map in full-screen mode allows users to visualize trends over longer timeframes, while the ability to save changes or create new graphs enables easy customization of visualizations. Existing Datadog customers can access this feature on their dashboards, and new users can sign up for a 14-day free trial.

Presto is an open source SQL query engine used for running analytics on large datasets from various sources such as Hadoop and Cassandra. It was initially developed by Facebook to analyze its Apache Hadoop data warehouse and has since been adopted by companies like Airbnb, Uber, and Netflix. Datadog's new integration with Presto provides comprehensive visibility into query performance and resource usage alongside the rest of a distributed architecture. This integration allows users to investigate and alert on Presto query performance issues, determine when and why queries fail with alerts and logs, and monitor Presto alongside other infrastructure components.

Miranda Kapin from Datadog has introduced a revamped full-screen view for their timeseries graphs, allowing users to explore data more deeply, apply advanced functions without building new graphs, and visualize trends over longer time periods. This feature is accessible by double-clicking on the graph's header or clicking the full-screen button in the upper right-hand side of the graph. Users can now quickly save and share their work and see how their data fits into larger trends, making it easier to create new visualizations that are easy to save and modify later.

Presto is an open source SQL query engine that runs analytics on large datasets queried from a range of sources, including Hadoop and Cassandra. It was originally developed by Facebook to run queries on its large Apache Hadoop data warehouse and is now used as an interactive analytics tool at companies like Airbnb, Uber, and Netflix. Datadog's new integration provides comprehensive visibility into Presto query performance and resource usage alongside the rest of your distributed architecture. With this integration, teams can investigate bottlenecks, determine when queries fail with alerts and logs, and monitor Presto alongside other infrastructure components. The integration also collects and processes Presto logs to provide granular details about query engine activity, allowing for more accurate troubleshooting and optimization of performance issues.

IBM DB2 is a database management system that supports various platforms such as Linux, UNIX, Windows, mainframes, and IBM Power Systems. It can be deployed in the cloud or clustered for high availability, making it suitable for numerous enterprise applications. Datadog now integrates with DB2 to monitor its health and performance alongside related applications and infrastructure. The integration includes a built-in dashboard displaying key metrics about DB2 instances' availability, connections, query rates, and more. Users can optimize buffer pools and improve query efficiency by adding indexes. Additionally, DB2 logs can be collected and analyzed for deeper insights into performance issues.

The text discusses the integration of IBM DB2 with Datadog, a monitoring and analytics platform. The integration allows users to monitor the health and performance of their DB2 instances alongside related applications and infrastructure. It provides a built-in dashboard that displays key metrics about instance availability, connections, query rates, and more. Additionally, it offers features such as caching query results in buffer pools, optimizing queries with indexes, bringing DB2 logs into Datadog for deeper insights, and starting to monitor DB2 performance. These features enable users to troubleshoot issues, optimize their infrastructure, and gain real-time visibility into DB2's performance.

Java Virtual Machine (JVM) runtime metrics are crucial for troubleshooting application performance issues. Datadog APM integrates these metrics with request traces to provide comprehensive visibility across the Java stack, from code-level performance to JVM health. By correlating JVM metrics with spans, developers can determine if resource constraints or excess load in the runtime environment impacted application performance. The new integration dashboard provides real-time insights into garbage collection, memory usage, and thread count, allowing for deeper investigation of changes in JVM metrics. Datadog's Java client automatically collects these metrics, enabling unified insights into Java applications and JVM runtime metrics in one platform.

Datadog APM provides detailed context for troubleshooting application performance issues with Java Virtual Machine (JVM) runtime metrics. These metrics are integrated into Datadog's APM, allowing users to correlate request traces with JVM metrics to determine if the bottleneck is the JVM or a code-level issue. The platform also collects runtime metrics locally from each JVM, providing unified insights into applications and their underlying infrastructure. By correlating JVM metrics with spans, users can investigate potential bottlenecks in their runtime environment and respond accordingly. Datadog's integration dashboard provides real-time visibility into JVM health and activity, including garbage collection, heap and non-heap memory usage, and thread count. The platform also automatically collects JVM runtime metrics starting from version 0.29.0, enabling users to get deeper context around their Java traces and application performance data.

Akamai, a leading provider of content delivery network (CDN) solutions, has partnered with Datadog to enable users to monitor the utilization and performance of their CDN. The integration allows for visualization, alerting, and correlation with data from other parts of the web stack. Datadog consumes metrics on request traffic, response times, HTTP response codes, and cache hits and misses via Akamai's DataStream API. This partnership provides enhanced visibility into CDN utilization and performance, allowing users to monitor their content delivery network alongside the rest of their infrastructure and applications.

Akamai is a leading provider of content delivery network solutions that handle many millions of HTTP requests per second, delivering 15 to 30 percent of global web traffic. Datadog has integrated with Akamai's DataStream API to provide users with real-time monitoring and visualization capabilities for their CDN utilization and performance. The integration allows users to track metrics such as request traffic, response times, HTTP response codes, and cache hits and misses, providing enhanced visibility into the performance of their content delivery network. With this integration, users can troubleshoot issues behind the CDN by tracking origin response time and latency, and apply anomaly detection to identify unusual patterns in request traffic. The partnership between Datadog and Akamai enables users to gain a better understanding of their CDN's performance and make data-driven decisions to optimize their infrastructure and applications.

Datadog has introduced a new feature that allows users to install updates to their Agent integrations as soon as they are released. This enables quicker access to new or updated integrations without having to wait for a full release of the Datadog Agent. The `datadog-agent integration` command, available on Linux and Windows, provides four subcommands: install, remove, show, and freeze. These commands enable users to manage individual integrations between Agent releases. To upgrade an integration to a newer version on a Linux host, users can run the following command: `sudo -u dd-agent -- datadog-agent integration install datadog-<integration_name>==<version>`. This feature is available in version 6.9.0 of the Datadog Agent and more information on its usage can be found in the Agent docs.

Hippolyte Henry and John Matson announce a new feature in Datadog that allows users to install updates to their Agent integrations as soon as they are released, providing access to new or updated integrations without waiting for a full release of the Agent. The `datadog-agent integration` command enables users to quickly and securely install individual integrations between Agent releases. This feature is available starting from version 6.9.0 of the Datadog Agent, which includes four subcommands: install, remove, show, and freeze. Users can upgrade their integrations to newer versions using this command, making it easier to stay up-to-date with the latest integration enhancements and updates.

The text discusses a challenge faced by developers at Datadog: ensuring end-to-end security when using automation to build, sign, and publish software integrations. To address this issue, the company uses two key technologies - The Update Framework (TUF) and in-toto. TUF is used for signing new integrations while in-toto guarantees that the CI/CD system packaged exactly the source code that one of their developers signed. These technologies are integrated to protect the authenticity and integrity of Agent integrations, from the moment that developers commit source code, to the point that end-users install them as packages. The four steps of the Datadog Agent integrations supply chain ensure end-to-end verification by only trusting wheels containing source code released by Datadog developers. TUF also provides a compromise-resilient mechanism for securely distributing, revoking, and replacing public keys used to verify the supply chain. Developers sign integrations using hardware keys (Yubikeys), which are trusted and support on-card generation and storage of GPG signing keys. The Agent transparently calls TUF and in-toto libraries on behalf of customers for installation or update of integrations, providing a seamless user experience while ensuring security.

The Datadog team has developed a system that empowers developers to release new and trustworthy Datadog Agent integrations on demand, without completely trusting automation. This system uses the Update Framework (TUF) and in-toto to guarantee end-to-end security by protecting the authenticity and integrity of Agent integrations from the moment they are signed by developers to the point when they are installed by end-users. The system includes a supply chain defined using in-toto, which specifies a series of steps that must be followed to produce signed metadata about the input received and the output produced. This ensures that tampering with any step in the supply chain is prevented, providing meaningful security guarantees. The system also uses TUF to securely distribute and revoke public keys used to verify the supply chain, ensuring compromise-resilience. Additionally, developers sign integrations using hardware keys (Yubikeys), which are trusted and support on-card generation and storage of GPG signing keys.