April 2020 Summaries

In February, Datadog hosted a global internal hackathon across its offices worldwide. The event allowed employees from various departments and roles to pitch ideas, explore new technologies, and collaborate with each other. Participants worked on diverse projects such as process improvements, feature enhancements, and fun hacks within 48 hours. After sharing their accomplishments with the organization, they look forward to further developing these projects before the next hackathon in summer.

Datadog held its winter edition of internal hackathon across all offices worldwide. The event was an opportunity for various teams, including engineers and designers, to pitch ideas, explore new technologies, and collaborate with each other. During the 48-hour period, participants worked on a range of projects, from process improvements to new features, as well as "just-for-fun" hacks. The event concluded with sharing the accomplishments with the rest of the organization, with plans to grow these projects in the coming months until the next hackathon this summer.

In Part 2 of the series, we discuss how to access observability data using tools and services that come with a standard OpenShift installation. We cover monitoring and analysis with the OpenShift web console, command line spot checks on cluster metrics, and viewing logs and events from your OpenShift control plane and pods. The OpenShift monitoring stack includes components for exposing metrics from various objects in your cluster, collecting those metrics, alerting on them, and visualizing them. We also discuss how to use the Prometheus backend for more detailed monitoring and alerting. Additionally, we cover spot checking metrics and data with the `oc` command line tool. Finally, we provide an overview of Datadog's full-stack visibility into OpenShift environments, including seamless integrations with Kubernetes, CRI-O, and all containerized applications, as well as advanced monitoring features such as outlier detection, forecasting, and automatic correlation of observability data.

The OpenShift monitoring stack provides a preconfigured system based on a Kubernetes monitoring stack that uses Prometheus for data collection and alerting via Alertmanager. The stack includes components such as kube-state-metrics, node-exporter, Prometheus Adapter, and Grafana for visualization. This system allows administrators to access key metrics and cluster data through the OpenShift web console or command line tools like `oc`. The monitoring stack also integrates with other OpenShift services, including logging and Telemetry. Datadog provides full-stack visibility into OpenShift environments, offering seamless integrations with Kubernetes, CRI-O, and containerized applications, advanced monitoring features, and automatic correlation of observability data.

Red Hat OpenShift is a Kubernetes-based platform designed to help enterprise users deploy and manage containerized applications, offering enhanced multi-tenancy, security, monitoring, and CI/CD tools compared to standard Kubernetes installations. OpenShift can be deployed as a self-managed cluster or via managed services from major cloud providers like AWS, Azure, and IBM Cloud. It builds on Kubernetes with features such as role-based access control, Security Context Constraints, and extensive support for Operators, which automate tasks like cluster monitoring and storage provisioning. Monitoring OpenShift involves tracking metrics from various cluster components, including cluster state, resource utilization, and control plane performance, to ensure smooth running of workloads and optimal resource allocation. The platform requires specific operating systems for its nodes and has moved from Docker to CRI-O as the default container runtime in its 4.x versions. OpenShift's monitoring capabilities include kube-state-metrics for cluster state information, and metrics are crucial for capacity planning, ensuring applications are not OOM killed, and maintaining efficient resource utilization.

The text provides a comprehensive guide on using Datadog to monitor Red Hat OpenShift environments, emphasizing the deployment of Datadog Cluster Agent and node-based Agents for end-to-end visibility. It discusses the integration of Datadog with OpenShift's metrics, logs, and application performance monitoring (APM) through features like Autodiscovery and security context constraints (SCC) management. The guide covers the use of Helm for deployment, enabling log collection, and setting up APM for detailed application insights, while highlighting the importance of choosing the appropriate security level for data collection. Additional benefits of the Datadog Cluster Agent include reduced load on the Kubernetes API server and enhanced security. The text also outlines the use of tags for monitoring dynamic infrastructures and provides instructions for enabling various Datadog features, such as log analysis and application tracing, to optimize and troubleshoot application performance in OpenShift clusters.

Cisco Meraki offers various IT infrastructure devices such as network security appliances, switches, and wireless access points. Datadog's new Meraki integration collects event logs and Simple Network Management Protocol (SNMP) metrics from Meraki network devices to provide comprehensive visibility into the health and performance of these devices. The integration includes device autodiscovery and automatic ingestion of metrics like packet error count and data throughput. Datadog also tags incoming Meraki metrics with device names and product codes for better organization and flexibility in monitoring. Additionally, it allows forwarding of Meraki event logs to Datadog's log stream for real-time tracking of network devices' status, performance, and connection volume. This integration extends Datadog's SNMP support for on-prem network devices and complements its Network Performance Monitoring product for cloud-based network traffic data insights.

AWS Fargate is a service that enables running applications in Amazon Elastic Container Service without managing the underlying infrastructure. Datadog has supported Fargate since its launch and provides integration to collect real-time, high-resolution metrics from all containerized tasks. The integration also supports Autodiscovery for detecting services running on Fargate and configuring checks automatically. With this integration, users can monitor system metrics such as CPU, memory, I/O, network performance, and code-level performance data with Datadog's Continuous Profiler. Deploying the containerized Datadog Agent on Fargate is simple and allows for real-time monitoring of tasks and services alongside the rest of the infrastructure.

Jenkins is an open source continuous integration server that automates building, testing, and deploying projects. It is widely used by organizations like GitHub, Etsy, LinkedIn, and Datadog. With hundreds of plugins, Jenkins supports a wide variety of use cases. The Datadog plugin enhances visibility into Jenkins environments by collecting real-time system and security events, as well as additional metrics such as queue size and executor counts. It also provides insights into job history and trends beyond Jenkins's standard weather reports. By integrating with Datadog, users can set alerts for important build failures, identify trends in build durations, and correlate Jenkins events with performance metrics from other parts of their infrastructure to resolve issues quickly.

Kafka and ZooKeeper performance monitoring can be achieved through various tools and methods, such as JConsole and JMX for native metric collection, Burrow for monitoring consumer health, and tools like ZooKeeper's four-letter words and AdminServer for ZooKeeper metrics. JConsole, a Java GUI tool, is useful for exploring metrics emitted by Kafka via JMX, though it requires specific setup on a dedicated host due to its resource consumption. More sophisticated monitoring systems are necessary for long-term trend analysis, which can integrate with JMX plugins or metrics reporter libraries. Burrow, a tool developed by LinkedIn, provides detailed insights into Kafka consumer offsets and health, requiring Go for installation and configuration. ZooKeeper metrics can be gathered using JConsole, the deprecated four-letter words, or the AdminServer, with each method offering varying levels of detail and configuration. For production-ready monitoring, dynamic systems like Datadog can provide comprehensive analysis and alerting capabilities by integrating with Kafka, ZooKeeper, and numerous other technologies, facilitating a holistic view of performance metrics across technology stacks.