December 2019 Summaries

The Datadog Agent and Cluster Agent are used to collect metrics, logs, and other telemetry data from a Kubernetes cluster. The Agent provides visibility into individual nodes, while the Cluster Agent collects cluster-level data and reduces the load on the Kubernetes API server. Autodiscovery enables automatic monitoring of new containers as they are deployed to the cluster. Datadog also integrates with other technologies such as Docker, containerd, etcd, Istio, and Apache (httpd), providing a comprehensive view of the entire infrastructure. The Agent can be deployed in various ways, including using Helm or deploying it directly onto nodes. To deploy the Cluster Agent, a Kubernetes secret is needed to provide authentication token information. Once deployed, the Cluster Agent enables auto-scaling of Kubernetes workloads and provides additional security benefits. Datadog APM tracks application performance, providing deep visibility into requests, dependencies between services, database queries, and other insights that enable optimization and troubleshooting. The Agent can collect logs from Kubernetes, Docker, and other technologies, enabling log collection and analysis. Additionally, Datadog provides a Log Explorer for viewing logs, an audit log feature for tracking API server requests, and categorization of logs using tags such as source and service. With the Cluster Agent and node-based Agent deployed, users can view their cluster's audit logs, track application performance, and collect custom metrics in Kubernetes.

The text provides a comprehensive guide on monitoring Kubernetes clusters using open-source tools, focusing on collecting and visualizing resource metrics, cluster status information, and pod logs. It emphasizes the deployment and utility of Metrics Server to access CPU and memory usage data, the use of commands like `kubectl top` and `kubectl describe` for real-time metrics and resource allocation details, and the installation of Kubernetes Dashboard for a graphical interface. Additionally, it discusses the role of kube-state-metrics in providing cluster state metrics beyond those offered by Metrics Server, and highlights the importance of logs for troubleshooting via `kubectl logs`. The text also introduces Datadog as a full-stack monitoring solution for Kubernetes environments, offering integrations, autodiscovery, and advanced analytics to provide comprehensive visibility into both infrastructure and applications.

Metric Correlations is a new feature by Datadog that helps users identify possible root causes of issues in their system by searching for correlated metrics and events. It automatically finds candidates for the causes of an issue, saving time and effort compared to manual browsing through dashboards and plotting metrics. The feature can be launched from multiple entry points such as dashboards, monitors, notebooks, and Watchdog, which detects abnormal trends within a metric. Metric Correlations helps users understand the full extent of a problem and its side effects, allowing them to quickly find remediation paths. Users can adjust the area of interest for their search and tailor it to include custom metrics or specific environments, services, or parts of infrastructure.

Metric Correlations is a new feature introduced by Datadog to help users pinpoint possible root causes of issues in their systems more efficiently. By automatically searching for correlated metrics, it enables users to launch investigations from multiple entry points and discover the full extent of a problem and its side effects. The feature groups results by source to help identify components involved in an issue and provides detailed views of correlations found from other sources, allowing users to drill down into specific areas of investigation. Additionally, Metric Correlations allows users to adjust their search area, include custom metrics, and scope their search to specific environments or services. With this feature, users can speed up their investigations and understand the scope of any issue more quickly.

SAP HANA is an advanced data analytics platform that uses in-memory and column-oriented data storage for efficient transactional and analytical queries. It integrates seamlessly with remote databases like Hadoop or SQL Server, and serves as the database foundation for SAP's S/4HANA ERP system. Datadog's new integration helps users monitor the health and performance of their SAP HANA systems by providing an out-of-the-box dashboard that displays key resource metrics such as memory usage and disk space utilization. Monitoring memory usage is crucial for ensuring optimal query execution, while tracking available storage prevents potential disk-full events that could halt SAP HANA's operation. By integrating with Datadog, users can monitor their SAP HANA systems alongside other data sources in a single platform, enabling comprehensive performance monitoring and capacity planning.

Datadog has integrated its platform with SAP HANA, allowing users to monitor the performance and health of their SAP HANA systems in real-time. The integration provides an out-of-the-box dashboard that displays key resource metrics such as memory usage and disk space utilization, enabling users to identify potential issues before they affect end-users. By tracking metrics like memory utilization and disk space usage, users can ensure that their SAP HANA systems are running smoothly and make data-driven decisions about capacity planning and workload balancing. The integration also enables users to set up alerts for critical thresholds such as disk full events or excessive log file growth, providing timely warnings before SAP HANA stops working due to lack of available storage space. With this integration, users can monitor their SAP HANA systems alongside other data sources in a single platform, including Oracle, Hadoop, and IBM DB2, and take advantage of over 850 integrations with Datadog's platform.

Cilium is an open source technology that provides network security to containerized environments at both packet and application levels. It integrates seamlessly with Kubernetes clusters and Docker environments using Mesos. Traditional firewalls filter traffic based on IP address and port, but Cilium overcomes this by using the Linux kernel's Berkeley Packet Filter (BPF) to enforce security policies using container identities or abstractions like Kubernetes service or pod. Datadog now integrates with Cilium to help users ensure their network policies are properly deployed and enforced, providing visualization and alerting on key metrics exposed by the Cilium Agent and Operator. The integration allows for tracking endpoint health and lifecycle events, monitoring endpoint regeneration duration, identifying when packets are dropped, and detecting issues with policy imports.

Cilium is an open-source technology that delivers network security to large-scale containerized environments at the packet and application levels, integrating easily with Kubernetes clusters. It uses the Linux kernel's Berkeley Packet Filter (BPF) to enforce security policies using container identities or abstractions like Kubernetes service or pod, providing low performance overhead. Cilium integrates with Datadog to help ensure network policies are properly deployed and enforced, enabling visualization and alerting on key metrics such as endpoint regeneration, packet flow, and policy import errors. The technology tracks endpoint health and lifecycle events, monitors endpoint regeneration duration, and detects issues with policy imports, providing a comprehensive solution for securing containers made easy.

Datadog has released version 7 of its Agent, which now exclusively supports the Python 3 runtime, marking a shift from previous versions that supported both Python 2 and 3, in line with Python 2's end of life in January 2020. The new release maintains all functionalities from Agent 6, and Datadog has tested over 850 integrations to ensure compatibility with Python 3. Users with custom checks written in Python 2 are provided with tools and documentation to facilitate migration, including an in-app Custom Check Compatibility tool and a detailed migration guide. Datadog Agent 6.16 allows users to test Python 3 compatibility by configuring the runtime version, ensuring a smooth transition before upgrading to Agent 7. The documentation also includes resources for using Containerized Agent images and libraries to aid in converting Python 2 code to Python 3, with a free 14-day trial available for new users.

Systemd is an initialization program that manages processes on Linux systems, designed to improve performance by creating a dependency tree of system components and using parallelization. Datadog's new integration with systemd provides comprehensive visibility into system management within Linux deployments. The integration includes detailed metrics for the status of units managed by systemd, such as active, activating, inactive, deactivating, and failed units over time. It also offers an out-of-the-box dashboard for systemd that surveys per-unit metrics across infrastructure, with a focus on commonly used units like cron, SSH, and syslog. Additionally, the integration runs service checks to report unit health statuses and detect if Datadog can no longer connect to systemd or if systemd is unavailable. Users can set alerts for critical statuses. The integration also helps in visualizing per-unit resource consumption over time, enabling users to understand typical usage levels and set reasonable limits.

Systemd is an initialization program that manages processes on Linux systems, creating a dependency tree of system components and initializing them only when needed. With the integration with Datadog, comprehensive visibility into system management within Linux deployments is provided, allowing users to monitor the health and performance of both systemd and its managed components. The integration offers detailed metrics for unit status, troubleshoot process initialization, and out-of-the-box dashboards for per-unit metrics across infrastructure. It also includes service checks to detect unit failures, resource consumption, and provides alerts and notifications when issues arise, enabling users to diagnose issues with process management more easily.

Microsoft Azure DevOps is now integrated with Datadog, providing users with new insights into their builds, releases, work items, and code events. The integration allows teams to understand how deployments impact application performance and halt bad updates automatically. Managers can use Datadog-derived metrics for tracking the duration of builds and completed work items, improving development and operations workflows. Azure DevOps is not just a tool for build/deploy pipelines but also used as a code repository, testing toolkit, and team management platform. The integration enables users to monitor all their Azure DevOps workflows in one place and analyze them to gain new insights into the effectiveness of their developer operations.

Steve Harrington and Rogan Ferguson from Microsoft have announced a new integration with Azure DevOps, which allows organizations to see the full picture as they build and deploy dynamic applications. This integration provides teams with new insights into their builds, releases, work items, and code events, enabling them to understand how deployments impact application performance and halt bad updates automatically. Managers can also utilize Datadog-derived metrics for tracking development and operations workflows, improving overall development and operations workflows. The integration allows users to track CI/CD pipelines in real-time, correlate events with data from the rest of their infrastructure, and overlay events on top of timeseries graphs to visually correlate build and release events with changes in application performance. Additionally, users can use Datadog monitors as deployment gates in Azure Pipelines to ensure deployments go off without a hitch, monitor devops processes, and set up the integration in minutes.

Mallory Mooney discusses the importance of tagging in modern dynamic environments where large-scale applications are distributed across multiple ephemeral containers or instances. She explains that tags provide context for services and infrastructure, enabling users to isolate individual services for more comprehensive analysis. The article covers best practices for tagging, including collecting metadata from infrastructure, using standard or recommended lists of tags, and assigning custom tags to services and hosts. It also highlights the benefits of integration inheritance, where Datadog automatically collects and applies tags from cloud providers, container platforms, or configuration management tools. Tags allow users to organize complex data streams, pivot between data points, and correlate application data for seamless troubleshooting and issue resolution. By applying these best practices, users can create a complete picture of their application activity, provide context to each moving piece of their stack, and be more proactive in addressing issues before they impact customers.

Datadog has integrated with AWS Identity and Access Management (IAM) Access Analyzer to provide critical visibility into the security of IAM resources alongside other AWS resources within a single pane of glass. This integration enables administrators and security teams to identify potential threats, discover unused access, and remove unnecessary credentials to strengthen their security posture. The integration also complements Datadog's existing support for Amazon GuardDuty and plans to integrate with Cloud Infrastructure Entitlement Management (CIEM) solution in the future, providing a comprehensive view of identity and access management risks and streamlining the principle of least privilege.

Apache Druid is an open-source data warehouse and analytics platform that enables real-time data ingestion from streaming sources like Kafka and batch data from static files, making it suitable for online analytical processing tasks such as reporting, ad-hoc querying, and dashboarding. Datadog has recently integrated with Druid to monitor its performance, data ingestion, and infrastructure health. This integration allows users to track Druid's ingestion activity, ensuring that data is loaded as expected and spotting any changes in the rate of ingestion. Additionally, it helps monitor query performance, cache hit rates, system resource consumption, and storage utilization. By integrating with Datadog, users can gain deeper visibility into their Druid clusters and troubleshoot potential issues more effectively.

Apache Druid is a data warehouse and analytics platform that captures streaming data from message queues like Apache Kafka and batch data from static files, making it suitable for OLAP tasks. Datadog now integrates with Druid, allowing users to monitor the performance of their queries, data ingestion, and infrastructure health. This integration provides insights into Druid's ingestion activity, query latency, cache hit rates, memory usage, storage utilization, and logs, enabling users to troubleshoot issues, optimize performance, and gain visibility into their infrastructure costs. With Datadog's comprehensive monitoring capabilities, users can maximize the value of their Druid clusters and integrate them with other related technologies for a unified view of their data ecosystem.

Datadog has introduced Real User Monitoring (RUM) to provide businesses with comprehensive visibility into their website's performance and user interactions, complementing existing tools like infrastructure monitoring and synthetic testing. By integrating the Datadog RUM SDK, businesses can automatically gather data on user interactions, such as first contentful paint and DOM events, to enhance understanding of web page interactivity and user experience. RUM organizes data into pageviews and sessions, enabling detailed analysis of user journeys and site performance, while allowing customization with global context attributes for targeted insights. Additionally, RUM captures user actions within pages, aiding in understanding custom activity like ad interactions, and supports troubleshooting by reconstructing user sessions to identify issues. By correlating real user analytics with performance data, RUM helps prioritize engineering efforts to optimize application performance and align with business objectives, offering full-stack visibility for applications regardless of their architecture or framework.

AWS has introduced Provisioned Concurrency, a new feature that enhances the resilience of AWS Lambda to cold starts during bursts of network traffic. This feature ensures that functions remain initialized and ready to handle requests in milliseconds, improving user experience and preventing revenue loss due to slow page loads or request timeouts. Datadog has updated its AWS Lambda integration to include Provisioned Concurrency metrics for comprehensive visibility into all Lambda functions. Understanding and optimizing the usage of Provisioned Concurrency is crucial for ensuring optimal performance and cost-efficiency in serverless applications.

AWS Lambda has released a new feature called Provisioned Concurrency, which makes the service more resilient to cold starts during bursts of network traffic. This feature ensures that functions remain initialized and ready to handle requests in milliseconds, mitigating cold starts and optimizing serverless application performance. The update includes metrics for monitoring Provisioned Concurrency usage, allowing developers to identify underprovisioned functions, adjust allocation in real-time, and troubleshoot performance issues with Datadog's integration.

AWS Fargate enables users to run containerized applications without managing infrastructure, and it is widely used within AWS for serverless containers, particularly with Amazon EKS. Datadog integrates with Amazon EKS on AWS Fargate to provide automatic metric collection and deep visibility into environments, employing features like Autodiscovery and APM to monitor application performance in real time. To deploy Datadog on Amazon EKS with AWS Fargate, users run the Datadog Agent as a sidecar container in each pod, utilizing role-based access control and Datadog's Helm chart for setup. This integration ensures end-to-end visibility without modifying application code, and allows for advanced configuration to control which pods receive the Agent sidecar injection. While AWS Fargate abstracts away host-level metrics, Datadog can still collect Kubernetes metrics and integrate with other AWS services for comprehensive monitoring.

Acorn RISC Machine (Arm) processors, initially released in 1985 for low-power computing, are now making significant strides in cloud computing, with AWS offering Arm-based EC2 instances like A1, M6g, C6g, and R6g powered by Graviton2 processors. To support this shift, Datadog has released an Arm-compatible agent that provides comprehensive visibility into infrastructure by collecting metrics, traces, and logs. This agent integrates with numerous technologies and supports monitoring Arm-based environments alongside traditional processors, whether through process monitoring, container maps, or distributed tracing and APM. It facilitates the management of horizontally scalable workloads and helps optimize deployments by providing insights into request rates, error rates, and performance impacts, enabling users to scale deployments efficiently and identify bottlenecks. The Datadog Agent for Arm is generally available, offering a 14-day free trial for those without a Datadog account.

Google Workspace is a collection of cloud-based productivity and collaboration tools developed by Google. Millions of teams use these tools to streamline their workflows, making it essential for security monitoring and audits. Datadog provides integration with Google Workspace audit logs, allowing administrators to monitor key events such as administrative access, activity in Google Drive, user login activity, changes to a user's account, and more. With this integration, users can search, analyze, and alert on their Google Workspace activity just like other system and application logs. The integration also enables the detection of suspicious trends in login attempts, tracking of changes to files in Google Drive, monitoring of access to the Admin console, and capturing real-time security events from Google's Alert Center. Additionally, Datadog provides tools for auditing, archiving, and analyzing these logs, enabling administrators to gather context for troubleshooting issues, conducting audits, and detecting suspicious activity.