December 2020 Summaries

Kubernetes is a popular platform for deploying containerized applications, but securing Kubernetes environments as you scale up can be challenging due to the increased attack surface. Monitoring audit logs can help detect and mitigate misconfigurations or abuse of Kubernetes resources before confidential data is compromised. Key types of Kubernetes audit logs to monitor include access to your Kubernetes environment, changes to Kubernetes resources, and user and service account activity. Datadog offers a built-in Kubernetes audit log integration for real-time tracking of environment activity and threat detection rules that automatically monitor audit logs and identify critical security issues in environments.

Kubernetes audit logs provide valuable insights into the security and configuration of containerized environments. Monitoring these logs can help detect misconfigurations, malicious activity, and potential security threats. Understanding how to interpret log entries, identify key audit logs, and use tools like Datadog to monitor and analyze logs is essential for maintaining a secure Kubernetes environment. By implementing effective monitoring and logging strategies, organizations can quickly surface suspicious activity, prevent attacks, and ensure the security of their containerized applications.

You can extract valuable insights from your authentication logs by implementing best practices such as logging all authentication events, including relevant information like user ID, IP address, and outcome. Using a standardized key-value format for your logs makes it easier to parse and analyze them with tools like Datadog's Grok Parser, allowing you to quickly identify security threats and trends in login sources and methods. By standardizing attributes like `usr.id`, `evt.category`, `evt.name`, `evt.outcome`, and `network.client.ip`, you can easily search and aggregate data across all your logs. Additionally, using Datadog's Cloud SIEM service provides turn-key Detection Rules that scan 100% of your ingested logs in real-time for common attacker techniques, generating Security Signals with key data about the event, including suggestions on response strategies.

Datadog has introduced new features to allow users to share their dashboards publicly while maintaining control over the data and who can access it. Users can now grant specific individuals access to their dashboards, even if they don't have a Datadog account. Time selectors and template variables can be used to curate the exact view being shared. Dashboards can be shared by email address or domain, with options for read-only access and adjustable time ranges. These features enable users to share insights while safeguarding sensitive information.

This post explains how to use Datadog's webhooks integration to automatically send an SMS message via Twilio when an alert is triggered in your application or environment. The process involves creating a webhook that calls the Twilio SMS API, using a Synthetic test to call the webhook if an API endpoint is unavailable, and adding dynamic data to the webhook's payload and linking it to the Datadog mobile app. This integration can help you quickly alert your team of issues affecting service reliability or user experience, allowing for faster troubleshooting and minimizing potential downtime.

Datadog dashboards allow users to create unified views of their application, infrastructure, and business data. To share these dashboards with others, users can generate a public URL or limit access to specific individuals by adding email addresses or company domains to an allowlist. Users can also control what data is shared by selecting time frames and template variables. This feature allows for the safe sharing of insights while safeguarding sensitive information. Additionally, Datadog's API enables programmatically managing and sharing public dashboards, making it easy to get started with a 14-day free trial.

Microsoft 365 is a suite of cloud-based productivity and communication services that includes popular Microsoft tools like Office applications, Skype, and Teams. Datadog can be used to ingest all Microsoft 365 audit logs for deeper insight into the security of these services and monitor usage across an organization. Audit logs contain valuable information about actions performed within a Microsoft 365 environment, which is useful in determining compliance status. With Datadog, users can analyze and alert on these logs in real-time for security threats, centralize monitoring, and eliminate friction across teams. The integration allows for tracking of service activity and file access, user activity, authentication and authorization actions, and security threats. Microsoft 365 audit logs also help track file activity across OneDrive and SharePoint, including content edits and sharing settings changes. Datadog's out-of-the-box dashboard provides a full-picture perspective of the security and performance of Microsoft 365 services and infrastructure.

Cloud Foundry is an open-source platform that simplifies the deployment and orchestration of applications without requiring developers to configure underlying infrastructure. VMware Tanzu Application Service (TAS) is a commercial, certified version of Cloud Foundry with additional features such as a partner network, auto-scaling CLI, and operations interface. Datadog's integration works on all distributions of Cloud Foundry and can be easily installed within Tanzu Application Service clusters using the Datadog Cluster Monitoring tile. This integration provides comprehensive visibility into virtual machines running on TAS deployments and helps track container-level health, performance, and resource utilization in real time. It also enables automatic collection of Cloud Foundry audit events for correlation with infrastructure and application metrics. Using Datadog's integration, developers can monitor applications running on their TAS deployment with minimal configuration and gain insights into possible issues within the cluster by using tagged metrics from TAS applications. Additionally, the integration allows users to create alerts on high numbers of Cloud Foundry audit events and configure notifications for their team.

Datadog's Microsoft 365 integration allows organizations to monitor and analyze their Microsoft services and tools, including Office applications, Skype, Teams, OneNote, OneDrive, SharePoint, Exchange, and more. The integration ingests Microsoft 365 audit logs directly into Datadog, providing rich information about actions performed within the environment, such as logins, file edits, and more. This enables organizations to track user activity, monitor service activity and file access, detect security threats, and analyze authentication and authorization actions. The integration also includes an out-of-the-box dashboard that provides a full-picture perspective of the security and performance of Microsoft 365 services and infrastructure. With Datadog's Microsoft 365 integration, organizations can gain deeper insights into their Microsoft cloud applications and ensure compliance with security standards.

Datadog's integration with Cloud Foundry and VMware Tanzu Application Service (TAS) provides comprehensive visibility into virtual machines running on TAS deployments, allowing developers to track container-level health, performance, and resource utilization. The integration enables automatic tracking of ephemeral containers across Cloud Foundry's dynamic environment, gathering resource metrics from application containers running in TAS deployments. With minimal configuration, developers can monitor their applications with the same frictionless experience as deploying them to TAS, using Autodiscovery to enable integrations for all containers and external service instances. The integration also collects Cloud Foundry audit events, enabling developers to correlate trends in a metric with changes in their applications, overlay events atop graphs, create alerts on high numbers of audit events, and share these graphs with colleagues or declare an incident to reduce investigation time.

Mallory Mooney's text discusses the importance of API testing in ensuring the efficient operation of applications, particularly in maintaining optimal user experiences by preventing slow endpoints and unexpected timeouts. Datadog Synthetic Monitoring is highlighted as a tool that offers comprehensive API testing capabilities, allowing users to assess HTTP requests, DNS records, TCP connections, and SSL certificates. The platform enables multistep API tests to monitor critical workflows, ensuring that data from initial requests can be effectively passed to subsequent ones, thus verifying the functionality of complex API interactions like user authentication and profile updates. Datadog also provides tools to track TCP connection performance and verify SSL certificates, ensuring secure data transmission and connection reliability. By offering insights into each step of API workflows, Datadog helps users quickly identify and resolve issues, enhancing the overall health and security of application services.

Kubernetes cluster machines are divided into worker nodes and control plane nodes. Worker nodes run pods and their applications while the control plane node runs the Kubernetes Control Plane responsible for managing worker nodes. Datadog's Kubernetes integrations now provide out-of-the-box telemetry for four main components of the Control Plane: API Server, Controller Manager, Scheduler, and etcd. These new integrations offer detailed insights into the health of your Kubernetes infrastructure, allowing you to monitor your container workloads alongside all the components of your Kubernetes cluster.

This guide demonstrates how to set up monitoring for a containerized application orchestrated by Kubernetes using Datadog. It covers collecting Kubernetes and Docker metrics, using Autodiscovery for default and custom check configurations, instrumenting applications to send custom metrics to Datadog, and more. The example used is the guestbook-go application from the Kubernetes project.

At Peloton Cycle, DevOps Engineer K.Z Win uses Datadog to collect server and application metrics for monitoring system health, sending alerts, and providing business insights. The company's infrastructure includes high-end indoor bicycles with Android tablets that stream live coaching sessions and send rider performance data via HTTP protocol to application servers running behind reverse-proxied NGINX. Datadog integrates easily with all software used in Peloton Cycle's infrastructure, including NGINX. The company uses a linear transformation of NGINX requests per second to estimate the number of live rides but later switched to using DogStatsD for more accurate ride counting. Additionally, they implemented custom checks for monitoring replica or standby instances in their Redis and PostgreSQL databases. Overall, Datadog has proven to be a developer-friendly metric collecting service with useful integrations and an open source framework that allows easy customization and extension.

The Datadog Clipboard is a feature that streamlines investigation by capturing relevant views of infrastructure and applications in a single place, allowing users to access the information they copied from the bottom of their screen as they navigate to other pages. Signals can be copied to the Clipboard, which includes dashboard widgets with existing copy-paste functionality, URLs of specific views within Datadog, and more. The Clipboard works on all pages of Datadog, storing up to 20 signals even when it is not open. It helps users organize findings from their investigations in one place without cluttering their browser with tabs, enabling frictionless troubleshooting and the creation of shared resources such as runbooks and dashboards. By using the Clipboard, users can focus on discovering root causes while investigating incidents, gather data from different contexts, and create dashboards more efficiently.

Datadog enhances application performance monitoring by providing a comprehensive trace view that consolidates telemetry data, such as traces, metrics, and logs, in a single interface, thereby overcoming the limitations of using multiple monitoring tools. This integration allows for efficient troubleshooting by displaying detailed request data alongside correlated information, which helps identify performance issues like errors and latency across both frontend and backend components. Key features include the use of tags to filter and compare traces, infrastructure context to isolate performance issues, and metrics that correlate with trace data to spotlight potential memory and CPU resource constraints. Additionally, the tool offers automatic log correlation, infrastructure process details, and network performance insights, which together facilitate a deeper understanding of application behavior and resource utilization. By visualizing code hotspots and leveraging continuous profiling, users can identify methods that require optimization to reduce latency, while network data aids in diagnosing connectivity issues affecting application performance. Overall, Datadog's end-to-end monitoring solution aims to enhance visibility, reduce mean time to resolution, and improve user experience by unifying application insights into a streamlined view.

Full-stack visibility is crucial for efficient troubleshooting and incident resolution in applications. Datadog's Real User Monitoring (RUM) provides real-time insights into user experiences, while distributed tracing offers visibility into individual request lifespans and key performance metrics. By connecting RUM data with corresponding traces, users can gain unified, end-to-end visibility across their stack. This rich context helps locate backend problems resulting in user-facing errors or identify the scope of affected users due to an issue within the application stack. Datadog's integration between RUM and APM allows for seamless pivoting from frontend data to view backend traces, enabling faster identification and resolution of root causes. Additionally, RUM provides deep context around incidents by analyzing who was affected and how widespread the impact was. With full-stack visibility, users can easily trace browser timeouts to database operations or link API failures to web component errors within a single pane of glass.

Datadog's Real User Monitoring (RUM) and Application Performance Monitoring (APM) enable full-stack visibility, allowing developers to investigate backend dependencies when troubleshooting frontend issues or track the source of database failures. RUM provides real-time insight into how users experience an application, while APM offers distributed tracing and key performance metrics. The integration between RUM and APM enables unified visibility across layers of a stack, providing rich context around problems and helping developers locate backend root causes more easily. By connecting frontend data with corresponding traces, developers can pinpoint issues, measure end-user impact, and debug the issue using full visibility into metrics, logs, network performance data, and code hotspots from within a single pane of glass.

Mary Jac Heuman discusses the challenges of creating clear, data-driven documentation in collaborative environments and introduces Datadog Notebooks as a solution. These notebooks are integrated with the Datadog platform, enabling teams to access monitoring data for crafting postmortems, runbooks, and other documents. With features like real-time collaboration, commenting, and automatic postmortem generation from resolved incidents, Datadog Notebooks streamline the process of documentation and incident analysis. They allow for live editing, commenting on individual cells, and facilitate the incorporation of graphs to illustrate incident details, maintaining up-to-date information for all stakeholders. The introduction of these collaboration tools aims to enhance team efficiency and data-driven storytelling within the unified Datadog platform.

Datadog recently ran a game day on one of their Elasticsearch clusters to test the resilience of their systems. They stopped Elasticsearch on various nodes including leader node, client nodes for recent and long-term data, and observed how their applications responded. The lessons learned include being prepared for 503s during leader election, handling dangling indices, and implementing health checks for client nodes. Game day exercises are a great way to test systems' fault tolerance and improve alerts and fixes.

Apache Mesos is a distributed systems kernel that abstracts data center resources into a single pool for running applications at scale, as evidenced by its adoption by companies like Twitter, Airbnb, Netflix, and Apple. However, deploying Mesos can be complex due to the need to find and configure compatible solutions for service discovery, load balancing, monitoring, etc. DC/OS (Datacenter Operating System) simplifies this process by bundling these technologies with an easy-to-install and configure full-fledged operating system for Mesos clusters. Datadog is a popular choice for monitoring Mesos clusters due to its scalability and compatibility with distributed infrastructure components. Installing the Datadog Agent on both agent and leader nodes of a Mesos cluster allows users to view metrics at various levels of granularity, while additional integrations enable monitoring of services and applications running in the cluster. Autodiscovery is an automated feature that detects which containerized services are running on which nodes and configures the Datadog Agent accordingly.

The integration between Amazon Web Services (AWS) Well-Architected Tool and Datadog Cloud Security Posture Management (CSPM) enhances the ability of users to build secure, efficient, and cost-optimized cloud architectures by offering a centralized method to track adherence to Well-Architected best practices. By leveraging over 200 out-of-the-box rules, Datadog CSPM helps identify potential misconfigurations in cloud services that could compromise security, allowing for immediate validation and adherence to best practices within the Well-Architected security pillar, such as managing identities, detecting security events, and protecting networks and data. The integration also allows for streamlined architectural reviews by mapping compliance findings back to Well-Architected best practices, facilitating quick investigation and remediation through Datadog. Additionally, the partnership provides prescriptive guidance for optimizing management and governance across AWS environments, with Datadog CSPM offering users the option to self-enroll or explore a free trial to enhance their security operations and compliance posture.

Threat intelligence is crucial in identifying suspicious activity before it escalates into a more serious attack. Datadog Cloud SIEM offers turn-key threat intelligence feeds curated by specialized partners, which are constantly updated to include data about known suspicious activity (e.g., IOCs). These feeds help enrich all ingested logs with expert-curated threat intelligence in real time and detect activity from known threat actors. Threat Intelligence also accelerates investigations and reduces noise by automatically surfacing relevant context within security alerts, making it easier to prioritize alert triage. By leveraging key knowledge from expert security partners, Datadog Cloud SIEM helps users automatically surface threats to their applications and infrastructure.

Datadog Cloud Network Monitoring (CNM) provides comprehensive visibility into dynamic environments, including support for Windows hosts. It enables quick identification of connectivity issues across regions, services, and operating systems. CNM offers live visualization of network topologies, allowing users to visualize their network architecture with metric and tag-based filters. The tool maps network traffic between hosts, making it easier to locate problems in the network. Users can also use the Network Overview to drill down into specific dependencies and identify inefficiencies in their network. With CNM Windows support, users can visualize network flow data alongside logs, infrastructure metrics, and application traces, enabling them to quickly pinpoint root causes of issues within their Windows or multi-OS network without switching contexts. The tool provides real-time insights into network latency and connectivity issues, allowing users to troubleshoot and optimize their network architecture.

Hyperconverged infrastructure (HCI) offers flexibility and control in building and securing virtualized on-premises environments. Microsoft's Azure Stack HCI is an operating system-as-a-service for HCI backends, allowing organizations to manage their virtualized resources alongside any Azure infrastructure they are running. Datadog can be used to collect and monitor key metrics, traces, logs, and more from across the entire Azure infrastructure and applications, including those running on Azure Stack HCI. This provides full visibility into every layer of an organization's environment, with support for Microsoft technologies like SQL Server, IIS, and .NET applications. Datadog also integrates with the full range of Azure services, allowing organizations to monitor hybrid scenarios and gain out-of-the-box visibility into the health and performance of all their resources side by side.

Microsoft's Azure Stack HCI is an operating system-as-a-service built for Hyperconverged Infrastructure (HCI) backends, allowing organizations to deploy and manage their virtualized resources alongside any Azure infrastructure they are running. Datadog integrates with Azure Stack HCI, providing full visibility into every layer of the environment by collecting key metrics, traces, logs, and more from across the entire Azure stack, including on-premises and cloud resources. With Datadog's support for Azure Stack HCI, organizations can monitor their entire Azure ecosystem, whether running in on-premise datacenters or a hybrid solution, and gain insights into health and performance data for every component in their environment.

In modern cloud infrastructure, network monitoring is crucial due to increased distribution and exposure to networking issues. Datadog's APM (Application Performance Monitoring) and NPM (Network Performance Monitoring) are integrated tools that help analyze application and network performance data together for efficient troubleshooting of root causes of application issues. By using these tools, users can easily correlate traces with network data to identify the source of application latency, examine spans showing runtime errors or long execution times, investigate resource bottlenecks, and visualize network traffic between any tagged object in their environment. Datadog's unified tagging feature allows for seamless integration between NPM and APM, enabling teams to gain more context into problems and streamline the process of managing services and triaging issues in production.

Datadog's Cloud Network Monitoring (CNM) and Application Performance Monitoring (APM) are tightly integrated tools that help troubleshoot application performance issues by correlating traces with network data. With CNM, teams can analyze network communication between services, cloud regions, containers, and more to identify root causes of latency and connectivity issues. By combining APM and CNM, engineers can quickly pivot between monitoring individual endpoints in their application and analyzing the underlying network behavior, allowing for a more comprehensive understanding of infrastructure health. This integration enables unified tagging, making it easier to correlate data across services, cloud regions, containers, and more. By using Datadog's tools, teams can streamline root cause analysis workflows, improve context into problems, and manage services more effectively in production.

Amazon Web Services (AWS) has introduced macOS instances on Amazon EC2, allowing iOS, macOS, and Safari developers to move their build and test pipelines to the cloud. This enables them to take advantage of AWS's ecosystem for security, storage, and networking. Datadog supports monitoring of these macOS instances along with other parts of the stack. The integration collects CloudWatch data from all instances and provides an out-of-the-box dashboard that visualizes key EC2 metrics. Deploying the Datadog Agent enables further granular insights into system-level metrics, logs, and traces. Datadog also offers tools to continuously monitor CI/CD workflows for Mac developers who can now integrate AWS EC2 into their development pipelines.

AWS has announced that macOS instances are now available on Amazon EC2, allowing developers to move their build and test pipelines to the cloud and take advantage of AWS services such as security, storage, and networking. This development enables iOS and macOS developers to modernize their CI/CD pipelines with flexibility and cost benefits similar to those offered by Windows and Linux counterparts. Datadog provides a suite of tools for monitoring macOS instances alongside the rest of the stack, offering insights into health and performance through dashboards that visualize key metrics such as CPU utilization, disk I/O, and network throughput. The Datadog Agent can collect additional system-level metrics and integrates with popular continuous integration services to track CI/CD workflows in real-time, enabling developers to identify problems and make intelligent decisions on development. With the support for macOS EC2 instances, developers of iOS and macOS applications can gain end-to-end visibility into their entire stack, allowing them to troubleshoot problems when they arise.

The serverless ecosystem has matured into an effective strategy for building modern, enterprise-scale products. However, developers often face challenges in gaining visibility between key applications when migrating to containers or serverless architectures. Datadog Serverless Monitoring aims to provide end-to-end visibility across serverless infrastructure, even as the ecosystem expands. With AWS Lambda's support for deploying functions packaged as container images, more companies can leverage serverless while continuing to use their existing container tools and development environments. Datadog's Lambda integration now includes support for functions packaged as container images, allowing seamless monitoring of all services in one platform.

Datadog has introduced Datadog Serverless Monitoring to provide end-to-end visibility across serverless infrastructure, helping companies navigate the complexity of rapidly changing technologies. The integration now supports functions packaged as container images, allowing developers to leverage the benefits of serverless while continuing to use their existing container tools and development environments. With this feature, developers can monitor the health of all services in one platform, even as their dynamic environment evolves. Datadog's monitoring capabilities include correlating traces, logs, and metrics across any kind of infrastructure, making it easier for companies to navigate complex systems and troubleshoot issues. The integration also provides comprehensive visibility across AWS Lambda environments at any scale, enabling developers to seamlessly monitor and troubleshoot their serverless applications.

Kubernetes has emerged as the standard for container orchestration, and Amazon EKS Distro (EKS-D) allows users to run on-premise clusters using the same tested versions and security features as Amazon EKS. This compatibility ensures a seamless migration of applications and provides users with control over their infrastructure while benefiting from the reliability of a maintained and tested Kubernetes distribution. Datadog partners with AWS to enhance visibility into EKS-D through integrations with technologies like containerd, Istio, and CoreDNS, enabling users to monitor clusters on-premise or in the cloud. Datadog's monitoring tools offer insights into the health and performance of Kubernetes clusters by collecting metrics, events, and logs from various components, including the control plane and worker nodes. These capabilities allow users to troubleshoot issues quickly and gain comprehensive visibility into their stack, with automatic discovery and configuration of integrations as clusters evolve.