February 2023 Summaries

StormForge Optimize Live is a machine learning-powered solution for Kubernetes workloads performance and resource optimization. It analyzes production observability data and recommends actions to optimize CPU and memory utilization, which can be implemented manually or automatically. The software license can now be purchased in the Datadog Marketplace, allowing users to access an out-of-the-box dashboard displaying key Kubernetes performance metrics alongside recommended optimization actions. By following these recommendations, users can adjust resource allocations to match actual usage and optimize cloud spend. Additionally, StormForge Optimize Live can be configured to automatically implement recommended resourcing levels across pods, working in concert with the Kubernetes HorizontalPodAutoscaler (HPA) or VerticalPodAutoscaler(VPA).

At Datadog, the Engineering team focuses on developing innovative solutions in monitoring and observability, particularly for cloud-native security. Dayspring Johnson, a Product Detection Engineer at Datadog, spends his day researching new attack techniques and creating effective detection tactics to protect against them. The Product Detection Engineering team plays a crucial role in enhancing the company's broad security platform by detecting and understanding threats before they can cause significant harm. Johnson emphasizes the importance of collaboration across teams at Datadog, working together towards a common goal of breaking down silos and providing customers with seamless experiences using multiple products. The Engineering team culture is supportive, collaborative, and innovative, fostering an environment where employees can grow and contribute to the broader security community through blogs, presentations, and conferences.

At Datadog, the Engineering team is a key driver of innovation and solution implementation in the monitoring and observability industry. The Product Detection Engineer, Dayspring Johnson, plays a crucial role in uncovering new security threats and vulnerabilities in the cloud landscape. His work involves researching attack techniques, replicating them in test environments, and developing effective detection tactics to combat these threats. Johnson's impact is significant, as his team works towards adding security context to observability data, enabling customers to better understand potential threats across their cloud environments. He collaborates with various teams to identify customer needs and develop products that help customers stay secure in modern cloud-native environments. Johnson attributes his attraction to Datadog to its culture of collaboration, innovation, and diversity, particularly the Black in Tech community, which provides a safe space for Black engineers to connect and share their experiences.

Sedai, an autonomous cloud management platform powered by machine learning (ML), helps organizations optimize their modern applications based on container or serverless architectures. By integrating with Datadog, Sedai can now automatically detect and analyze Lambda instances, reducing manual effort required to maintain complex systems. The integration allows teams to configure Sedai to start autonomously optimizing functions for cost and performance, as well as detecting and remediating issues in serverless function metrics. This enables SREs and DevOps personnel to focus on more important tasks while keeping cloud costs low.

Sedai is an autonomous cloud management platform that uses machine learning (ML) to optimize modern applications based on container or serverless architectures. It helps teams achieve the appropriate balance of performance and cost, while reducing manual effort required to maintain complex systems. Sedai integrates with Datadog out of the box and offers a subscription to their software in the Datadog Marketplace to automatically detect and analyze Lambda instances. With this integration, users can configure Sedai to start optimizing functions with minimal configuration overhead and continuously monitor its optimization actions from within Datadog. Sedai autonomously optimizes Lambda functions for cost and performance by setting memory levels, tuning concurrency, and mitigating issues such as out-of-memory errors and timeouts. It also detects and remediates performance and availability issues in serverless function metrics, allocating more resources to resolve them. The integration with Datadog enables teams to incorporate Sedai's autonomous remediation capabilities into their existing incident response workflows, making it easier to manage cloud infrastructure deployments.

Identifying and evaluating security vulnerabilities is crucial in software development and system management due to the increasing occurrence of zero-day attacks. The Common Vulnerabilities and Exposures (CVE) Program and the Common Vulnerability Scoring System (CVSS) are vital tools for assessing these threats, but they have limitations. To effectively handle emerging vulnerabilities, security teams should monitor CVE and CVSS data, determine their attack surface, evaluate hype and proof-of-concept code, compute an enriched score, and stay prepared for swift remediation. Tools like Datadog Workflows can help automate and expedite this process.

Aaron Kaplan and Frederic Baguelin discuss the importance of identifying and evaluating security vulnerabilities in software development and system management. They highlight the increasing trend of zero-day attacks, which exploit previously unidentified vulnerabilities, and emphasize the need for vigilance and effective methods of triage. The authors outline industry-defined best practices for evaluating and responding to emerging vulnerabilities, including assessing vulnerability severity using the Common Vulnerabilities and Exposures (CVE) Program and the Common Vulnerability Scoring System (CVSS). They also discuss the importance of enriching CVSS scores with additional factors such as exploitability, impact, temporal metrics, environmental metrics, hype, and trendiness. The authors provide guidance on establishing a base score, assessing attack surfaces, evaluating hype and searching for proof-of-concept code, computing enriched scores, and determining next steps in response to emerging vulnerabilities. They also highlight the role of tools such as Datadog's Service Catalog and Incident Management in streamlining the process and improving collaboration among teams.

Datadog's third-generation event store, Husky, is a distributed, time-series oriented, columnar store optimized for streaming ingestion and hybrid analytical and search queries. To ensure exactly once ingestion of every event into Husky’s storage engine, the company developed auto-scaling, multi-tenant data ingestion pipelines. They introduced locality by deterministically mapping events to groups of partitions called shards by their ID and timestamp. This allowed for efficient deduplication within a shard and reduced storage costs and improved performance. The Sharding Allocator ensures all Shard Router nodes have a consistent view of allocated Shard Placements, while the Autosharder periodically adjusts configured shard counts on a tenant-by-tenant basis to better fit observed traffic volume. Load balancing is achieved by shifting tenant Shard Placements around using a salting technique and a balancing algorithm that shifts placements around until all shards are roughly balanced. The Writers, responsible for exactly-once ingestion to Husky, persist event IDs in separate Husky tables from the raw event data to ensure consistency between the event data itself and the event IDs once they’ve been committed to the Metadata store.

Husky, Datadog's third-generation event store, uses a distributed, time-series oriented, columnar store optimized for streaming ingestion and hybrid analytical and search queries. Husky decouples storage and compute to scale independently. To ensure exactly-once ingestion of every event into Husky's storage engine, Husky Writers consume events from assigned shards and persist them with the goal of making them queryable as quickly as possible without introducing duplicates. Husky achieves this by using a hybrid approach to storing IDs in Husky itself, while also storing them in separate Husky tables for fast lookups. Additionally, Husky uses FoundationDB's strong transactional guarantees for conflict detection and resolution, ensuring that every piece of the ingestion path is stateless, scalable, and performant.

Cold Start Tracing is a new feature of Datadog Serverless APM that helps developers visualize and understand the causes behind cold starts in their Lambda functions. By providing an under-the-hood view of dependencies loaded during a cold start, this tracing tool allows developers to identify which processes contribute most to the duration of a cold start. Cold Start Tracing also enables users to test changes made to their Lambda function configurations and assess their effectiveness in mitigating cold starts. Currently available for Node.js and Python runtimes, with more support coming soon, this feature aims to improve the performance of serverless functions by providing actionable insights into reducing cold start overhead.

Cold starts in serverless functions can be caused by various factors such as new function containers being initialized, unneeded dependencies, and tightly scoped module imports. Datadog Serverless Monitoring detects cold starts, visualizes their impact on services through distributed traces, and allows for alerts based on the rate they occur. Cold Start Tracing via Datadog Serverless APM provides an under-the-hood view of dependencies loaded during a cold start, enabling developers to identify root causes and gain actionable insights to optimize performance and reduce costs. This feature helps developers visualize the steps executed during a function's cold start, such as downloading libraries and modules, and enables them to go beyond simple detection by identifying which parts of their code are contributing to cold starts. By using Cold Start Tracing, developers can fine-tune their Lambda function configurations, test changes, and improve the performance of their serverless functions with support for Node.js and Python runtimes coming soon.

Datadog Code Analysis is a tool designed to help engineering teams maintain high code quality and security by detecting and addressing issues early in the development process. It leverages a Static Analyzer to scan codebases for vulnerabilities and maintainability issues, integrating directly into IDEs such as VS Code and JetBrains, and GitHub pull requests, providing real-time feedback and suggestions for fixes. The platform also includes Software Composition Analysis to identify vulnerabilities in open source libraries. By using these tools, developers can improve their code quality and security, reducing the risk of vulnerabilities reaching production. Additionally, Datadog offers features like Quality Gates to enforce coding standards and prevent unaddressed issues from being merged. The platform helps manage existing code violations and mitigate risks from third-party libraries, ensuring comprehensive code security and quality.

Datadog's Application Security Management (ASM) provides a comprehensive solution for securing modern production systems, offering a range of measures including secure coding practices, security testing, web application firewall protection, threat detection, and vulnerability scanning. ASM leverages Datadog's distributed tracing to provide security coverage below the network level and with rich application context, enabling teams to track suspicious requests, visualize attack flows, and see which parts of the code are being targeted. The solution includes Protection capabilities that empower teams to block attackers, get code-level context about an attack, and block attack attempts with the In-App WAF, as well as Denylist functionality for managing blocked IP addresses and authenticated attackers. ASM's Protection capabilities enable teams to move beyond detection and take immediate action against attackers, providing a high degree of confidence in blocking legitimate users.

Securing modern production systems, especially with distributed, cloud-hosted applications, requires sophisticated measures beyond traditional methods, which can be cumbersome and inadequate for zero-day coverage. Datadog's App and API Protection (AAP) centralizes application security for DevOps and security teams by using distributed tracing to provide below-network-level coverage with detailed application context. AAP's Protection capabilities allow teams to block attackers automatically, track suspicious requests, visualize attack scopes, and gain code-level insight to mitigate false positives. By integrating security context with distributed trace data, AAP accurately identifies and blocks attackers by IP or compromised accounts, offering customizable blocking durations to allow time for vulnerability remediation. The In-App Web Application Firewall (WAF) enhances security by combining perimeter detection with application-specific context, reducing false positives while automatically blocking threats. AAP empowers teams to act against attackers with high confidence, ensuring legitimate users are not mistakenly blocked, thus safeguarding customer relationships and revenue. These capabilities are available to all AAP customers, and Datadog offers a 14-day free trial for new users.

Argo CD is a continuous delivery tool for Kubernetes that automates application deployment by monitoring live state and comparing it to desired state in manifest files. It enables GitOps compatibility, allowing changes to be tracked via Git. The Datadog Argo CD integration helps visualize activity across clusters, troubleshoot with metrics from every component, and quickly detect application sync issues using preconfigured monitors for key Argo CD metrics. By monitoring all three main components of Argo CD - repository server, application controller, and API server - users can ensure their Kubernetes cluster stays up to date with manifest files. The integration also includes a recommended, preconfigured monitor that alerts users to any app sync failures.

SonarQube is a static code analysis tool that integrates with Continuous Integration (CI) pipelines to monitor code quality as it changes. Its Clean-As-You-Code philosophy helps avoid technical debt by running regular code checks and alerting developers to any issues early on. Datadog's SonarQube integration collects key metrics and logs, providing greater visibility into the quality and stability of your code during development. The integration also allows for monitoring of SonarQube server health and performance, whether it is on-premise or containerized. By visualizing and aggregating metrics and logs, developers can monitor their code deployments alongside their entire CI/CD pipeline.

The Cloud Native Computing Foundation (CNCF) developed Argo CD, a declarative continuous delivery tool for Kubernetes that automates application deployment by continuously monitoring the live state of containers and comparing it against desired state in Kubernetes manifest files. The Datadog Argo CD integration helps ensure that Kubernetes clusters are up to date with latest manifest files via metrics from every Argo CD component, allowing users to visualize activity across their clusters, troubleshoot with metrics from every component, and quickly detect application sync issues with preconfigured monitors. With the integration, users can access performance data, logs, utilization metrics, and Kubernetes cluster stats to help with troubleshooting, and use template variables to narrow down investigations based on specific cluster attributes. The integration is available for customers of Datadog Container Monitoring, who can set up the integration using documentation or sign up for a 14-day free trial.

In this article, the author discusses best practices for protecting different types of data that flow through cloud applications' components. The article first provides an overview of how modern applications process and manage data, highlighting common weaknesses and security threats to these systems. It then delves into specific measures organizations can take to secure their data, such as encrypting it at rest and in transit, protecting credentials with secrets managers, controlling access to data across services and accounts, and monitoring data sources and their activity. The author emphasizes the importance of understanding when sensitive data enters different states (at rest, in transit, or in use) and implementing appropriate security measures accordingly. Additionally, the article discusses the role of management planes in enhancing data security through secrets management and authorization and access controls, as well as providing adequate visibility into data activity. The author concludes by recommending Datadog as a tool to monitor an application's data activity and create additional layers of security for data flowing in cloud environments.

Encrypt sensitive data at rest and in transit to protect it from unauthorized access. Organizations should implement measures across their cloud application's data, control, and management planes to mitigate security threats and ensure the confidentiality, integrity, and availability of their data. This includes using encryption methods like symmetric and asymmetric encryption, as well as secrets managers to securely store and distribute sensitive credentials. By implementing these best practices, organizations can build an effective data security strategy that protects their cloud-native applications from data breaches and exposure.

OpenTelemetry (OTel) is an open source observability framework that provides APIs, SDKs, and tools for instrumenting cloud-native applications and services. It enables the collection of metrics, logs, and traces from various sources and routes them to different backends. However, OTel alone cannot help analyze this data or correlate telemetry from different parts of a stack. To achieve full visibility, it needs to be paired with a monitoring platform that allows visualization of telemetry data across an application's frontend and backend. Datadog's APM and RUM integration already provides complete visibility into the journey of an API request issued from an application through the entire backend stack by automatically connecting distributed traces to RUM resources captured from web and mobile apps. With the support for W3C and B3 trace headers in Datadog's Browser and Mobile RUM SDKs, users can now bring this full-stack visibility to their OTel-instrumented applications with minimal added configuration. This enhanced OTel header support enables users to gain full-stack visibility into OTel-instrumented apps, pinpoint the root cause of increased latency and failed requests, and streamline collaboration between frontend and backend teams. By linking these traces to related resources directly within RUM user sessions, users can leverage powerful RUM features for quick troubleshooting and effective root cause analysis for their OTel-instrumented apps. With the end-to-end correlation of user actions, requests, and backend traces provided by RUM, users can easily investigate issues by working their way from frontend impacts to backend root causes without ever leaving the page. This helps identify which backend services are responsible when an OTel-instrumented app is responding slowly or failing altogether. To start correlating OTel-instrumented traces across APM and RUM, users can follow the documentation provided by Datadog to collect traces from OTel-instrumented apps in Datadog, enable the Datadog APM and RUM integration, and add W3C and B3 trace context to their RUM SDKs. Alternatively, new users can sign up for a 14-day free trial of Datadog.

Datadog has integrated its APM and RUM with OpenTelemetry (OTel), an open-source observability framework, to provide full-stack visibility into OTel-instrumented apps. This integration enables users to collect metrics, logs, and traces from various sources, route them to backends, and analyze data across the application's frontend and backend. With this integration, users can gain full-stack visibility, pinpoint root causes of issues, and troubleshoot through a single lens. The integration provides end-to-end correlation of user actions, requests, and backend traces, allowing users to easily investigate issues by working their way from frontend impacts to backend root causes.

The OpenTelemetry (OTel) project aims to provide vendor-neutral standards and tools for collecting telemetry data from any source in an environment and sending it to any backend. Datadog, a company that offers a unified observability platform, has been actively contributing to the OTel project since its inception. Over the past six months, Datadog has built a dedicated team to develop support for OpenTelemetry, including general availability of OTLP ingest and exporters, as well as improvements to trace context support and infrastructure monitoring. The company is also preparing to roll out support for OTel runtime metrics and additional improvements to its OTel product and feature support. Datadog remains committed to supporting OpenTelemetry and actively engages with the community to contribute back upstream to improve the project for everyone.