January 2023 Summaries

The text discusses the process of optimizing the performance and efficiency of the Datadog Agent, a tool that processes large amounts of data quickly with minimal CPU usage. By profiling the agent using tools provided by Golang, the team identified the bottleneck in the part of the agent responsible for computing unique keys for every metric received. They then implemented several strategies to improve performance, including specialization, better raw performance through micro-benchmarks and a new hash set implementation, and a redesign of the context generation algorithm. The results showed significant improvements in processing speed and reduced CPU usage.

The Datadog Agent aims to process a large amount of data quickly while using minimal CPU. The team identified the bottleneck in the metric context generation algorithm, which computes a unique key for every metric received by the DogStatsD server. By leveraging the Go runtime's tools and profiling, they were able to pinpoint the sorting algorithms as the performance bottleneck. To improve performance, they specialized the sorting algorithm based on the number of tags, used micro-benchmarks to test various hash implementations, and implemented a custom hash set for deduplicating tags. The new design removed the need for sorting while maintaining efficient deduplication. Validation through micro-benchmarks showed significant performance wins, with the new algorithm spending less time computing metric contexts. The CPU flamegraph revealed reduced CPU usage in the Agent's network traffic processing, and dashboards displayed a 40% decrease in CPU usage on average, using fewer cores to process the same number of metrics. The team hopes this article demonstrates how beneficial performance optimizations can be, from simple to complex ones, and invites readers to join them in solving Agent performance problems.

The OpenTelemetry (OTel) project is an open source initiative that aims to provide vendor-neutral standards and tools for collecting telemetry data from any source in the environment and sending it to any backend. Datadog, a platform provider, has been actively supporting OTel since its inception and has made significant investments in improving its support for the standard. The company has built a dedicated team to develop Datadog's support for OpenTelemetry and has released several features, including the Datadog Exporter for the OpenTelemetry Collector, general availability of OTLP ingest in the Datadog Agent, and improved support for OTel histograms. Additionally, Datadog is currently preparing to roll out support for OTel runtime metrics and other improvements. The company's goal is to provide a first-class product and customer experience for OTel users and continues to invest in interoperability and accessibility of its products for OpenTelemetry users.

Organizations often use distributed tracing in Datadog APM to monitor the performance of their Kubernetes applications. However, as teams grow and new applications are added, manual configuration becomes impractical. To address this issue, a workflow has been developed that automates instrumentation, allowing for the configuration of tracing for Kubernetes applications without changing or rebuilding the application image. This is achieved by injecting Java, Node.js, and Python tracing libraries directly into the pod at runtime via the Datadog Admission Controller, which only requires writing a single annotation in the application's YAML file. This method does not require altering application code or pushing new container images, thus avoiding the CI/CD pipeline that would otherwise be necessary for building and deploying a new image. As a result, tracing can be set up quickly without any downtime.

The text discusses the challenges of manually configuring tracing in Kubernetes applications and introduces a new feature that automates instrumentation, enabling developers to configure tracing without changing or rebuilding their application image. The Datadog Admission Controller allows for library injection, which injects Java, Node.js, and Python tracing libraries directly into pods at runtime via annotations in the YAML file. This method avoids the need to rebuild or redeploy the application image, reducing operational overhead and enabling faster time-to-value for APM. The feature also enables unified service tagging, which automatically cross-references traces, logs, and metrics through common reserved tags, simplifying troubleshooting and aggregation of services.

HashiCorp Boundary is a tool designed to manage secure remote access to applications and infrastructure, operating under a Zero Trust model to ensure identity-based authorization without exposing network credentials. Recently launched as a fully managed service on the HashiCorp Cloud Platform (HCP), it integrates with tools like Vault for enhanced secret management. The integration with Datadog offers comprehensive monitoring capabilities, including real-time audit log streaming and a dashboard for tracking user sessions and performance metrics, such as active connections and request latencies. This monitoring is crucial for maintaining the health and performance of Boundary instances, particularly in large-scale, multi-cloud environments. Datadog's integration also includes alerting features for monitoring the health of Boundary controllers and investigating potential access policy misconfigurations through event logs. This setup provides users with detailed insights into infrastructure access and supports seamless management through additional integrations with Terraform and Vault.

Automatic Faulty Deployment Detection is a feature of Datadog APM that uses machine learning algorithms to spot faulty deployments within minutes and reduce mean time to detection (MTTD). It compares the performance of each new version of a service with its previous versions to identify new types of errors or increases in error rates introduced in a deployment. The tool provides details about affected services, including error types, error rates, request rates, and latency metrics for each deployed version. Automatic Faulty Deployment Detection also enables users to troubleshoot faulty deployments quickly by exploring the service's traces and collaborating with their team through notebooks or incident management processes. Additionally, it allows users to create alerts that automatically page them if a release appears to be faulty, ensuring rapid response to errors and maintaining both development velocity and service quality.

Datadog's APM Security View, part of their Application Security Management (ASM), is a comprehensive tool designed to help businesses efficiently detect, mitigate, and remediate security incidents within their applications. It integrates security data into the Service Catalog, enabling teams to identify services exposed to risks, understand vulnerabilities, and respond to malicious activity effectively. By displaying vulnerabilities in real-time, not in outdated code branches, and categorizing them by severity, APM Security View prioritizes remediation efforts. Additionally, it generates security signals for anomalies that may indicate attacks, providing detailed insights and recommended actions for incident responders. With features like distributed tracing and flame graphs, it offers a deep understanding of attack propagation, helping teams to quickly pinpoint and fix vulnerabilities in their code. ASM leverages existing APM libraries, making it easy for current APM users to gain full visibility and reduce their mean time to resolution for security threats.

Datadog is committed to fostering employee growth and development through collaboration and hands-on experience. The company offers opportunities in both commercial and enterprise sales, with a focus on identifying customer needs and promoting the value of their product. Sales Development Representatives (SDRs) undergo comprehensive training before progressing along a clearly defined career path. Employees are encouraged to seek personal development from leadership and explore different roles within the company. Datadog supports various career advancement options, including management and individual contributor positions, with promotional opportunities typically occurring after 12-15 months of performance. The company's emphasis on open communication and feedback helps ensure employee satisfaction and overall growth.

At Datadog, employees are empowered to grow and develop their careers in ways that fulfill their professional goals. The company's culture of collaboration allows individuals to seek personal development from leadership and explore different career paths, processes, and roles through hands-on experience. Sales Development Representatives (SDRs) at Datadog undergo comprehensive tech sales training to equip them with knowledge and skills to connect with quality prospects and nurture leads. One SDR, Maggie Valdez, was promoted to a Commercial Account Executive role after just eight months of learning and adopting sales techniques and skills. Another employee, Vanessa Martella, became a leader on the ground in Amsterdam after being promoted to Senior Account Executive, while Soul Song transitioned to a different role on the Enterprise Sales team, citing Datadog leadership's support for career growth. The company promotes open communication and feedback as a primary goal, ensuring that all team members feel satisfied and empowered to make career decisions that work best for them.

Organizations are rapidly expanding their cloud environments, which increases the risk of compliance issues and potential data breaches due to misconfigurations. To address these challenges, Datadog Cloud Security Management (CSM) aids governance, risk, and compliance (GRC) efforts by offering a centralized platform for tracking compliance posture and automating audit evidence collection. Recently, Datadog CSM has expanded its posture management capabilities to include Google Cloud, complementing its existing support for AWS and Azure. It provides tailored reports, detects deviations from compliance controls, and correlates findings with other security issues, enabling Security and DevOps teams to understand and mitigate threats efficiently. Datadog CSM supports industry-standard compliance frameworks and allows for a comprehensive view of cloud resources' security states, helping organizations ensure their cloud environments adhere to the latest security standards.

VMware's vSphere and Tanzu Kubernetes Grid (TKG) allow enterprises to combine the economic advantages of virtual machines with the agility, portability, and scalability provided by Kubernetes. Datadog now supports monitoring TKG clusters deployed on vSphere as well as their underlying VM resources. The new out-of-the-box dashboard provides a comprehensive overview of the entire TKG and vSphere environment, enabling users to monitor key data on TKG clusters and host VMs in real time. Additionally, dedicated overviews for pods and containers provide detailed visibility into their states and performance, allowing users to optimize and troubleshoot their vSphere environment's Kubernetes resources. The vSphere overview leverages metrics and events to provide critical visibility into the VMs and bare-metal hypervisors that underpin TKG environments.

Datadog now supports monitoring Tanzu Kubernetes Grid (TKG) clusters deployed on vSphere as well as their underlying VM resources. The new support comes with an out-of-the-box (OOTB) dashboard and base configurations that enable users to start monitoring TKG VMs immediately. Datadog's vSphere integration provides real-time metrics and events for both TKG clusters and their underlying vSphere hosts and VMs, offering a holistic view of the entire environment. The OOTB dashboard features key data on TKG clusters and host VMs via the vSphere Containers map and the TKG event stream, allowing users to monitor container activity, resource utilization, and performance in real-time. Additionally, the dashboard provides dedicated overviews of TKG pods and containers, enabling users to oversee, optimize, and troubleshoot their vSphere environment's Kubernetes resources in a single pane of glass. The integration also offers visibility into vSphere resources, including VMs and bare-metal hypervisors, providing critical insights into resource utilization, storage capacity, and event tracking. By leveraging these features, organizations can continuously optimize their virtualized and containerized resources, rapidly troubleshoot issues, and make the most of their usage of TKG on vSphere.

Datadog has introduced Online Archives, an always-on log warehousing solution that allows users to retain and search all of their log data for 15 months or more at the same cost as indexing data for one month. This feature enables historical analysis and investigations, effortless configuration and data exploration, and adherence to federal regulations such as Executive Order 14028's requirement for critical infrastructure owners to retain logs for 12 months in active storage and 18 months in cold storage. Online Archives is available in all Datadog regions including AWS GovCloud and can be accessed by installing the 1-click AWS integration or reaching out to a Datadog representative.

InsightFinder is a SaaS platform that uses AI-backed predictive analytics to forecast and prevent production incidents by identifying hidden correlations in application metrics, logs, and events. By integrating with Datadog, users can visualize predicted incidents using an out-of-the-box dashboard and send InsightFinder's incident predictions to Datadog Incident Management for a coordinated response. This integration helps preemptively address potential outages and maintain the health of applications. To start monitoring InsightFinder predictions with Datadog, sign up for a trial in the Datadog Marketplace and install the free integration.

InsightFinder is a SaaS platform using AI-backed predictive analytics to predict and prevent production incidents. By integrating with Datadog, users can identify hidden correlations in application metrics, logs, and events to address issues before they cause customer impact. After signing up for a trial and installing the integration, users can visualize predicted incidents on an OOTB dashboard and track patterns over time to determine root causes of incidents. Analyzing high volumes of predicted incidents can indicate unhealthy performance, while sending incident predictions to Datadog Incident Management enables teams to get ahead of potential outages and make adjustments before they occur. By using InsightFinder with Datadog, users can promote their branded marketing tools as part of the Datadog Partner Network.