October 2023 Summaries

Cloudcraft, a cloud infrastructure modeling solution by Datadog, provides users with the ability to visualize, document, budget, and plan cloud architectures in real time, particularly for AWS and Azure platforms. It enables automatic generation of architecture diagrams, allowing users to identify issues, refactor designs, and estimate costs efficiently. Recent enhancements to Cloudcraft include support for new architecture objects and improved navigation between Cloudcraft diagrams and Datadog monitoring tools, along with a new pricing model offering a free trial of Cloudcraft Pro. The integration with Datadog facilitates access to telemetry data, enhancing the ability to track infrastructure health and performance. Cloudcraft's features are designed to save time by providing dynamic, editable diagrams that are easily exportable for collaborative use, compliance documentation, and financial analysis.

In this interview, Jeromy Carriere, SVP of Product Engineering at Datadog, shares his day-to-day responsibilities, which include planning cycles, execution cycles, people management, and process improvement. He discusses his journey to the position, highlighting his passion for observability and the importance of learning from mistakes. Additionally, he talks about his experience with the co-op program at the University of Waterloo and shares his hobbies outside of work, such as restoring old computers and driving race cars. Finally, Carriere discusses the evolution of the observability space and its future direction, emphasizing the importance of AI and Large Language Models in this field.

Datadog's Dynamic Instrumentation enables users to add logs to their code without restarting or redeploying applications, reducing the mean time to resolution (MTTR) and minimizing system disruption. This feature allows for easy capture of crucial data at runtime and analysis of app performance with enriched context. Users can create probes to add instrumentation to their code, access relevant files through source code integration, and set conditions for log generation. The logs captured via Dynamic Instrumentation are standard application logs that can be analyzed using tags and metadata within Datadog. This tool helps close observability gaps and troubleshoot performance issues with minimal user impact.

Rosa Trieu sat down with Jeromy Carriere, SVP of Product Engineering at Datadog, to discuss his journey to the role and share insights on how he approaches engineering, career growth, and innovation. Carriere emphasizes the importance of learning from mistakes, striking a balance between autonomy and accountability in teams, and fostering a culture of continuous improvement. He highlights Datadog's commitment to innovation, platform integration, and customer-centricity as key drivers of its success. The conversation also touches on his experiences with the co-op program at the University of Waterloo, which he credits with setting him on a successful path, and his passion for hobbies like restoring old computers and driving race cars. Throughout the interview, Carriere conveys his enthusiasm for the observability space and its potential to drive significant impact in industries.

The Pub/Sub-to-Datadog Dataflow template is an efficient method to route logs from across the Google Cloud ecosystem to Datadog. This serverless, fully managed framework enables automation and autoscaling of data processing tasks. By using this template, users can configure Dataflow jobs to pull processed logs and send them to Datadog Log Management for visualization, alert creation, and correlation with metrics from across the stack. The Pub/Sub-to-Datadog Dataflow template supports batch processing and enables customers using virtual private networks to access endpoints outside their VPC perimeter. This solution offers scalable ingestion while potentially lowering excess network egress costs.

The Datadog Service Catalog introduces Service Scorecards to consolidate knowledge of an organization's services and evaluate their performance, reliability, and ownership in a central location. Service Scorecards automatically assess each service against pass-fail rules in three categories: Production Readiness, Ownership & Documentation, and Observability Best Practices. This helps teams align with best practices, plan work to improve observability, communicate and collaborate with stakeholders and other teams. By tracking scorecards, managers can ensure that all teams are consistently using Datadog features for maximum visibility into their services. Service Scorecards also help identify areas for improvement by highlighting the rules with the highest and lowest scores. Additionally, they provide detailed information about failed rules and relevant documentation to assist teams in making revisions to align with the conditions of the rule. Overall, Service Scorecards contribute to improving the performance, reliability, and availability of services within an organization.

Google Cloud Dataflow is a serverless framework that enables automation and autoscaling of data processing. It allows users to collect, transform, enrich, and send log data from various sources to external sinks, such as Datadog. The Pub/Sub-to-Datadog Dataflow template provides a pre-built pipeline for routing logs directly from Google Cloud to Datadog, enabling efficient and scalable ingestion with reduced network overhead. This template supports batch processing and compression, allowing users to automatically process large volumes of log data without incurring excessive network costs. By using this template, users can leverage Dataflow to easily ingest logs into Datadog, view them alongside other logs, and create effective troubleshooting tools such as dashboards and alerts.

CloudNatix is a platform designed to monitor and optimize cloud infrastructure, offering features such as automatic configuration and optimization workflows for Kubernetes and VM workloads. These features are aimed at improving resource efficiency and reducing costs. Through a new partnership with Datadog, users can now integrate CloudNatix's data into Datadog's comprehensive observability platform, allowing them to track infrastructure usage and cloud spend in real-time. The integration facilitates users to configure optimizations directly within CloudNatix, addressing issues such as low resource utilization by applying automated solutions like cluster rightsizing. The collaboration enables users to receive alerts on infrastructure issues, optimize resources, and manage cloud costs effectively, with the integration available via the Datadog Marketplace.

This post discusses the challenges faced when investigating high response times in Java applications and introduces Datadog's Java wallclock profiler as a solution. The profiler helps improve request latency without making any code changes or even viewing the code. It covers topics such as transitioning from metrics to profiles, using async-profiler, implementing wallclock profiling, investigating latency, and considering cost implications. By using this profiler, developers can gain a complete picture of CPU time and understand latency better, leading to improved application performance.

Richard Startin`, a Senior Software Engineer at `Datadog`, shares his experience on using the Java wallclock profiler to improve request latency in Java applications. He highlights the limitations of traditional profiling approaches and introduces `async-profiler`, which he worked on to help address this issue. The article explains how `async-profiler` can be used to create a flame graph that shows a complete picture of CPU time, even if the bottleneck is not directly related to CPU usage. It also discusses the challenges of sampling threads in a high-thread-count environment and introduces a new thread filter approach that only samples threads with an active trace context set up by the tracer. The article concludes with examples of how to use `async-profiler` to investigate and fix performance issues, including increasing the heap size and adding more vCPUs.

Datadog Session Replay in Real User Monitoring (RUM) enables customers to capture and visually replay the web and mobile experience of their end users, allowing them to quickly find and address UX errors. To bridge gaps in saving session replays, a new feature called Playlists has been introduced. Playlists allow for efficient grouping and organization of session replays, making it easier to share and collaborate with teams. The Search Recordings page provides an overview of all sessions, allowing users to filter for specific sessions and save them to Playlists with one click. This feature helps improve UX workflow analysis and troubleshooting by identifying key behavioral patterns emerging from the UI.

Datadog Session Replay in Real User Monitoring (RUM) enables customers to capture and visually replay the web and mobile experience of their end users, allowing for easier troubleshooting and debugging. A new feature called Playlists is being introduced, which allows customers to group and organize their session replays, share them with teams, and streamline their workflows. Playlists can be used to identify lessons learned, formulate deliverables, and provide evidence to justify prioritization strategies. The Search Recordings page provides a convenient way to find and add important session replays to Playlists, allowing customers to quickly access and analyze behavioral patterns emerging from their UI. By using Playlists, customers can simplify their UX workflow and analysis, identify and resolve issues faster, and take steps to improve their end-users' experience.

Drew Csillag, a Staff Engineer at Datadog, has developed strategies for fostering collaboration and influence without authority to accomplish broad, cross-team projects. His approach involves persuading engineers to help out on projects for the greater good by talking in terms they care about, seeing if they're already inclined to help, finding ways to minimize their effort, offering to do something in return, and giving generous timelines. Drew's philosophy focuses on appealing to colleagues' interests and concerns while investing effort to make the work easier for them. By sharing his ideas and experiences, he helps other managers and personnel at Datadog learn new soft skills, become more effective in their roles, and nurture a positive work culture.

Scroll Maps is a new feature of Datadog Heatmaps that helps product managers, designers, and website developers understand how users interact with web pages. It leverages Session Replay data to show the percentage of users who scroll down a page and displays this information in various ways such as helpful metrics, mini scroll maps, and distribution graphs. Scroll Maps also offers new searchable facets for creating queries in RUM to compare different pages based on user engagement. This feature can be used to gain insights into user interactions and optimize a page or compare viewership trends across different pages.

Thomas Sobolik, a Staff Engineer at Datadog, has developed a philosophy for fostering collaboration and influencing colleagues to help with projects without manipulating or strong-arming them. His approach, called "influence without authority," emphasizes the importance of soft skills, sensitivity to developers' bandwidth and incentives, and investing effort to make work easier for others. By appealing directly to stakeholders' interests and concerns, minimizing their effort, offering something in return, and giving generous timelines, Drew is able to bring colleagues on board with his initiatives and support a healthy and productive work culture at Datadog. His ideas have been shared through internal talks and courses, aiming to help other managers and personnel learn new soft skills and nurture the organization's positive work culture.

Nicholas Thomson and Jamie Milstein from Datadog Product Analytics announce the launch of Scroll Maps, a feature that leverages Session Replay data to visualize how users interact with web pages. This feature helps product managers, designers, and website developers understand user behavior, identify areas for improvement, and optimize their applications to fulfill user needs. With Scroll Maps, teams can gain insights into user interactions, compare viewership trends across different pages, and troubleshoot problematic page elements to improve the overall user experience. The feature is now part of Datadog Product Analytics and is available to existing Datadog customers or those who sign up for a 14-day free trial.

OpenTelemetry (OTel) is an open-source observability solution that provides components to collect and communicate telemetry data from cloud-native applications and services. Datadog Agent now supports the ingestion of OTLP logs, in addition to traces and metrics, allowing teams to consolidate all OTel data through a single mechanism. This unified approach simplifies the telemetry data collection process and reduces administrative overhead. The Agent also offers advanced log processing capabilities, such as filtering, scrubbing, and multi-line aggregation, enhancing observability and ensuring high security of log data.

Cloud applications face challenges in identifying system failures due to their large scale and complexity. To proactively find potential points of failure within cloud infrastructure, engineers are turning to chaos engineering (CE). Security-focused CE verifies that systems respond appropriately to common threats, similar to red, blue, and purple teaming. Key steps for security-focused chaos engineering include questioning, identifying, experimenting, and detecting vulnerabilities or other security issues. Practical examples of SCE experiments for Kubernetes include identifying misconfigured API servers and assessing visibility into new resources. Tools like Datadog enhance chaos engineering experiments by automatically discovering common issues that leave systems vulnerable.

The OpenTelemetry (OTel) is an open-source, vendor-neutral observability solution that provides a suite of components to collect and communicate telemetry data from cloud-native applications and services. Datadog has announced support for the ingestion of OTLP logs, allowing teams to collect and export observability data from their environment to their preferred backend. This feature simplifies the telemetry data collection process, eliminates workarounds, and provides a unified approach to ingesting all OTel data through a single mechanism. The Datadog Agent now enables out-of-the-box support for all features of log processing pipelines, including automatic log parsing, enrichment, and trace-log correlation. Teams can configure their Datadog Agent to collect OTLP logs from applications instrumented using the OTel SDKs, and enjoy benefits such as unified telemetry ingestion, enhanced observability, robust re-trying logic, advanced log processing, optimized network costs, and ongoing support and performance improvements.

Mallory Mooney discusses the challenges of identifying system failures in modern cloud applications and introduces chaos engineering as a solution to proactively find potential points of failure within cloud infrastructure. Chaos engineering tests a system's resilience by deliberately injecting failures into various components, allowing engineers to gain a better understanding of how their applications function under stress. Security-focused chaos engineering verifies that systems respond appropriately to common threats, similar to red, blue, and purple teaming. The key components of security-focused CE include defining a steady state for your systems, asking questions about potential vulnerabilities, injecting failure, and monitoring results. Engineers can use threat modeling practices like asking high-level questions about business-critical workflows and user journeys to guide the process. They can then focus on individual services and resources that make up those workflows, identifying components such as API servers, workloads, pods, and containers. Security-focused experiments can take many forms, including controlled denial-of-service (DoS) attacks or attempts to spin up new resources with malicious code. The last step in the process involves detecting vulnerabilities or other security issues that surfaced as a result of the experiment, using monitoring and threat detection systems. Mallory Mooney also discusses practical ways to apply these steps to Kubernetes workloads, which are critical yet complex components of cloud infrastructure. She provides examples of experiments, such as identifying misconfigured API servers or assessing visibility into new resources, and highlights the importance of tools like Datadog in enhancing chaos engineering experiments by automatically discovering common issues that leave systems vulnerable.

API sprawl can be challenging as it involves managing numerous loosely connected APIs that are potential points of failure. To prevent issues with broken endpoints and maintain good user experience (UX), effective API testing is crucial. Datadog Synthetic Monitoring now integrates data from the API Catalog to help monitor API test coverage and identify any endpoint issues. This feature allows users to spot gaps in their testing, investigate failed tests, and take control of their API testing for better service availability and performance.

Datadog has introduced its API Catalog, a unified inventory of all APIs and their endpoints that leverages the company's distributed tracing capabilities. The tool aims to help manage and secure modern applications made up of thousands of loosely connected private and publicly exposed APIs. It provides complete transparency into an organization's API landscape, allowing teams to monitor endpoint performance and security, efficiently document and manage their APIs, and more. Datadog API Catalog helps solve challenges related to managing API sprawl by offering a centralized location for visualizing how each of the managed and unmanaged API endpoints are connected, finding orphaned and legacy APIs in production that lack proper documentation, understanding which API endpoints are seeing traffic across all services and environments, and quickly identifying poorly performing endpoints.

Datadog API Catalog is a unified inventory of all team APIs and their endpoints, providing visibility into the API landscape through Datadog's distributed tracing capabilities. This allows teams to discover their API landscape, monitor endpoint performance and security, establish API ownership and governance, improve API performance and reliability, detect and alert on API issues, and enhance overall API security posture. The catalog integrates with Datadog Synthetic Monitoring, Application Security Management (ASM), and other tools to provide a comprehensive view of APIs, enabling teams to manage standardized, approved, and production-ready APIs within Datadog, monitor their performance and reliability, and quickly identify who owns certain endpoints for faster triage during incidents.

Datadog Synthetic Monitoring has introduced an API Test Coverage feature that helps teams monitor and improve the quality of their APIs. This feature integrates with the API Catalog to provide a comprehensive view of endpoint testing, including test coverage metrics, error rates, and performance data. By leveraging this feature, teams can easily spot gaps in their testing, investigate issues, and take control of their API testing, ultimately delivering better performance and availability for their services.

Security Operations Centers (SOCs) are responsible for building comprehensive threat detection strategies in the constantly evolving world of cybersecurity. The level of security coverage is a key indicator of success, which depends on the breadth, depth, and accuracy of threat detection tools and workflows. Building adequate security coverage involves challenges such as defining the boundaries of coverage, ambiguity about where to start documenting attacks, and inherent biases in building security coverage. To address these challenges, recommendations include generalizing the majority of detection rules and mapping them to industry-standard frameworks and models, preprocessing telemetry data to create generalized rules, and using multiple industry-standard sources like MITRE ATT&CK and PCI compliance frameworks for better visibility into a threat actor's entire attack path.

In this article, best practices are discussed for creating efficient detection rules using the Datadog Security Platform. These rules help detect potential threats to applications in real time by querying ingested logs for key activity or changes in an environment. The article covers building queries with sufficient granularity, customizing security signal messages, and fine-tuning signals to reduce noise through suppression lists. It also briefly explains how Datadog's out-of-the-box detection rules work and provides examples of each best practice.