April 2024 Summaries

The Datadog Agentless Scanning feature provides visibility into security risks and vulnerabilities within hosts, containers, and serverless functions without requiring agents to be installed on every host. This enables comprehensive monitoring of entire cloud accounts in minutes, including misconfigurations, identity risks, and other capabilities provided by Datadog Cloud Security Management (CSM). The Agentless Scanning is now generally available for AWS cloud environments and offers two deployment options: cross-account scanning or same-account scanning. It works by analyzing hosts and Lambda functions to identify associated packages and running containers, and then checks these against a list of known open-source vulnerabilities. The feature can be used in conjunction with the Datadog Agent to gain additional context through CSM, including suspicious file, process, and network activity signals related to cloud workloads.

The text discusses the importance of managing machine learning (ML) platforms and monitoring their performance to ensure accurate predictions and optimal model deployment. It highlights the need for data preparation, model training, evaluation, and deployment in an ML workflow, and emphasizes the role of managed platforms such as Amazon Sagemaker, Azure Machine Learning, and Google Vertex AI in simplifying and expediting each step of the workflow. The text also discusses key metrics to monitor, including rate, error, and duration (RED) metrics, and provides recommendations for optimizing model efficacy through these metrics. Furthermore, it highlights the importance of monitoring model performance, resource utilization, and data quality issues to ensure reliable and accurate predictions.

Thomas Sobolik and Léopold Boudard highlight the challenges of maintaining the functionality of machine learning (ML) models in production, where subtle trends in the environment can radically alter their behavior. Unlike conventional applications, ML models require continuous evaluation to ensure accuracy and effectiveness. The authors discuss key metrics and strategies for monitoring model performance, including identifying evaluation metrics, monitoring proxy metrics like data and prediction drift, detecting data processing pipeline issues, directly evaluating prediction accuracy, and monitoring prediction and data drift. They also introduce the concept of drift, which refers to changes in the input data distributions or the model's predictions over time, and provide tools such as managed ML platforms and data analytics tools to detect these changes. The authors emphasize the importance of continuous evaluation and retraining of ML models to ensure they remain stable and performant over time.

Google Cloud's Private Service Connect (PSC) enables secure access to Google Cloud services and third-party partner services within Virtual Private Clouds, keeping network traffic secure and reducing egress costs. Datadog is a Google Cloud partner with PSC, allowing customers to send high-volume network logs, load balancer logs, and VPC Flow Logs privately to Datadog, reducing data transfer costs and providing cost-effective analytics and long-term retention options. The Datadog-PSC integration enables monitoring of services consumed or produced via PSC, tracking network traffic flow, and optimizing costs and resources, with features such as alerts for bytes, dropped packets, and used IP addresses, allowing customers to fine-tune their service performance and detect anomalies.

Datadog`, a company that provides monitoring and analytics tools, uses `gRPC` (a Remote Procedure Call framework) to enable efficient communication between its distributed systems. Implementing a networking solution for such large applications poses several challenges, including scalability, load balancing, fault tolerance, compatibility, and latency. Datadog started using gRPC due to its integration with Protocol Buffers (protobuf), which allows developers to easily create bindings for their services in various languages. As Datadog grew, they discovered that gRPC's built-in client-side load balancing features were key to scaling their backend. However, they also encountered challenges such as silent connection drops and IP recycling issues. To address these problems, they set the `round_robin` policy for load balancing on the client side, leveraged TLS to handle IP recycling, set `MAX_CONNECTION_AGE` to force gRPC to re-resolve from DNS, and configured the `keepalive` feature to mitigate silent connection drops. Proper monitoring of services is also crucial in identifying issues such as load imbalance and failed transmissions.

Datadog SCA is a software composition analysis tool that helps teams understand their software's dependencies and the security implications of using open source code. It uses both static analysis and runtime analysis to monitor for vulnerabilities throughout the code's lifecycle, providing efficient, end-to-end vulnerability detection and management. The tool uses frameworks such as the Common Vulnerability Scoring System (CVSS) and the Exploit Prediction Scoring System (EPSS) to prioritize remediation efforts, and provides clear steps for remediating detected vulnerabilities. Additionally, Datadog SCA's source code integration helps detect vulnerabilities early in the development cycle, while its service integration adds a layer of protection by observing running services and providing visibility into impacted infrastructure. The tool also offers Quality Gates to manage the risk of new and existing vulnerabilities without slowing down development cadence. By incorporating severity and risk scores, threat activity data, and runtime context, Datadog SCA gives teams complete visibility into each vulnerability's impact on their services, enabling them to focus remediation efforts effectively.

Yassir Ramdani and Austin Lai from Datadog have been using SwiftUI since the company's inception, starting with prototyping and internal tools, then adopting it for small features and eventually building full products. They introduced APM Services with data visualization in 2022, which led to developing DogGraphs, an internal graphing library to bring Datadog's data visualization to iOS using native technologies like Swift and SwiftUI. The team identified performance bottlenecks, fixed them, and measured the results, optimizing the library's performance by reducing unnecessary body evaluations, batching shapes, splitting views into smaller ones, removing time-consuming processes from view bodies, eliminating dependencies not needed by views, and minimizing calls to `@Published` properties. DogGraphs is now used in several products across the Datadog mobile application and mobile widgets, providing a native, interactive data visualization experience with ease of use through a flexible API and fast rendering.

Datadog Observability Pipelines provides a smooth and affordable migration path to Datadog Log Management by enabling users to archive a copy of their logs in low-cost storage while continuing to send logs to their existing log solution. This allows organizations to prepare historical logs for later use in the Datadog platform without disrupting their workflows, making it easier to maintain visibility into their environment before, during, and after the transition. Observability Pipelines offers a reliable and cost-efficient way to perform this critical preparatory step, enabling users to send a copy of their logs to their own low-cost data store and later rehydrate these archived logs in Datadog to ensure uninterrupted observability throughout the migration process.

Datadog Observability Pipelines is a tool that helps organizations manage their log data by pre-processing it before routing it to higher-cost indexed storage solutions, allowing them to control log volumes and optimize the value of their log data while staying within budget. By applying filters, sampling, editing, and deduplication to their source log data, teams can amplify the signal within the noise and focus on more valuable log data. Additionally, Observability Pipelines provides granular control over logs through its five processor types, including filter, sample, quota, and dedupe processors, which enable teams to define rules for what data to drop and what to send along to premium storage. By implementing these features, organizations can ensure budget compliance and optimize the value of their log data.

Datadog Observability Pipelines is a solution that helps organizations aggregate, process, and route their logs to multiple destinations for various use cases such as security, auditing, log management, and long-term storage. It enables teams to send the same logging data to two destinations with minimal configuration and operational disruption, making it easier to trial new solutions and migrate to shared platforms without disrupting existing workflows. By using Observability Pipelines, teams can simplify their log management, reduce vendor lock-in, and minimize the cost of trialing and onboarding new solutions.

Datadog Observability Pipelines provides an on-prem solution to automatically redact sensitive information within your infrastructure, helping you protect confidentiality and comply with data protection laws. The Sensitive Data Scanner integration detects and redacts standard patterns in logs, including secrets and credentials, credit card numbers, personally identifiable information, and network and device information. To improve accuracy, users can define keyword dictionaries and configure secondary validations for credit card numbers. Additionally, custom scanning rules can be created to identify unique sensitive data not covered by OOTB rules. The Observability Pipelines Worker runs inside your infrastructure, scrubbing logs in real-time, allowing you to safely ship redacted logs to external destinations.

Datadog Observability Pipelines is a log management solution that enables organizations to take control of their log volume, process them efficiently, and derive measurable value from them. With its preconfigured templates and granular configuration options, Datadog Observability Pipelines allows organizations to quickly design pipelines, customize log processing, monitor pipeline components, and gain flexibility and control over their data. The solution helps organizations strike a balance between maximizing control, minimizing operational costs, and maintaining visibility, while also protecting sensitive data and avoiding vendor lock-in.

We've redesigned the navigation of our platform to help users quickly find what they need. The new layout organizes features in a way that reflects how most users interact with Datadog, placing core features at the top and bottom of the sidebar for easy access. Users can also quickly pivot to different areas of the platform using the search bar or recently accessed pages. Features are organized by product area, allowing teams to easily find relevant tools for their day-to-day needs. The redesign prioritizes readability, with updated colors, contrast, and space for favorites to make it easier to scan the sidebar. With this new navigation, users can navigate our platform more intuitively, explore features quickly, and take action on issues faster.

Datadog recently hosted its latest Datadog Summit in London, celebrating the growth of their community over the past six years. The event featured talks from product managers and customers, hands-on workshops, product demos, and interactive challenges. Speakers discussed topics such as leveraging observability for enhanced IoT development at Electrolux, security at the speed of cloud, serving up loyalty one dish at a time with Flipdish, improving user experience through frontend engineering, managing complex distributed payment systems across 185 locations, what constitutes an anomaly in Watchdog, and Aviva's journey to carbon net zero. The event aimed to share knowledge and foster collaboration among Datadog users and developers, looking forward to meeting them at future summits.

Datadog has released Bits AI, a generative AI-powered DevOps copilot that provides a conversational UI to explore observability data using natural language. This tool solves common pain points for DevOps teams and SREs by providing auto-generated incident summaries, responding to natural language queries to surface key information, and performing incident management tasks directly from Slack. Bits AI streamlines troubleshooting by leveraging generative AI for real-time insights into ongoing incidents, enabling teams to quickly understand the context and scope of an incident, reduce mean time to resolution (MTTR), and focus solely on remediation.

The Misconfigurations Overview dashboard in Datadog Cloud Security Management (CSM) helps organizations understand their compliance posture and identify the most impactful steps to improve it. The dashboard surfaces misconfigurations mapped to PCI DSS, SOC 2, and other popular compliance frameworks, providing out-of-the-box dashboards to measure progress over time for AWS, Azure, and Google Cloud resources. It enables teams to prioritize misconfigurations, track remediation progress, and understand the underlying causes of changes in their Posture Score. The dashboard also provides insights into trends in misconfigurations across different environments and services, enabling organizations to focus on high-risk areas and improve their security posture.

Achieving and maintaining compliance in the cloud poses significant challenges for organizations due to the complexity of safeguarding sensitive data and ensuring infrastructure is correctly configured. Datadog Cloud Security addresses these challenges by providing tools to monitor compliance trends, identify and address misconfigurations, and visualize compliance progress over time for major cloud providers like AWS, Azure, and Google Cloud. The Misconfigurations Overview dashboard enables organizations to prioritize and remediate misconfigurations, offering insights into compliance posture changes and trends that affect posture scores, which are calculated based on the severity and frequency of compliance failures. This tool also helps organizations track where misconfigurations occur or are resolved within their environment, facilitating targeted remediation efforts. Datadog Cloud Security's features, including tagging and report pages for compliance frameworks, aid in understanding and improving an organization's compliance posture, and the platform is accessible through a 14-day free trial for new users.

Datadog Cloud Security Management (CSM) for Windows is designed to provide complete visibility into the security posture of cloud-native and on-premises workloads. It enables administrators to detect security threats, respond to incidents promptly, optimize resource utilization, and maintain system integrity and reliability. CSM for Windows allows users to monitor suspicious activity on their Windows servers in real-time, contextualize security signals with observability data, and quickly remediate issues. By leveraging the Datadog platform, organizations can gain visibility into their attack surface, detect threats more effectively, and respond to incidents faster, ultimately enhancing their overall system reliability and security.

Datadog's Data Streams Monitoring (DSM) provides detailed visibility into event-driven applications and streaming data pipelines, allowing users to track performance and improve it. DSM offers a topology map that visualizes all pipeline components and their dependencies, including every service and queue, from end to end. This map shows throughput for each service as well as the time messages spend in queues, highlighting issues such as slow-performing services. Users can select any service or queue from the map to open a side panel of key health and performance metrics, which helps troubleshoot issues. DSM also provides supplementary SQS-specific metrics within its topology map, allowing users to configure monitors to alert on these metrics if they breach expected limits. The integration enables users to track throughput and latency, ensuring the health of their event-driven applications and meeting SLAs.

The .NET continuous profiler builds on its previous parts by covering exceptions and lock contention profiling. The profiler fetches details on thrown exceptions, including exception type, thread ID, and message, using the `ICorProfilerCallback::ExceptionThrown` implementation and the `FrameStore`. To get the exception message, it uses a combination of metadata import, class layout, and field offset calculations. For lock contention, the profiler listens to CLR events, such as `ContentionStart` and `ContentionStop`, but these events are not yet supported for older versions of the .NET runtime. The profiler samples exceptions and lock contention events to minimize performance impact, using techniques like reservoir sampling and discrete PID controllers to control subsampling. It then upscales sampled values to provide estimates of real numbers, applying proportional ratios based on counts or durations. The goal is to provide a good statistical distribution of exception contexts and lock contention duration, allowing for accurate troubleshooting and analysis.

Mallory Mooney and Christina Berardi discuss the importance of APIs in modular application development, highlighting their role in both internal services and public-facing datasets. They emphasize that APIs are a top target for threat actors due to their dual nature. The authors outline three types of threat actors: opportunistic, sophisticated, and internal, each with distinct motives and vulnerabilities. Opportunistic attackers exploit security gaps in publicly accessible APIs, while sophisticated actors use advanced social engineering techniques to gain access to intellectual property and data. Internal threat actors, often disgruntled employees or contractors, target misconfigured internal APIs. The authors identify API inventory management, authentication and authorization controls, and resource management as primary security threats, including poor inventory management, inefficient authentication and authorization controls, and unrestricted access to resources. To address these risks, teams should document and categorize their APIs using standards like OpenAPI, implement strong authentication mechanisms, and enforce the principle of least privilege. By understanding these vulnerabilities and taking proactive measures, organizations can enhance their API security strategy and protect against various types of attacks.

Datadog Cloud Cost Management now supports Google Cloud, providing total cost observability for engineering teams. This platform empowers engineers to take ownership of their costs by unifying cost data with telemetry data from their environment, allowing them to understand how changes impact costs and uncover savings opportunities. With granular visibility into cost data, engineers can identify underutilized resources, clean up abandoned resources, and avoid cost overruns. By incorporating cost data into the same platform as observability data, teams can collaborate more efficiently to optimize cloud spend without impacting their customer experience.

Datadog's new Kubernetes Security Posture Management (KSPM) capabilities are now available in Datadog Cloud Security Management (CSM), helping organizations proactively strengthen their Kubernetes security posture by benchmarking their environment against industry best practices. The solution provides 100+ built-in detection rules, customizable security detections, and the ability to assess security posture against industry-standard frameworks like CIS. KSPM also enables users to monitor risk across Kubernetes deployments, create custom detection rules, and identify vulnerabilities in container images. With its ease of use and flexibility, Datadog's KSPM is designed to empower organizations to take proactive control of their Kubernetes security.

Datadog's Log Explorer now offers subqueries to facilitate easier log correlation across multiple sources, enabling organizations to quickly investigate bugs for remediation, gauge the impact of a security breach, and identify key users. Subqueries allow users to filter logs with nested queries, overcoming common challenges such as fragmented data and tedious manual processes. By using subqueries, teams can surface critical data needed to troubleshoot issues, understand user behavior, and respond to security threats in real-time.