July 2024 Summaries

Datadog's integration with Amazon MemoryDB provides visibility into the performance of in-memory databases, enabling users to optimize and troubleshoot issues. The integration offers an out-of-the-box (OOTB) dashboard that displays key cluster-level and Redis metrics, allowing users to gauge the overall health and performance of their database at a glance. By tracking host-level metrics such as memory usage and network traffic, users can understand their resource consumption, plan scaling, and preempt performance issues. The integration also provides alerts on active connections and compute usage metrics, enabling users to stay ahead of user-facing performance issues. Additionally, the integration offers visibility into MemoryDB memory and network performance metrics, including bytes allocated, freeable memory, swap usage, cache growth, active connections by node, and network packets in, packets out, and bytes-out. By leveraging these insights, users can optimize their MemoryDB database for high performance and prevent data loss or unexpected costs.

Datadog Cloud Cost Management is a tool designed to help service owners understand and manage the costs of their cloud-based services. It provides a holistic view of costs, allowing users to quickly grasp and flexibly analyze the total cost of each service. The tool breaks down all associated costs in clear terms, enabling users to take greater accountability and control as service owners. Cloud Cost Management simplifies cost analysis for containerized services that share resources by deriving precise costs of each workload. It also provides fine-tuned cost data alongside other service KPIs, allowing users to track and analyze costs over time and make informed decisions about optimization and scaling. By leveraging this tool, service owners can take total ownership of their services, making it easier to manage costs and ensure they align with business goals.

Datadog Observability Pipelines offers a comprehensive system for processing and managing logs from various IT infrastructure sources, converting them into standardized formats to enhance data analytics, security, and debugging. The platform introduces tools like the Grok Parser, which allows users to apply preconfigured or custom parsing rules to transform logs efficiently. Additionally, the Enrichment Table processor adds contextual data to logs for improved usability, while the Add Hostname processor embeds system information to aid in debugging. Furthermore, the GeoIP Parser integrates geographical data into logs, enhancing security and analytical capabilities. These features collectively enable users to streamline log management, ensuring logs are enriched, structured, and routed effectively for downstream applications, thereby supporting better security monitoring, data analysis, and operational troubleshooting.

The Continuous Profiler in PHP has been updated to include exception profiling, which aims to show the hidden costs of exceptions in hot code paths. The feature was implemented to highlight cases where exceptions are being used for flow control, revealing that a service was throwing 495,000 exceptions per 14.2 seconds due to using an incompatible library's comparator method, which throws exceptions when comparing values. This highlights the importance of using the right tools and libraries for the task at hand. The profiler also surfaces cases where exceptions are being thrown unnecessarily, such as configuration mistakes, and provides guidance on how to clean up these extraneous exceptions. Exception profiling is a sampling profiler that collects both uncaught and caught exceptions, providing valuable insights into performance bottlenecks in applications.

Tom Sobolik, Barry Eom, and Shri Subramanian discuss the proliferation of managed LLM services like OpenAI, Amazon Bedrock, and Anthropic, which have introduced possibilities for generative AI applications. However, introducing non-deterministic LLM services can increase the need for comprehensive observability, as debugging LLM chains can be challenging. The authors argue that tracing your LLM chains can help examine each step across the full chain execution to more quickly spot errors and latency and troubleshoot issues. They also explain how chains are essential for integrating LLMs into application workflows, helping structure interactions between the model and other components of the application. Chains extend LLM functionality to overcome limitations such as limited knowledge or ability to execute tasks not related to text generation. The authors discuss challenges and goals of monitoring LLM chains, including tracking code and request errors, pinpointing root causes, identifying hallucinations, and tracking token consumption. They also explain how instrumenting LLM chains presents unique considerations, requiring different approaches for instrumentation. Datadog's LLM Observability SDK can be used to instrument your LLM chains for debugging and performance analysis, allowing you to create trace spans within your code, add metadata and context, and monitor incoming requests within a central view.

Snowflake is a fully managed data platform that enables users to store, process, and analyze large volumes of data across their cloud environments. Datadog's Security Research Team has released an updated integration for Snowflake that converts many threat hunts into proactive detections, developed by detection engineers and tested by the internal security team. The new integration ingests additional data tables from Snowflake, making them available in Cloud SIEM, which provides out-of-the-box (OOTB) threat detections for Snowflake environments. These detections include Initial access, Persistence, Credential access, Defense evasion, Collection, Exfiltration, and signal correlation rules that combine multiple types of detections into a new rule. The team at Datadog has fine-tuned these detections using OOTB rule cloning, signal correlation rules, suppression rules, and workflows to adapt them for their environment. To get started with the new Snowflake detections in Cloud SIEM, users can configure log ingestion, review and customize detections, set up alerts and notifications, and monitor and refine their setup.

SSRF attacks are a growing concern in modern cloud applications, exploiting vulnerabilities in web applications and APIs to gain access to internal resources. These attacks can be difficult to detect due to their disguised nature, but monitoring API calls' timing and response, as well as analyzing user input, can help identify suspicious activity. Datadog's Application and API Protection (AAP) provides out-of-the-box detection rules to block SSRF attacks, offering a more efficient solution than traditional perimeter-based WAFs. To mitigate these threats, OWASP recommends several measures, including upgrading to IMDSv2, validating user-supplied URLs, and using robust security libraries.

The text discusses the growing adoption of containerized infrastructure and the emergence of container-native tools for managing CI/CD pipelines in these environments. It highlights several integrations between Datadog and popular container-native CI/CD solutions, including Argo Rollouts, Argo Workflows, ArgoCD, FluxCD, and Tekton. These integrations enable users to monitor deployment data, track workflow execution, troubleshoot CD pipelines, monitor continuous delivery processes, and track pipeline executions. The text emphasizes the importance of adapting monitoring strategies to gain complete visibility into container-native CI/CD pipelines and invites readers to sign up for a 14-day free trial with Datadog to achieve this goal.

The Essential Cloud Security Controls (ECSC) ruleset is a set of cloud security controls designed to help security teams prioritize their efforts and improve their security posture. The ECSC v2 includes an updated list of detections for each major cloud provider, including AWS, Azure, and Google Cloud, as well as support for Kubernetes. This allows cloud security teams to quickly identify which clusters they defend are at risk and work to remediate them. The ruleset is based on industry best practices, the risk of significant impact, and their potential to have prevented a known breach. It includes controls such as restricting public access to S3 buckets, disabling FTP deployments, and using secure authentication methods for Kubernetes API servers. The ECSC v2 also introduces support for Kubernetes, allowing cloud security teams to identify and remediate vulnerabilities in their containerized applications.

The text discusses the issue of alert storms in microservices architectures and provides techniques to reduce their impact. Alert storms occur when monitoring platforms generate excessive alerts simultaneously or in succession, causing confusion, delay incident response, and alert fatigue. The article recommends five techniques: mapping dependencies, using exponential backoff or service checks, scheduling downtimes, leveraging notification grouping and event correlation, and implementing automated remediation. These techniques help prevent alert storms by visualizing relationships between services, minimizing unnecessary alerts, and automating response actions. The text also highlights the benefits of implementing these techniques, including improved reliability, resilience, operational efficiency, reduced risk of unplanned outages, and enhanced user experience.

Foundation models, or large AI models, are crucial for advancing generative AI applications. However, existing large language models (LLMs) struggle with understanding observability metrics, which require processing numerical time series data and identifying trends, seasonality patterns, and anomalies. Dedicated foundation models for time series and structured data have the potential to complement general-purpose LLMs. An example of such a model is Toto, developed by Datadog, which achieves top performance on several open time series benchmarks and consistently outperforms existing models in key accuracy metrics. Toto is trained on nearly a trillion data points, including 750 billion fully anonymous numerical metric data points from the Datadog platform and time series datasets from Large-scale Open Time Series Archive (LOTSA). It excels in zero-shot forecasting and generalizes well to other time series domains. Toto outperforms existing foundation models and full-shot models on several benchmarks, showcasing its reliability and precision in forecasting capabilities. The model is still early in its development but holds promise for improving AI, ML, anomaly detection, and forecasting algorithms within the Datadog platform and powering products such as Watchdog and Bits AI.

The Datadog community gathered for their biggest event yet, DASH 2024, which took place in New York City over two days. The event featured numerous new product launches and updates, as well as breakout sessions, workshops, certification exams, one-on-one consultations, and a bustling expo hall. Breakout sessions were hosted by 49 speakers covering various topics, including Continuous Profiler and machine learning solutions. The Spotlight Theater showcased special sessions on AI, Remote Configuration, and Datadog On LLMs, while workshops covered topics such as Kubernetes, RUM, APM, and DevSecOps. The largest ever Datadog Partner Network presence was also present at the event, with 24 sponsors and 345 partner attendees participating in the third annual Partner Summit.

Microsoft announced several updates and features during Microsoft Build 2024, including new large language models and feature updates in Azure AI, the general availability of Azure Database for PostgreSQL Azure AI extension, previewed built-in vector database capabilities for Azure CosmosDB, and new features to increase developer productivity. These updates aim to help organizations deliver better AI experiences across on-prem, hybrid, and multi-cloud environments, while also improving monitoring capabilities for Azure services.