May 2022 Summaries

Amazon RDS (Relational Database Service) has been widely used by companies to manage and scale databases for web and mobile applications. However, high-scale applications with thousands of clients can consume significant resources when maintaining direct connections between the database and application. To address this issue, AWS introduced RDS Proxy as an intermediary between the application and database, enabling pooling and sharing of connections from application clients rather than creating new ones for every query. This improves database efficiency and application scalability while also enhancing security by providing access through Identity and Access Management (IAM). Datadog's Amazon RDS Proxy integration and out-of-the-box dashboard provide visibility into client connections, query throughput, and other key metrics to help users troubleshoot connection errors, identify when to scale connections, and ensure optimal proxy performance and connection reusability.

The new Helm integration in Datadog allows users to monitor the availability and status of Helm-managed applications deployed in Kubernetes clusters. With this integration, users can visualize the status of their Helm releases using an out-of-the-box dashboard that displays data about Helm releases, including failed and successful installations and upgrades. The integration also includes a service check that automatically detects if any Helm releases are in a failed state, along with an out-of-the-box monitor to notify users when a release fails. This helps users quickly detect and troubleshoot issues affecting the performance of their services. To enable the Helm integration, set the Datadog Helm chart's datadog.helmCheck.enabled and datadog.helmCheck.collectEvents parameters to true. The integration requires Datadog Agent version 7.36.0+ and Cluster Agent version 1.20.0+.

Datadog has integrated its monitoring platform with Helm, allowing users to monitor the status of their Kubernetes applications and receive notifications when issues arise. The integration provides a dashboard that displays data about Helm releases, including release counts, event history, and timeseries graphs. Users can also enable monitors to notify them of changes in their Helm environment, such as failed releases or service checks. To use this feature, users must set specific parameters in the Datadog Helm chart and upgrade to certain versions of the Datadog Agent and Cluster Agent. The integration provides deep visibility into Helm releases alongside monitoring data from other parts of the orchestrated environment.

Amazon RDS (Relational Database Service) is a popular managed database service that has been used by companies to set up, operate, and scale databases for web and mobile applications. However, modern high-scale applications can have tens of thousands of clients, which can lead to maintenance issues and higher costs when paying per database instance. To solve this problem, AWS launched RDS Proxy, an intermediary between the application and database that pools and shares connections from application clients, improving database efficiency and application scalability. Datadog's Amazon RDS Proxy integration provides visibility into client connections and query throughput, allowing users to quickly troubleshoot connection errors, identify when to scale connections, and ensure optimal proxy performance and connection reusability. The integration also helps users monitor key metrics such as connection setup failures, database connection setup failures, target group availability, and pinned connections, enabling them to identify issues with authentication, security, and database instance availability, and optimize their RDS Proxy configuration for better performance and scalability. With Datadog's Amazon RDS Proxy integration, users can get started by monitoring key metrics from their proxies alongside other AWS infrastructure and services, troubleshoot database connection errors, understand when to scale connections, and ensure proxy performance and connection reusability, all from a single pane of glass.

PlanetScale, a serverless, MySQL-compatible database platform powered by Vitess, offers features like branching, non-blocking schema changes, and built-in connection pooling to enhance development velocity. The recent integration with Datadog provides users with deep visibility into their database activities, allowing them to monitor queries, understand storage costs, and track trends in database activity. The Datadog integration offers a comprehensive dashboard with features such as Cumulative Query Count and Time Breakdown graphs, Storage Breakdown graphs, and detailed insights into rows written per table. This integration helps users optimize database usage, manage costs effectively, and identify potential issues, thereby enhancing their ability to meet users' needs as they scale.

The KubeCon + CloudNativeCon Europe 2022 conference, held in Valencia, Spain, highlighted key topics such as software supply chain security, platform engineering, autoscaling and bin packing, and unconventional use cases for Kubernetes. Datadog participated as a platinum sponsor, showcasing its latest updates to Kubernetes monitoring solutions. The event emphasized the growing adoption of Kubernetes in various industries and highlighted emerging trends and best practices within the cloud-native ecosystem.

KubeCon + CloudNativeCon Europe 2022 marked a return to hybrid events after two years of virtual editions, with the in-person portion held in Valencia, Spain. Datadog showcased its latest Kubernetes monitoring solution updates, including improved OpenTelemetry support and the new Kubernetes resources overview. The company's engineers delivered talks on adopting Cilium, debugging difficult-to-resolve incidents, and updating SIG Security. Modern software delivery and supply chain security were prominent themes, with experts discussing strategies for mitigating the effects of increasing software architecture complexity. Platform engineering emerged as a key topic, with discussions on shifting left, building internal platforms on top of Kubernetes, and improving security and reliability through developer self-service platforms. Autoscaling, bin packing, and FinOps were also explored, with talks on resource allocation efficiency, cloud cost management, and custom schedulers for unconventional use cases like machine learning and network functions virtualization. The event highlighted the growing adoption of Kubernetes in diverse verticals and the need for continued innovation and best practices in the ecosystem.

David Chao, Vice President of Marketing at Datadog, leads a team of over 40 professionals focused on building the company's brand and establishing it as a leader in the SaaS market. He joined Datadog through an acquisition and has found that the company is truly customer-centric, with a strong focus on ensuring customer success. The marketing team at Datadog embraces collaboration and experimentation, working together to achieve business value for customers. Datadog celebrates diversity and inclusion by sourcing candidates from diverse backgrounds and actively promoting an inclusive work environment through Community Guilds. David Chao is the Executive Sponsor of AMPED, a community guild representing Asian, Middle Eastern, and Pacific Islander employees at Datadog. The company encourages growth within teams and provides opportunities for career advancement. Datadog is currently hiring for various roles across its global teams, offering an inclusive and supportive work environment that fosters collaboration and innovation.

David Chao, Vice President of Marketing at Datadog, has a diverse career background in management consulting, product management, and marketing. He joined Datadog through the acquisition of Sqreen in 2021, which aligns with his passion for building and scaling B2B Enterprise SaaS companies. At Datadog, Chao has learned that the company is customer-centric and committed to ensuring customers' success, with a strong focus on delivering business value. The Marketing team's culture is collaborative and supportive, allowing employees to grow their careers alongside the company. Datadog celebrates diversity and inclusion through inclusive hiring practices, employee feedback, and community guilds like AMPED, which represents Asian, Middle Eastern, and Pacific Islander Employees. Chao serves as Executive Sponsor of AMPED and is committed to creating a sense of community and mentorship for underrepresented groups within the company.

Content Security Policy (CSP) is a W3C standard that helps protect web applications against cross-site scripting (XSS), clickjacking, and other code injection attacks by specifying which types of resources are allowed to load on your site. Datadog can help you collect CSP reports, monitor violations, and configure security rules to detect noteworthy trends in CSP violations. By using CSP reporting with Datadog, you can gain critical visibility into violations and build effective policies that protect your users without breaking your web applications.

CSP is a W3C standard that helps defend web applications against cross-site scripting (XSS), clickjacking, and other code injection attacks by specifying which types of resources are allowed to load on a site and where those resources can come from. CSP reporting provides critical visibility into violations, allowing developers to build effective policies and ensure they are configured correctly. Datadog helps collect CSP reports, enabling the detection and analysis of violations that may be degrading end-user experience. The platform also supports security rules to automatically notify users about noteworthy trends in CSP violations, which can be triggered by problematic deployments or misconfigurations. By leveraging CSP reporting with Datadog, developers can get immediate visibility into violations, build effective policies, and ensure their web applications remain secure.

This article discusses the development of "Husky", an event storage system built by Datadog. It explains how the company transitioned from storing pre-aggregated timeseries data to handling logs, which require a different approach due to their larger size and high cardinality requirements. The initial version of Datadog's Logs system had limitations in terms of reliability and scalability, leading to the development of a new architecture that separated storage from clustering. However, as the company's platform grew, more issues emerged, necessitating the creation of an entirely new storage system called "Husky". This new system separates compute from storage, allowing for independent scaling of ingestion, storage, and query paths. It also provides flexibility in terms of isolation and performance tuning. The article concludes by noting that this is just the first part of a series discussing Husky's storage system in more detail.

The story of "Husky", a new event storage system built by Datadog, revolves around the need to improve upon their existing Log Management product. The old system had limitations in handling log data, such as pre-aggregating many similar events into a single aggregated datapoint, which made it difficult to store and query log data efficiently. To address these issues, the team decoupled storage from clustering and introduced new roles for writers, readers, and compactors, each serving distinct purposes. The system is designed to be highly scalable, flexible, and performant in the face of unpredictable and aggressive multi-tenant workloads. Husky's architecture separates compute from storage, allowing for better control over isolation, performance, and quality of service in a highly multi-tenant environment. The system has improved reliability, performance, and efficiency compared to its predecessor, with reduced latency and increased flexibility in how queries are isolated and allocated costs.

The latest update of Datadog Live Containers extends its real-time visibility into Kubernetes environments by adding support for RBAC and storage resources. It also introduces the Related Resources Map, which helps navigate through each layer of clusters and access critical information such as performance metrics, related traces and logs, network telemetry, and configuration details. The update allows users to monitor their security and storage configurations effectively and maintain a holistic view of workloads while monitoring the health and performance of constituent resources.

Datadog has extended its Live Containers feature to provide real-time visibility into Kubernetes environments, including support for RBAC and storage resources. The new feature offers a multi-dimensional view of clusters' health, encompassing complex dependency relationships among resources. Datadog's Related Resources Map enables users to navigate through each layer of their clusters and access critical information such as performance metrics, related traces, logs, network telemetry, and configuration details. This empowers engineers to effectively own the operation and maintenance of their complex containerized workloads, reducing the time needed to surface and resolve critical issues.

OpenTelemetry is a Cloud Native Computing Foundation initiative offering open, vendor-neutral standards and tools for instrumenting services and applications to collect observability data. Datadog has enhanced its support for OpenTelemetry by integrating the OpenTelemetry Protocol (OTLP) into the Datadog Agent, allowing users to ingest traces and metrics from OpenTelemetry-instrumented applications without altering their code or workflows. This integration enables direct export of OTLP traces to the Datadog Agent via gRPC or HTTP and supports visualization and monitoring through Datadog APM, which provides deep insights into application performance, dependencies, and anomalies. Datadog also offers an alternative method to export OTLP data using the Datadog exporter and the OpenTelemetry Collector, catering to workflows that require exporting telemetry data to multiple backends. This commitment to OpenTelemetry reflects Datadog's dedication to providing flexible and extensible observability solutions, ensuring comprehensive visibility into user services and applications.

FoundationDB is a distributed NoSQL database designed with ACID transactions support and an unbundled architecture that separates transaction management, storage system, and configuration system. Datadog's new integration allows users to visualize and monitor key metrics from their FoundationDB clusters by deploying the Datadog Agent and enabling the FoundationDB integration. This enables monitoring of transactions, operations, log queues, and storage queues to identify bottlenecks and potential problems. Additionally, tracking coordinator status helps prevent data loss. Datadog's single pane of glass allows users to monitor their FoundationDB clusters alongside other technologies for a comprehensive view of system health and performance.

FoundationDB is a distributed NoSQL database designed to support fully ACID transactions. It uses an unbundled architecture that consists of an in-memory transaction management system, a distributed storage system, and a built-in distributed configuration system. This enables developers to manage and configure each part of their database layer separately for scalability, high-availability, and fault tolerance. With Datadog's new integration, users can visualize and monitor key metrics from their FoundationDB clusters, including transactions and operations, log queues, storage queues, and coordinator status. Monitoring these metrics helps identify bottlenecks and potential problems, such as queue length spikes or storage server failures, which can lead to data loss. The Datadog integration provides a single pane of glass for monitoring multiple technologies, including FoundationDB, alongside 850 other tools.

Datadog developed Clarity License Manager (CLM) to optimize the use of third-party software licenses within the company, addressing the inefficiencies and costs associated with managing numerous SaaS applications. Initially focusing on security, Clarity evolved into a comprehensive tool that uses automation to monitor and deactivate inactive accounts, thereby reducing unnecessary expenses and improving license management. The system integrates with HR data and applications like Google Workspace, notifying employees via Slack and email about inactivity and allowing for easy reactivation when necessary. Its modular architecture, built on AWS Lambda and Postgres, accommodates the scalability required for a growing SaaS footprint by using microservices and application-specific adapters to ensure flexibility and reusability. By leveraging Datadog's metrics and dashboards, the team gains insights into software usage trends, helping inform procurement decisions, while alternative commercial solutions were deemed less customizable and cost-effective. Future enhancements include improving user interaction, incorporating more detailed user data, and optimizing processing efficiency to further refine software licensing strategies.

JumpCloud is a cloud-based directory platform that provides unified Active Directory and LDAP services for user authentication and network management. Datadog now integrates with JumpCloud, allowing users to monitor, visualize, and retain event logs alongside other data across their environment. The integration enables the creation of dashboards to track changes in user behavior, quick searching, filtering, and sorting of logs for ad hoc investigations, and flagging non-compliant user activity with Datadog's out-of-the-box Cloud SIEM rules. This integration provides deep visibility into JumpCloud networks, audit logging, compliance features, and cloud security rules.

Datadog has integrated with JumpCloud, enabling users to ingest and monitor their JumpCloud event logs alongside other data in the Datadog environment. This integration allows for the creation of dashboards to track changes in user behavior, security assertion markup language (SAML) authentication metrics, and audit logs. The integration also enables the use of Datadog's Cloud SIEM rules to detect unusual behavior and generate Security Signals for investigations. Additionally, Datadog provides deep visibility into JumpCloud networks with custom and OOTB dashboards, audit logging, and compliance features.

Superwise is a monitoring platform for high-scale machine learning operations, offering out-of-the-box metrics on models' production behavior. It helps teams detect and address issues such as drift and data quality problems before they impact the business. Datadog's Superwise integration automatically generates metrics based on specific model data entities, allowing users to visualize trends without manual configuration. The platform enables monitoring of model drift and other issues using flexible policy builders and customizable dashboards. Superwise incidents can be investigated with Datadog, providing cross-platform visibility for triaging and analyzing downstream impacts of model issues.

Superwise is a monitoring platform for high-scale machine learning operations that provides model observability and out-of-the-box metrics on production behavior, enabling teams to address drift, data quality issues, and other problems before they impact the business. The Superwise integration with Datadog automatically generates metrics based on specific data entities, allowing immediate visualization of trends without manual configuration. This platform offers flexible policy building for monitoring model drift and other issues, automated incident detection, and integrated model visibility with insights, making it easier to track performance, data quality, and drift alongside other services in the infrastructure.

The integration of the Datadog Synthetics CircleCI orb with CircleCI allows organizations to implement Synthetic tests as part of shift-left testing in their CI/CD pipelines. This enables teams to automatically detect and block code changes that would break key workflows or endpoints, ensuring reliable new code deployments and preempting potential problems for end users. The Datadog CircleCI orb can be set up to run Synthetic tests, debug failing tests, and monitor CircleCI pipelines together with Datadog's existing CircleCI integration. This integration extends support for Synthetic Monitoring in CI/CD pipelines, complementing the Datadog GitHub Action and Datadog plugin for Jenkins.

The CircleCI orb is an open source, reusable config file that automates processes and integrates with services across the stack. It allows organizations to rapidly build, test, and deploy within their pipelines on a single platform using CircleCI's CI/CD service. The Datadog Synthetics CircleCI orb enables Synthetic tests as part of shift-left testing, allowing teams to detect and block code deployments that would break key workflows or endpoints. This integration simplifies testing across different environments, including development, staging, and production. Additionally, the orb provides rich context around pipeline health, enabling teams to optimize their CI/CD workflows and improve the reliability and efficiency of their pipelines.

Flutter is an open-source framework developed by Google that allows developers to build high-performance, multi-platform applications with a single codebase. The framework's native debugging tools can be useful for investigating ad hoc issues, but they lack the full context needed to understand user sessions and troubleshoot past errors. Datadog Mobile Real User Monitoring (RUM) offers support for Flutter, providing deep visibility into user experience across iOS and Android devices, including hybrid apps. With RUM, developers can gain insights into user journeys, analyze performance issues, troubleshoot errors, and optimize their end-user experience. The platform automatically tags session information by OS and web or native component, making it easy to compare design efficacy across platforms and views. Additionally, Datadog's Flutter integration allows for multi-platform visibility into Flutter apps, enabling developers to resolve issues and ensure a smooth user experience.

Azure App Service is a cloud-based platform-as-a-service (PaaS) that simplifies the deployment and scaling of applications, but it presents challenges in observability. To address these challenges, Datadog has released an extension for Azure App Service that enhances visibility into .NET applications by collecting distributed traces, application logs, and custom metrics. This extension allows developers to automatically capture request traces, visualize them with flame graphs, and correlate them with logs and metrics to identify performance bottlenecks and errors. The integration also supports trace ID injection, enabling seamless correlation between logs and traces for more effective troubleshooting. By using Datadog’s APM and visualization tools, developers can gain deeper insights into their applications and the underlying Azure infrastructure, facilitating better performance monitoring and optimization. The Datadog extension is now available, offering Datadog users an easy way to enhance observability in their Azure applications, while new users can start with a free 14-day trial to explore its capabilities.

The Datadog Summit in Denver, Colorado marked the return of in-person customer summits after a two-year hiatus. This year's summit included technical talks, hands-on workshops, and small-group Q&A sessions focused on scaling container operations, improving processes like on-call, and creating smoother feature releases. The event featured introductions to automated root cause analysis with Watchdog RCA, live container resource maps, and recent Log Management advances. Datadog users and partners shared their experiences, discussing topics such as building search infrastructure, shift-left observability, empowering developer teams during hyper-growth, microservices operations, compliance strategies for observability data, serverless applications, Kubernetes scaling, and fast remediation with app observability.

Datadog has returned with its customer summits, featuring a series of technical talks, hands-on workshops, and small-group Q&A sessions. The event covered various topics such as scaling container operations, improving on-call processes, creating smoother feature releases, automated root cause analysis, live container resource maps, log management advances, and more. Security experts discussed the importance of security being everyone's job, while also sharing their experiences with mitigating Linux vulnerabilities in production infra. Other talks focused on building a search pipeline to power a robust search experience, shift-left observability, empowering developer teams during hyper-growth, microservices operations with Datadog Service Catalog, and troubleshooting in serverless applications. The event also featured discussions on Kubernetes adoption at scale, simplifying scaling container operations, fast remediation with app observability, and more.