December 2023 Summaries

The text outlines how Datadog's Service Catalog integrates with CI Visibility and Static Analysis to enhance the monitoring and evaluation of CI/CD pipelines and code quality. By providing a consolidated view of application data—including performance insights, security analysis, and cost allocation—the Service Catalog helps teams optimize their CI/CD processes by identifying and addressing issues like flaky tests, build inefficiencies, and code vulnerabilities before they reach production. The Delivery tab in the Service Catalog allows users to filter services by various facets to gain insights into production code quality and pipeline efficiency, while Static Analysis surfaces warnings and provides fix suggestions for identified code issues. The integration also enables users to track the health of CI pipelines, allowing them to pinpoint problems such as failed tests or long-running jobs, and optimize testing to prevent future failures. Ultimately, these tools support shift-left practices by allowing teams to address potential issues early in the development process, ensuring more reliable and secure software deployments.

At Datadog, Jean-Mathieu Saponaro and his teams have successfully scaled self-serve analytics up to 5,000 employees by building tools that empower all teams to make data-informed decisions. They achieved this through a single source of truth for internal data, intuitive tools, and support and training in data literacy. The three pillars of their strategy are data, tools, and knowledge. By understanding the needs of different user categories within the company, they were able to provide tailored solutions that enabled employees to leverage data without requiring assistance from a centralized team. They also acknowledged limitations of self-serve analytics and how to mitigate them while tracking success metrics for their strategy.

Jean-Mathieu Saponaro, Data & Analytics Senior Engineering Manager at Datadog, delivered a presentation about scaling self-serve analytics within the company up to 5,000 employees. Self-serve analytics is a dream for any Data & Analytics team: empowering everyone in the company to make data-informed decisions without requiring their help. The Data & Analytics department at Datadog built a suite of tools, including data intake, transformation, quality, discovery, and reporting, on top of open-source technologies. They focus on three pillars: data, tools, and knowledge. Jean-Mathieu explains how his teams provide a single source of truth for all internal data, intuitive tools to leverage it, and data literacy thanks to support and training. He acknowledges limitations of self-serve analytics and how to mitigate them. The presentation also covers the importance of education/user training, user support, experts, and defining success in a self-serve analytics strategy. Jean-Mathieu shares Datadog's experience in scaling self-serve analytics as the company grew from 200 employees to 5,000 global employees, highlighting key metrics for tracking success and the importance of unmeasurable signals. The presentation concludes that any digital company can learn from Datadog's experience in implementing a robust self-serve analytics offering.

Datadog has introduced the Scanner, a no-code solution that provides real-time inspection of log events as they travel through different pipelines and processing rules. The Log Pipeline Scanner helps organizations solve issues related to log processing by enabling teams to trace log processing flows, discover, diagnose, and remediate processing issues, ensure security and compliance requirements are met, remove conflicting or redundant processing rules caused by team silos, and streamline the end-user experience. The tool also assists in identifying log processing issues, editing pipelines, and providing comprehensive visibility into log pipelines.

Datadog has introduced the Log Pipeline Scanner, a no-code solution that provides real-time inspection of log events as they travel through different pipelines and processing rules. This tool helps organizations discover, diagnose, and remediate issues with their log processing, ensuring security and compliance requirements are met, removing conflicting or redundant processing rules, eliminating onboarding bottlenecks, and preventing lost remediation time. The Scanner samples and annotates logs to track specific log events as they pass through different pipelines, enabling users to quickly identify changes and investigate the behavior of processing pipelines across their organization. It provides a consolidated look at all pipelines applied to logs, their ownership, and constituent steps, making it easy to fix issues and maintain comprehensive visibility into log pipelines. The Log Pipeline Scanner is now available for all Datadog Log Management customers.

Datadog has partnered with GitHub and GitGuardian to protect its customers' accounts from the risk of leaked credentials, which can lead to data breaches and account takeovers. The company identifies potential leaks by scanning public repositories for keys in various locations such as configuration files, documentation, scripts, and more. Once a key is detected, Datadog notifies users via email or in-app notification on the API key management page. To mitigate the impact of leaked credentials, customers should revoke the compromised key immediately and monitor their account for suspicious activity. Additionally, implementing best practices such as defining scopes for application keys, using service accounts to manage them, regularly reviewing and revoking unused API keys, and creating multiple API keys for different purposes can help prevent future leaks.

Ray is an open source compute framework that simplifies the scaling of AI and Python workloads for on-premise and cloud clusters. It integrates with popular libraries, data stores, and tools within the machine learning ecosystem, including Scikit-learn, PyTorch, and TensorFlow. Datadog now integrates with Ray, enabling users to collect key metrics and logs that help monitor the health of their Ray nodes as AI applications scale. The integration provides visualization of telemetry from Ray environments, alerting on Ray issues, and improving resource efficiency of Ray clusters.

Datadog has partnered with GitHub and GitGuardian to protect its customers' accounts from leaked credentials, which are a major cause of data breaches. Datadog uses regular expressions and detection rules to identify potential leaked credentials in public repositories, and notifies users as soon as it detects a leak. Once notified, users can take immediate action to limit the impact of the leak by revoking the key and reviewing their account for suspicious activity. To prevent leaks from occurring in the first place, Datadog recommends best practices such as defining scopes for application keys, using service accounts, revoking unused API keys, and creating multiple API keys for different purposes. By implementing these strategies, users can safeguard their data and reduce the risk of unauthorized access to their accounts.

Bowen Chen and Anjali Thatte introduce Ray, an open-source compute framework that simplifies the scaling of AI and Python workloads for on-premise and cloud clusters. The platform integrates with popular libraries and tools within the machine learning ecosystem, allowing developers to scale complex AI applications without changes to their existing workflows or AI stack. Datadog now integrates with Ray, enabling real-time monitoring and alerting on Ray issues, such as identifying bottlenecks, failing tasks, and optimizing resource efficiency. The integration provides visualizations of telemetry from Ray nodes, alerts for critical issues, and the ability to manage resource consumption and optimize GPU utilization. With this integration, developers can monitor their Ray clusters alongside other AI toolkits and sign up for a 14-day free trial if needed.

Azure App Service is a platform-as-a-service (PaaS) used to deploy applications and APIs, functions, mobile apps, and more. It offers flexibility and reliability but introduces new security risks due to reduced visibility into infrastructure and deployment of applications. Datadog Application Security Management (ASM) now supports Azure App Service, enabling continuous, real-time monitoring of vulnerabilities in web applications and APIs. ASM integrates with distributed traces and code-level context, helping development, operations, and security teams build secure applications in production. Monitoring Azure App Service with ASM allows for detecting attackers targeting your deployed applications, prioritizing fixing the most impactful vulnerabilities, and correlating security and performance data.

IsDown provides a detailed status page aggregator and uptime monitoring for all your third-party dependencies. By using IsDown, you can monitor the availability of all these tools and services in one place, and receive instant alerts when one starts experiencing issues. Datadog’s IsDown integration enables customers to track emerging outages and their statuses over time by sending outage events and status information from IsDown to Datadog's unified observability platform. The integration is available at no additional cost to current Datadog and IsDown customers, and can be leveraged for tracking outages with IsDown using Datadog dashboarding, event management, and alerting.

Datadog's Incident Management tool aims to streamline incident response by integrating with various platforms like Slack, Zoom, Opsgenie, PagerDuty, and Microsoft Teams. It allows users to declare incidents from multiple places within the platform or directly from Slack when enabled. The tool offers customizable automated workflows, role-based teams, and leverages dashboards for efficient incident analysis. Datadog's mobile app, collaborative Notebooks, and cross-platform Clipboard enhance seamless problem resolution and documentation. While the larger Datadog platform provides significant advantages, users can also use Incident Management as a standalone product. The tool ensures stakeholders are notified at the right time with customizable rules and message templates. It enables real-time incident tracking through an overview page and timeline tab, facilitating collaborative documentation for postmortem analysis. Customization options allow users to tailor settings based on their specific needs and integrate with existing tools like Slack, Microsoft Teams, Zoom, CoScreen, and Jira.

Datadog Application Security Management (ASM) now supports Azure App Service, enabling real-time monitoring of vulnerabilities in and threats against web applications and APIs deployed on the platform. This allows developers to detect attackers targeting their Azure App Service-deployed applications, prioritize fixing the most impactful vulnerabilities, correlate security and performance data, and secure their systems with actionable next steps for remediating threats. With ASM, teams can quickly identify risks, view suggested remediation actions, and pivot to APM for more information about affected services, all within a single platform.

IsDown provides a detailed status page aggregator and uptime monitoring for all third-party dependencies, enabling users to track the availability of these tools and services in one place. The Datadog integration with IsDown allows customers to monitor their third-party dependencies from within Datadog's unified observability platform, providing a unified view of outage statuses. Users can leverage Datadog dashboarding, event management, and alerting to track outages with IsDown, receiving instant alerts when new outages begin or resolve. The integration is available at no additional cost to current Datadog and IsDown customers, and users can configure notifications to automatically notify their team members of new outages. By using IsDown with Datadog, users can access critical context about ongoing outages for their third-party service providers, enabling them to act quickly when problems arise.

Azure App Service is a widely used platform-as-a-service (PaaS) that facilitates the deployment of various applications and APIs, though it also introduces certain security vulnerabilities due to limited visibility into the infrastructure. To address these security challenges, Datadog's App and API Protection (AAP) now supports Azure App Service, offering continuous, real-time monitoring of vulnerabilities and threats. This integration allows users to detect and prioritize vulnerabilities, link security with performance data, and create custom rules for specific threats, while providing comprehensive attack coverage that includes the OWASP Top 10 threats. AAP not only identifies risks but also provides actionable steps for remediation, thereby enhancing the security posture of cloud-hosted applications. Additionally, Datadog AAP helps in correlating security insights with performance metrics, facilitating quicker investigation and resolution of issues by integrating with other Datadog products such as APM and the Service Catalog. This service is available to current users and those interested in a trial, providing a robust solution for securing applications on Azure App Service.

Datadog sponsored this year's AfroTech conference, the largest Black tech conference in the US, held from November 1-5 in Austin, Texas. The event featured 25,000 participants and 210 speakers discussing various topics related to building a career, network, and personal brand in tech. Datadog's presence at AfroTech was organized by its BiT Community Guild, which played a vital role in raising awareness about the company and connecting with the wider Black tech community. The sponsorship helped bring attention to Datadog's commitment to diversity and inclusivity, as well as its increasing role within the broader tech community.

The National Institute of Science and Technology (NIST) has developed the Federal Information Protection Standard (FIPS), which outlines requirements for hardware and software components responsible for data encryption. This is crucial for organizations in regulated sectors like government and healthcare to protect sensitive information. Datadog, a monitoring service, recently released a FIPS-enabled Agent that automatically encrypts payloads using compliant cyphers before forwarding them to the platform. The FIPS-enabled Agent is FedRAMP Moderate-certified, ensuring secure monitoring of cloud infrastructure according to FIPS standards. This release aligns with Datadog's commitment to pursuing FedRAMP High authorization and maintaining environments compliant with NIST and FedRAMP standards.

AfroTech, a prominent Black tech conference in the US, took place in Austin, Texas, from November 1-5, with over 25,000 attendees. Datadog was a proud sponsor of the event, which featured 162 major brands in tech, including Google and Amazon, showcasing various booths, talks, and career opportunities. The company's Black in Tech (BiT) Community Guild played a significant role in organizing its presence at AfroTech, allowing team members to connect with the wider Black tech community and raise awareness about Datadog's brand and values. This event highlights Datadog's commitment to diversity and inclusivity, as well as its efforts to invest time and resources to meet diverse talent where they are. The company continues to grow and is looking for talented individuals to join its teams around the world.

The National Institute of Science and Technology (NIST) developed the Federal Information Protection Standard (FIPS) to ensure organizations use compliant cryptographic algorithms, key management, and authentication controls for storing and processing sensitive data. Datadog has released a FIPS Agent that automatically encrypts payloads using compliant cyphers before forwarding them to Datadog, making it FedRAMP Moderate-certified. This enhancement provides visibility into the agent's activity and confirms encryption of data as expected. The FIPS Agent is now available in the US1-FED region and can be configured for hosts with detailed logs about traffic passing through it.

Steadybit, a software reliability platform, has partnered with Datadog to help organizations improve their applications' stability and performance through chaos engineering and fault injection. The partnership includes an integration that brings Steadybit data into Datadog and a software license offering in the Datadog Marketplace. By using Datadog's preconfigured dashboards, users can monitor chaos engineering experiments, optimize application performance, verify monitor configurations, and refine their incident response process. This collaboration aims to help businesses identify potential vulnerabilities and create more resilient systems.

Steadybit is a software reliability platform that uses chaos engineering and fault injection to help organizations improve the stability and performance of their applications by simulating turbulent scenarios in a controlled environment. Through its partnership with Datadog, Steadybit enables users to monitor and correlate the impact of their chaos engineering experiments, optimizing application performance and refining incident response processes. By integrating Steadybit data into Datadog, users can validate crucial infrastructure and application capabilities, ensuring reliable service for end-users even under pressure. The partnership also allows users to verify monitor configurations, refine incident response processes, and get more out of chaos engineering with Steadybit and Datadog.

The blog post delves into the intricacies of managing CPU resources in Kubernetes, focusing on the CPU Manager's default and static policies. It explains how CPU requests and limits impact container scheduling and resource allocation, emphasizing the role of the Completely Fair Scheduler (CFS) in enforcing these limits under the default policy. The post highlights the importance of monitoring metrics such as CPU usage and throttling to optimize performance and prevent overprovisioning or underutilization. Additionally, it discusses scenarios where setting CPU limits is beneficial, such as in multi-tenant environments or for ensuring predictable application performance. The static policy, which allows containers exclusive access to specific CPU cores, is examined for workloads sensitive to context switching and cache misses. Throughout, the post underscores the value of tracking key performance metrics to troubleshoot and improve application efficiency, offering insights into how these settings influence application behavior in Kubernetes environments.

At re:Invent 2023, key areas of focus included investing in all layers of the generative AI stack, making security accessible, innovating on the developer experience, and enhancing observability everywhere. AWS unveiled several new launches for their GenAI portfolio, including Amazon Bedrock's general availability, Party Rock, Guardrails for Amazon Bedrock, and Graviton4 processor. In terms of security, IAM enhancements were rolled out, as well as improvements to AWS IAM Access Analyzer and the introduction of Cloud Infrastructure Entitlement Management (CIEM) inside of Cloud Security Management. For developer experience, CodeWhisperer for MacOS added terminal-based autocompletion for hundreds of CLIs, and Amazon CodeCatalyst with Amazon Q was designed to expedite software delivery by automating various development tasks. Observability enhancements included support for custom instrumentation for AWS Lambda using the OpenTelemetry Tracing API, a high-level visualization of Step Functions workflow, and Data Jobs Monitoring (DJM) for data platform and data engineering teams.

AWS unveiled several generative AI (GenAI) advancements, including Amazon Bedrock, a platform that democratizes private access to popular language models. The company also announced new launches for its GenAI portfolio, such as Party Rock, which offers a free sandbox for users to play with Bedrock. Additionally, AWS invested in all layers of the GenAI stack and made security accessible by releasing products like Application Security Management and Cloud Security Management. The company also enhanced observability everywhere, including serverless workloads, with features like OpenTelemetry support for AWS Lambda and a new State Machine Map for AWS Step Functions executions. Furthermore, AWS introduced Amazon S3 Express One Zone, which delivers consistent single-digit millisecond data access for latency-sensitive applications.

Datadog has introduced Cloud SIEM 15-Months Retention, an offering that automatically stores logs for 15 months after ingestion. This new feature caters to customers' dynamic needs and enables real-time and historical security analytics on log data from longer time periods at a low cost. Additionally, Datadog Cloud SIEM Investigator is now generally available, allowing users to better understand and query their past data. With this update, teams can quickly detect threats, investigate them, and efficiently search logs for context when needed. New customers will have access to the new plan by default, while existing customers can migrate before their contract expires.

GitHub is a crucial platform for software development and version control, storing sensitive data such as source code. Monitoring logs generated by activity in your GitHub environment can help detect potential threats or insider activities. Datadog's GitHub integration now enables users to collect and monitor audit logs from GitHub, with detection rules that alert users to suspicious GitHub activities like anomalous cloning of repositories or the addition of new enterprise administrators. These features allow for real-time analysis and evaluation of GitHub audit logs against multiple out-of-the-box detection rules, helping identify potential threats and enabling swift action to secure your GitHub organization.

Datadog's GitHub integration allows users to collect and monitor GitHub audit logs in real-time, enabling them to identify suspicious activities that may indicate attacker behavior or insider threats. The integration includes detection rules that can quickly detect anomalous cloning of repositories, addition of new enterprise administrators, and SSH key additions from suspicious IP addresses. Datadog Cloud SIEM provides automated analysis of all GitHub audit logs against multiple out-of-the-box rules, generating Security Signals to alert users of potential issues. By using this integration, users can uncover suspicious activity in their GitHub environment and secure it against threats.

Datadog has introduced a new offering in Cloud SIEM: Cloud SIEM 15-Months Retention, which automatically stores logs for 15 months after ingestion, providing customers with the ability to perform real-time and historical security analytics on log data from longer time periods. This change caters to dynamic needs, allowing teams to easily identify and detect threats, investigate them, and quickly search their logs for context when needed. All new Cloud SIEM customers will have access to this feature by default, while existing customers can continue using the current features without changes to their contract. The new offering aims to provide a unified view across infrastructure, DevOps, and security, helping teams detect problems and follow built-in remediation steps to stop attackers from further accessing and damaging their environments.