July 2023 Summaries

1Password is a password manager that helps organizations reduce weak and reused credentials. Datadog has introduced an integration with 1Password to collect and process logs in real time, enabling users to monitor them for suspicious activities such as credential theft or brute-forcing. The integration provides visualization of 1Password activity using an out-of-the-box dashboard and detection rules in Datadog Cloud SIEM. It also offers automatic analysis of all 1Password logs, evaluating them against multiple detection rules to identify malicious activities. Additionally, users can use Workflow Automation for orchestrating automated actions in response to alerts and streamlining the triage process.

Datadog has announced an integration with 1Password to collect and process logs from 1Password's Events API, enabling real-time monitoring of 1Password activity. The integration provides a dashboard for visualizing activity in real time and detection rules to identify suspicious activities such as credential theft or brute-forcing. Datadog Cloud SIEM analyzes the logs and generates Security Signals to alert users to potential attacks, allowing them to investigate and respond quickly using automated workflows and customizations. The integration is available now to existing Datadog customers and can be started with a 14-day free trial for new customers.

Cloudflare has expanded its integration with Datadog Cloud SIEM to enhance security monitoring and threat detection. This collaboration allows for the seamless transfer and analysis of Cloudflare Zero Trust logs through Cloudflare Logpush, facilitating comprehensive security insights and incident response. The integration introduces pre-configured threat detection rules, such as those identifying suspicious activities like impossible travel scenarios and DDoS attacks, and provides automated workflow options for triaging and responding to security incidents. An updated Cloudflare dashboard in Datadog offers real-time visualization of security metrics, enabling a holistic view of user and application activity within the Zero Trust environment. These enhancements aim to streamline the detection and remediation of security threats, offering a centralized solution for managing the risks associated with modern applications and cloud computing.

Stratus Red Team is an open-source project that provides cloud security teams with a collection of AWS and other cloud provider-specific attack techniques. It enables seamless emulation of adversary behavior within sandboxed environments, allowing security engineers to focus solely on detection development and validation by abstracting the intricacies of infrastructure setup and teardown. By using Stratus Red Team in conjunction with Datadog Cloud SIEM, cloud security teams can elevate their threat detection and investigation capabilities for dynamic, cloud-scale environments. This includes continuous validation through Threatest, an open-source project by Datadog that allows users to detonate attack techniques and verify the generation of expected alerts in Datadog Cloud SIEM.

Stratus Red Team is an open source project that enables cloud security engineers to seamlessly emulate adversary behavior within sandboxed environments, abstracting the intricacies of infrastructure setup and teardown. Datadog Cloud SIEM provides real-time analysis of operational and security telemetry, along with curated out-of-the-box rules for detecting and responding to threats. By using Stratus Red Team and Cloud SIEM together, cloud security teams can enhance their threat detection capabilities against diverse AWS threats, improve resilience, and streamline the detection development and validation processes. Threat emulation enables cloud security teams to leverage their understanding of threat actor behaviors as a feedback loop for developing effective cloud-based detections. However, setting up and tearing down infrastructure introduces friction, highlighting the need for efficient tools like Stratus Red Team and Cloud SIEM.

The Dependency Map Navigator is a tool that provides detailed visibility into the flow of traffic through your resources from upstream to downstream service dependencies. It helps identify load amplification in downstream services and expedite incident response by quickly identifying large upstream callers, allowing for faster troubleshooting and optimization of performance in microservices environments.

The Dependency Map Navigator is a tool provided by Datadog that offers detailed visibility into the flow of traffic through resources, such as specific endpoints or queries, from upstream to downstream service dependencies. This visibility enables users to quickly pinpoint repetitive requests to services in downstream call paths and troubleshoot issues in upstream services. The tool helps identify load amplification in downstream services, which can drag down application performance and drive up costs, by displaying the upstream and downstream service dependencies of any resource selected. It also facilitates expedited incident response by quickly identifying large upstream callers and providing vital information for internal coordination among teams. By using the Dependency Map Navigator, engineering teams can quickly zero in on costly performance inefficiencies in downstream services and speed up troubleshooting during incidents.

The Datadog Service Catalog integrates with Backstage to consolidate and manage all your service information in one platform. By automatically importing your Backstage YAML, Datadog enables you to view service data side-by-side with observability data from applications and infrastructure, eliminating knowledge gaps. You can populate the Service Catalog with your Backstage YAML, import multiple service definitions at once, and validate service metadata by viewing schema version information. This integration allows teams throughout an organization to have a central repository for managing services and troubleshooting, investigating, and collaborating across all their services in a single platform.

AWS Lambda occasionally pre-initializes execution environments to reduce the number of cold start invocations. This feature is known as proactive initialization and can be observed in functions using unreserved (on-demand) concurrency. Proactive initialization serves both the interests of AWS and developers running applications on Lambda, as it minimizes the time spent on initializing functions that will serve invocations. The percentage of true cold start initializations to proactive initializations varies depending on many factors, but you can clearly observe it. Depending on your workload and latency tolerance, you may need Provisioned Concurrency. AWS Lambda functions without steady traffic do not typically benefit from proactive initializations, while functions with more traffic tend to have a higher percentage of proactive initializations.

AWS Lambda occasionally pre-initializes execution environments to reduce the number of cold start invocations. This is known as proactive initialization, and it can occur even when features like Provisioned Concurrency are not being used. Proactive initialization serves the interests of both AWS and developers running applications on Lambda, as it minimizes CPU time in a shared, multi-tenant system. It occurs more frequently than true cold starts, with a percentage that varies depending on factors such as workload and latency tolerance. Developers can observe proactive initializations using tools like CloudWatch Logs Insights or Datadog. By tracking this behavior, developers can focus on their own applications while AWS improves the underlying infrastructure.

Datadog's Software Catalog centralizes knowledge about services' ownership, reliability, performance, costs, and security. By automatically importing Backstage YAML, Datadog lets teams manage all services in a single platform, viewing service data alongside observability data from applications and infrastructure. The integration populates the Software Catalog with Backstage YAML, imports multiple service definitions at once, validates metadata by viewing schema version information, and enables teams to troubleshoot, investigate, and collaborate across all services. With support for Backstage YAML, Datadog automatically creates and maintains Software Catalog entries, making it easy to see which service definitions are based on the Backstage schema and identify opportunities to enrich them with additional metadata.

Stream-based monitoring solutions are gaining popularity as they allow real-time troubleshooting and eliminate the need to store data that only needs to be used transiently. A new enhancement to Live Tail, a stream-based product, provides the ability to search, correlate, and perform analytics across all ingested logs for 15 minutes, giving users full access to the data needed during real-time investigations and time-sensitive analyses. This enables users to troubleshoot issues in their CI/CD pipeline, verify new deployments, and correlate directly between live traces and logs, ultimately providing a continuous, real-time stream of logs that can help engage with audiences effectively and troubleshoot time-sensitive incidents. The solution is designed to handle data at petabyte scale, allowing users to view and query all ingested logs without pressure to retain them.

In this guest blog by Mike Stemle, a software engineer and Principal Architect for Arc XP division of the Washington Post, he discusses his love for Datadog Service Catalog and how it can be used to improve quality of life during production support. He explains that to get the full benefit from Service Catalog, you need to flesh out your organization’s service definitions using GitHub integration, Terraform, or manually supplying definition details to the Service Definition API. Stemle recommends using GitHub Actions to send information to Datadog and Service Catalog with his custom action called the Datadog Service Catalog Metadata Provider (DSCMP). He also covers how to use GitHub Actions together with DSCMP, organizational controls, and examples of enforcing organizational controls.

The author of the article, Mike Stemle, loves using the DataDog Service Catalog because it puts service owner ship and production support in one place. He believes that having this information readily available helps teams detect, mitigate, and resolve issues before they reach production. However, to get full benefits from Service Catalog, organizations need to flesh out their service definitions. To do so, Mike recommends using the GitHub Actions framework to send metadata to Datadog via GitHub repositories or managing service definitions through Terraform. He suggests using a custom action called the DataDog Service Catalog Metadata Provider (DSCMP), which allows teams to supply service metadata to Datadog without giving any third parties special access or permissions. The DSCMP uses an open-source custom action, and its benefits include providing light-touch technical controls to enforce policies that define which metadata must be included in service definitions. The author also discusses the importance of organizational controls, including limiting values for certain fields, enforcing rules, and using YAML syntax.

Static web architecture is an efficient solution for many lightweight web applications such as blogs and documentation repositories. In this architecture, pages are generated and pre-rendered at build time from markup files and cached in a content delivery network (CDN) for fast delivery. This reduces the need for server management and enables quick page load times. However, monitoring the health and performance of static web apps presents some challenges due to reliance on CDNs and limited options for frontend tracking. Key metrics and data should be collected to optimize caching and delivery from the CDN, monitor the build-to-deploy cycle, and maintain visibility into infrastructure dependencies.

Thomas Sobolik discusses the challenges of monitoring static web applications and provides best practices for effective monitoring. Static sites are currently a popular solution for many lightweight web applications, such as corpsites, blogs, job listings, and documentation repositories. With the majority of backend logic migrated to the build process and CI/CD, reliance on CDNs for delivering both static assets and page markup, and limited options for instrumenting the frontend to track UX performance, monitoring the health and performance of static web apps presents some challenges. The author suggests collecting real user session data as well as performance and security data from the CDN provider to optimize caching and delivery. They also recommend monitoring the build-to-deploy cycle, including the supplemental build infrastructure, CI pipelines, and synthetic tests. Additionally, they suggest collecting frontend session data to track page loads and their dependent CDN requests, CDN request logs, as well as health, performance, and security data from the CDN provider. The author emphasizes the importance of tracking bandwidth usage for visibility into CDN cost, security monitoring data from the CDN, and maintaining healthy testing to avoid deploying regressions. Finally, they recommend using a comprehensive solution like Datadog to monitor each layer of the static web stack in a unified platform.

The US federal government has proposed the adoption of a Software Bill of Materials (SBOM) across the software industry to help organizations detect vulnerabilities in third-party software. SBOMs provide an inventory of the software components in a product's codebase, including third-party dependencies. While useful for identifying vulnerabilities, SBOMs have limitations that can make them difficult to work with for vulnerability management purposes. Datadog Application Vulnerability Management addresses these limitations by providing real-time insights into vulnerabilities in open source libraries running in production and integrating with multiple well-known vulnerability databases.

OpenTelemetry (OTel) is an open source observability framework that provides APIs, SDKs, and tools for instrumenting applications and services. Datadog now supports the ingestion and visualization of runtime metrics from OTel-instrumented applications in Java, .NET, and Go. Runtime metrics offer deep visibility into application resource usage such as memory usage, garbage collection, parallelization, etc. By sending these metrics to Datadog, users can gain additional insights for troubleshooting issues throughout the platform. This complements Datadog's existing support for runtime metrics through native tracing libraries, allowing comprehensive visibility into all applications regardless of whether they are instrumented with Datadog or OTel SDKs.

Stanley Liu announces that OpenTelemetry (OTel) is now supported in Datadog for the ingestion and visualization of runtime metrics from OTel-instrumented applications in Java, .NET, and Go. This integration allows users to gain additional insights into their applications' resource usage, such as memory usage, garbage collection, parallelization, and more. By sending runtime metrics from OTel-instrumented services to Datadog, users can get comprehensive visibility into all their applications, regardless of whether they are instrumented with Datadog or OTel SDKs. The integration enables users to visualize and monitor runtime metrics in context with other data from their applications, providing a single lens for optimizing application performance through insights such as memory usage, garbage collection, and parallelization.

The text discusses the increasing reliance on open-source, third-party components in software development and the associated risk of introducing security vulnerabilities, exemplified by incidents like Log4Shell and Dirty Pipe. To address these risks, the U.S. government has advocated for the adoption of a Software Bill of Materials (SBOM), which inventories software components, including third-party dependencies, as a means of enhancing software supply chain security. However, SBOMs have limitations, such as being difficult to query and lacking contextual information about whether vulnerabilities are actively exploited in production environments. Datadog Software Composition Analysis (SCA) seeks to augment SBOMs by providing runtime context, making them more informative, queryable, and linkable, thus facilitating more effective vulnerability management. SCA integrates with Datadog’s platform to surface vulnerabilities actively present in production, allowing for more targeted remediation efforts and keeping organizations compliant with regulatory frameworks.

Datadog has been named a Leader in the 2023 Gartner Magic Quadrant for APM and Observability for the third consecutive year. The company's observability and security platform helps break down organizational silos, providing all teams with a unified view into their technology stack. Products such as APM, Universal Service Monitoring, and Service Catalog simplify microservice monitoring and governance. Datadog's AI engine, Watchdog, and Workflow Automation speed up root cause analysis and remediation, allowing customers to focus on innovation. The company believes that customer feedback is a core aspect of analyst reports and is grateful for the users who continue to share their experiences and help improve its platform.

Flutter is an open source framework developed by Google that enables the creation of high-performance, multi-platform applications with a single codebase. Datadog Mobile Real User Monitoring (RUM) supports Flutter and provides deep visibility into user experience across iOS and Android devices. RUM allows developers to gain insights into user journeys, troubleshoot errors and performance issues, and obtain multi-platform visibility for Flutter apps. By using RUM with APM traces, developers can debug the source of user performance issues regardless of where they originate in their stack.

React Native is an open-source framework used for building cross-platform mobile applications. While it abstracts away device-specific architecture and APIs, developers still need to monitor their applications in production to ensure the best user experience. Datadog Real User Monitoring (RUM) provides detailed insights into user behavior and application performance across mobile devices and browsers. Mobile RUM supports React Native monitoring, allowing developers to track errors, crashes, launch time, user actions, and network requests. It also offers comprehensive visibility into user behavior, helps identify which errors need immediate attention, and assists in finding the causes of long load times and dropped frames. With Datadog's Mobile RUM SDK, developers can easily set up monitoring for their React Native applications with minimal configuration.

Datadog has been named a Leader in the 2023 Gartner Magic Quadrant for APM and Observability, reflecting its commitment to helping customers understand their most complex challenges. The company's observability and security platform breaks down silos and provides a unified view of technology stack health and performance. Products such as APM, Universal Service Monitoring, and Service Catalog simplify microservice monitoring and governance, while Watchdog and Workflow Automation speed up root cause analysis and remediation. Customer feedback is valued by Datadog, which supports its customers in their day-to-day lives through continuous platform improvement and advancement of its mission. Gartner Peer Insights reviews from Datadog customers can be shared here.

Monitoring Windows event logs is crucial for maintaining the security and health of your systems. These logs record system and application activity, user actions, and background processes, providing valuable information for detecting potential security breaches and other issues in your environment. Key events to monitor include successful or failed logon attempts, changes to user accounts and security groups, Windows Task Scheduler tasks, application crashes, and Microsoft Defender Antivirus activities. By collecting and analyzing these logs with tools like Datadog, you can quickly respond to suspicious activity and improve the overall security of your systems.

Datadog introduces Remote Configuration, a new feature that enables users to remotely configure the behavior of Datadog components deployed in their infrastructure for select product features. This allows real-time updates and management of various security settings across applications and services. Currently supported features include Application Security Management (ASM), Cloud Workload Security (CWS), Application Performance Monitoring (APM), Observability Pipelines, and Dynamic Instrumentation. Remote Configuration helps users strengthen their overall security posture, decrease management costs, and shorten resolution times. The feature is generally available for ASM, while other supported features are in beta.

Windows event logs are an invaluable tool for monitoring the security and health of systems. They record system and application activity, user actions, and background processes, providing insights into potential security breaches and issues. Key events to monitor include successful/failed logon attempts, admin privileges assigned to new users, changes to user accounts and security groups, Windows Task Scheduler tasks, application crashes, and Microsoft Defender Antivirus events. Monitoring these events can help improve the health and security of an environment, and tools like Datadog can be used to visualize, analyze, and alert on key events. By configuring channels in the `win32_event_log.d/conf.yaml` configuration file, users can collect Windows event logs as Datadog logs and gain a deeper understanding of their system's activity.