May 2023 Summaries

Firewall systems are essential for safeguarding networks and devices from unauthorized traffic. They come in various forms such as hardware, software or cloud-based solutions, falling under either network-based or host-based categories. Network-based firewalls monitor and filter incoming and outgoing traffic while host-based ones manage traffic to and from a specific device like a laptop. Firewall logs capture valuable information about network traffic based on standard configurations or custom rules. These logs typically include timestamps, actions taken by the firewall (like allow, deny or drop), source and destination IP addresses, ports used for communication, and the protocols involved in the request. This data helps determine who is accessing your network, which resources they are trying to interact with, and how they are doing so. Monitoring these logs can help detect malicious activity as well as network performance issues. Key firewall logs include port scans on your network, inbound connections from external sources, outbound connections to external sources, and changes made to the firewall configurations or rules. Datadog provides turn-key integrations for various firewall systems including Palo Alto Networks Firewall, AWS Network Firewall, Amazon Web Application Firewall, and Microsoft Azure Firewall. It also allows direct collection of logs from hosts or external servers. Datadog's advanced filtering, analysis, and threat detection capabilities can be used to maximize visibility into firewall traffic. Datadog's Pattern Inspector helps identify trends in firewall activity by providing a visual breakdown of patterns based on search queries. Log-based metrics enable monitoring network activity over time, creating alerts, and detecting anomalies. Cloud SIEM Threat Detection automatically flags suspicious activity using industry-standard threat intelligence lists. Online Archives retain logs for forensic analysis up to 15 months while exclusion filters control the volume of logs that are indexed, helping manage costs and focusing on specific scenarios. By efficiently managing firewall logs, Datadog provides comprehensive visibility into network traffic patterns, enabling early detection of potential threats or performance issues.

The Cloudera Data Platform (CDP) is a data analytics and management platform that centralizes, visualizes, and governs data. It offers flexibility to integrate with open source technologies and can be deployed in hybrid, cloud-native, or multi-cloud environments. Datadog's integration with CDP Public Cloud provides complete visibility into clusters, helping SREs and developers monitor workloads and prevent issues before they impact end users. The certified Cloudera integration allows users to understand the health of their CDP Public Cloud clusters and hosts for capacity planning purposes. Additionally, it offers out-of-the-box (OOTB) Cloudera Powerpacks that provide full visibility into the Hadoop stack, enabling users to detect and remediate performance issues early.

An engineer team faced a high-urgency issue with an application in their usage estimation service, which led to excessive backlog and alert fatigue due to frequent paging about startup latency. The problem was resolved after addressing four separate issues including misconfiguration in the network proxy and a Linux kernel bug. They followed system-level metrics and inspected each component in the network path to investigate and resolve these issues in production. This process helped them improve observability of their network, leading to significant cost savings and better visibility for future troubleshooting.

Datadog has integrated its platform with Cloudera Data Platform (CDP), enabling users to gain real-time visibility into their CDP Public Cloud clusters and hosts. The integration allows customers to collect and view their CDP metrics and logs, providing complete visibility into cluster performance and health. This enables SREs and developers to ensure that workloads are running smoothly and at optimal performance, preventing issues before they impact end users. Additionally, the integration provides customizable dashboards using Cloudera Powerpacks, which offer full visibility into the Hadoop services running on CDP Public Cloud clusters, enabling users to detect and remediate performance issues early. The integration is available for free with a 14-day trial, making it easy for users to get started with real-time monitoring of their CDP clusters today.

The authors of the article encountered a high startup latency issue in their usage estimation service application, which was not related to their changes. They investigated the issue and found that it was caused by several bottlenecks in the system, including a misconfigured network proxy, a Linux kernel bug, and saturation of network bandwidth due to packet drops at the hypervisor level. After implementing several fixes, including allocating more CPU to the Envoy sidecar, patching the Linux kernel bug, optimizing AWS instance network configurations, and routing client requests away from terminating pods, they were able to resolve the issue and improve the reliability of their application. The article highlights the importance of observability and monitoring in identifying and resolving complex system issues.

On March 8, 2023, Datadog experienced an outage that affected all services across multiple regions due to a systemd update in Ubuntu 22.04. The new systemd-networkd behavior led to the flushing of IP rules and loss of network connectivity for both host and pod traffic on AWS and Azure, while only affecting host traffic on Google Cloud. This incident impacted multiple regions across distinct cloud providers and delayed the recovery process due to the different actions required by each provider.

Watchdog Insights for Datadog Real User Monitoring is an AI-powered engine that helps identify and resolve performance issues in frontend applications. It autonomously surfaces context related to errors, latency, and Core Web Vitals values in your views by analyzing real-time RUM events. This feature assists teams in investigating ongoing issues more efficiently and identifying potential problems before they worsen. Watchdog Insights can be used with Datadog's APM Trace Explorer, Log Explorer, Live Containers view, and now the RUM Explorer to provide intelligent insights into application performance.

Watchdog Insights for Datadog Real User Monitoring is an AI-powered engine that helps teams quickly investigate and resolve frontend application performance issues. By analyzing real-time RUM events, Watchdog identifies tagged outliers associated with higher-than-usual errors, latency, and Core Web Vitals values in frontend views, providing context to help teams more efficiently investigate ongoing issues and identify potential problems that may worsen in the future. The tool surfaces anomalous patterns of latency or errors associated with specific facets such as user location, browser version, or code release, helping teams find root causes when their application is experiencing degraded performance. Watchdog can supplement Datadog RUM's alerting and incident response workflows by providing key context, including total customer impact, breakdowns of related tags, and highlighted errors. The tool also helps teams characterize frontend performance during broader investigations, spotting UX performance bottlenecks and potential problems that affect views' load performance, interactivity, and visual stability. Watchdog Insights is generally available in Datadog query tools, providing teams with more clarity with less effort to investigate ongoing issues and identify potential new ones more efficiently.

The outage on March 8, 2023, was caused by a new behavior introduced in Ubuntu 22.04's systemd-networkd that flushed all IP rules it did not know about when starting up. This change affected nodes running Ubuntu 22.04, which had adopted the new version of systemd, and led to instances losing network connectivity on multiple regions across distinct cloud providers. The impact was particularly severe on AWS due to its auto-scaling features, which terminated and replaced thousands of instances in a matter of minutes, resulting in lost data stored on local disks. In contrast, Google Cloud and Azure instances could be recovered by simply restarting them, while Kubernetes hosts running on these providers were able to recover their connectivity more quickly. The incident highlighted the importance of understanding the network configuration and potential interactions with cloud provider-specific features when deploying software.

Migrating on-prem applications to Azure can improve scalability, reliability, and security while reducing costs. However, understanding and managing Azure costs during migration and post-migration can be challenging due to limited cost visibility and lack of correlation with resource utilization. Datadog Cloud Cost Management for Azure aims to address these issues by providing granular cost data across multiple billing accounts, mapping cost data to observability data, and enabling automated cost reporting. This helps stakeholders throughout the organization track costs, identify cost overruns, optimize usage, and maintain accountability, ensuring effective management of cloud costs during migration and beyond.

Bottomline Technologies has introduced a non-invasive mainframe security solution called Bottomline Record and Replay, which allows organizations to monitor legacy IBM Mainframe 3270 and System 5250 users via network traffic. This solution can now be integrated with Datadog's platform, enabling users to view mainframe user and system activity alongside resource performance and availability within a single interface. The integration provides an out-of-the-box dashboard for tracking and analyzing mainframe user sessions, security events, and resource performance. Additionally, it allows users to create custom alerts and detection rules for monitoring unusual activity. By using Bottomline Record and Replay in Datadog, organizations can enhance their visibility into mainframe systems, detect threats and data breaches, manage and improve performance, and gain a comprehensive view of user and system activity.

Datadog Cloud Cost Management for Azure helps stakeholders track granular costs across multiple billing accounts and clouds, providing visibility into cost trends and drivers. It allows cloud cost owners to group Azure cost data by meaningful dimensions like product, team, and resource, making it easier to understand changes in costs over time. The platform also unifies cost data with observability data from Datadog APM, enabling teams to determine whether their Azure usage is cost-efficient or identify opportunities for optimization. By providing a broad view of an organization's Azure spend and automated cost reporting, Cloud Cost Management helps maintain accountability and prevent surprises, ensuring that stakeholders can make informed decisions about cloud costs and optimize the cost effectiveness of services in Azure.

The Bottomline Record and Replay solution allows organizations to monitor legacy IBM Mainframe systems without the need for native software or agent deployment. This non-invasive mainframe security solution enables monitoring of user activity, system performance, and resource usage through network traffic, providing a single platform view within Datadog Log Management. The integration offers an out-of-the-box dashboard, pre-built log pipeline, alerting capabilities, and metrics to help teams monitor and optimize mainframe resources, track user sessions, security events, and resource response times, and create custom alerts for performance issues. Additionally, the solution enables users to replay suspicious activity, promoting enhanced visibility into mainframe systems and aiding in threat detection and data breach prevention. The integration is available in the Datadog Marketplace with a software license purchase, offering a 14-day free trial for new customers.

Azure OpenAI, a service designed to develop generative AI applications using OpenAI’s resources, benefits from Datadog's integration, which enables comprehensive monitoring of performance and costs. The integration provides easy access to metrics, such as API usage patterns and token consumption, through Datadog’s built-in dashboards and monitors, allowing users to identify and troubleshoot performance issues efficiently. Datadog's Azure OpenAI integration, which requires no additional setup, collects detailed performance data, offering insights into API call efficiency, latency, and rate limiting, thereby helping optimize resource use and manage costs. Users can also configure Datadog to collect logs for deeper troubleshooting and track usage costs, with automated alerts for unusual activities, ensuring better control over resource allocation and security threats.

Microsoft Azure has introduced several enhancements to improve observability and onboarding of new teams and applications in Datadog. These updates include faster metric collection using the Azure Monitor Metrics Data plane API, streamlined monitoring setup for multiple Azure subscriptions, automatic custom metric collection from Application Insights, and recommended alerts for popular services with Azure recommended monitors. The aim is to provide comprehensive observability and seamless integration of new teams and applications in Datadog's platform.

Datadog has announced several enhancements to its Azure integration, enabling enterprise customers to easily onboard new teams and applications while ensuring comprehensive observability of their business-critical applications on Azure. The updates include faster metric collection with the Azure Monitor Metrics Data plane API, streamlined monitoring setup for multiple Azure subscriptions, automatic custom metric collection from Application Insights, recommended alerts for popular services with Azure recommended monitors, and simplified onboarding options that enable visibility across entire organizations in just minutes. These enhancements aim to help large-scale enterprises manage their complex Azure environments more efficiently.

Datadog's Azure DevOps integration for Continuous Integration (CI) Pipeline Visibility offers end-to-end visibility into Azure pipelines, allowing users to proactively maintain pipeline health and performance, increase throughput, and control CI spend. The integration provides high-level overviews of the entire CI system, enabling users to identify slowest and most error-prone pipelines, stages, and jobs. It also offers granular visibility into pipeline errors and performance regressions, allowing for in-depth analysis and quick identification of issues. With this integration, users can optimize Azure Pipelines performance and ensure the reliability of their overall CI system.

Datadog's new Azure DevOps integration for CI Pipeline Visibility provides end-to-end visibility into Azure Pipelines, enabling proactive monitoring of pipeline health and performance, identification of errors and performance regressions, and optimization of pipeline performance. The integration allows users to proactively survey their entire CI system, scope dashboards to view pipelines by provider, pipeline name, branch, and more, and receive granular routing of notifications to expedite troubleshooting and reduce alert fatigue. With this visibility, users can identify potential issues in their pipelines, stages, and jobs, and investigate in order to prevent these issues from snowballing, ultimately increasing release velocity and ensuring the health and performance of their CI system.

Roku is a popular streaming platform with over 70 million active users and content available through more than 350 channels. Datadog Real User Monitoring (RUM) has introduced a new library to enable monitoring for Roku channels, helping publishers troubleshoot and optimize the user experience (UX). The integration provides insights into user journeys, error tracking, and quality of experience (QoE) issues with RUM and APM. Datadog RUM shines light on previously hidden components of the Roku technical stack, allowing teams to analyze every part of it for better visibility into channel health.

Roku is a popular streaming platform with over 70 million active users and 350+ content channels, offering a wide variety of TV shows, movies, and online video content. As a result, intense competition exists among publishers to offer compelling services that attract and retain viewers, making it challenging for content providers to troubleshoot and optimize the user experience (UX). Datadog Real User Monitoring (RUM) now includes a new library that enables monitoring for Roku channels, providing deeper visibility into user journeys, troubleshooting errors, detecting quality of experience (QoE) issues, and optimizing UX. With RUM, teams can analyze sessions and funnels to better understand user behavior, identify root causes, and ensure smooth experiences. The platform also provides insights into error tracking and QoE issues, enabling content providers to resolve issues and optimize their services.

On March 8, 2023, Datadog experienced a major outage affecting US1, EU1, US3, US4, and US5 regions across all services. The incident began at 06:03 UTC, causing users to lose access to the platform and various services via browser or APIs. Data ingestion was also impacted initially. Investigation started immediately, with the first status page update indicating an issue at 06:31 UTC. Initial signs of recovery were seen at 09:13 UTC (web access restored), and major service operational by 16:44 UTC. All services in all regions were declared operational on March 9, 2023, at 08:58 UTC, with the incident fully resolved on March 10, 2023, at 06:25 UTC once historical data was backfilled. The root cause of the outage was identified as a security update to systemd that caused a latent adverse interaction in the network stack on Ubuntu 22.04 via systemd v249. This issue affected tens of thousands of nodes between 06:00 and 07:00 UTC, causing enough of Datadog's fleet to be offline that the outage was visible to customers by 06:31 UTC. The legacy security update channel responsible for this has been disabled across all affected regions, preventing it from happening again. Datadog's response involved an emergency operation center with over 750 incident responders working in four shifts to bring the incident to full resolution. Frequent updates were provided on the status page to keep customers informed of progress. The high-level steps of recovery included restoring compute capacity, recovering each service in parallel, and addressing issues related to cloud providers' auto-scaling logic. Lessons learned from this outage include improving communication during incidents, prioritizing access to live data over historical data, and enhancing chaos testing to consider larger scale disruptions. Datadog is committed to verifiably improving the resilience of its services based on these learnings.

The Datadog team experienced a global outage starting March 8, 2023, at 06:03 UTC, affecting US1, EU1, US3, US4, and US5 regions across all services. The incident was caused by an automatic security update to systemd on Ubuntu 22.04, which deleted routes managed by the Container Network Interface (CNI) plugin, leading to network stack issues. The outage resulted in data ingestion problems, unavailability of monitors, and limited web access. Recovery efforts involved restoring compute capacity, recovering services in parallel, and addressing cloud provider-specific auto-scaling logic differences. The incident highlighted the importance of considering indirect couplings between regions, prioritizing live data processing, and improving communication with customers during outages. The team learned valuable lessons to strengthen their foundational infrastructure and improve resilience.

Datadog's platform has experienced significant growth over the years, with engineers working across a variety of specializations and technologies. The company encourages employees to own their code throughout its lifecycle in alignment with modern DevOps practices. Datadog offers numerous career paths for engineers interested in becoming individual contributors or climbing through senior leadership ranks. Tanguy Le Barzic transitioned from Director to Staff Engineer, focusing on driving engineering goals and building bridges between teams. Claudia D'Adamo became the Boston Engineering Site Lead, working to build a strong engineering culture and support her fellow engineers' career development. Jaclyn Verga transitioned from Product Marketing Manager to Security Engineer, leveraging her previous experience to help her team operate cross-functionally and practice user-centric development. Datadog is always looking for individuals who want to develop their skills, build their careers, and make an impact on the company's growing customer base.

Datadog's platform has grown rapidly over the past few years, with engineers working in a broad range of specializations to develop products and integrations. The company encourages its employees to own their code throughout its lifecycle, promoting modern DevOps practices. Several Datadog engineers have made career shifts, such as Tanguy Le Barzic transitioning from Director to Staff Engineer, Claudia D'Adamo growing a regional team as an Engineering Site Lead, and Jaclyn Verga transitioning from Product Marketing to Security Engineering. These individuals have found fulfillment in their roles, with Tanguy aiming to build bridges between teams, Claudia advocating for her engineers' career development, and Jaclyn leveraging her PMM experience to help her team operate cross-functionally. Datadog offers ample opportunities for employee growth and development, including formalized processes and resources for mentorship, onboarding, training, and career development.

XDP, or eXpress Data Path, is a Linux networking feature designed to enhance high-performance packet processing by running programs in the kernel, specifically before packets reach the kernel's network stack. Built on the extended Berkeley Packet Filter (eBPF), XDP allows for faster processing by handling packets as soon as they arrive at the network interface card (NIC), enabling actions like dropping, redirecting, or retransmitting packets through predefined return codes. These programs are attached to NICs and can operate in native, offloaded, or generic modes, with offloading offering the best performance when supported by SmartNICs. XDP’s efficiency is particularly beneficial for applications such as DDoS protection, firewalls, and load balancers, and can be implemented using libraries like the BPF Compiler Collection (BCC). The feature's ability to work safely in kernel space after code verification makes it a valuable tool for developers aiming to build observability tools and network performance solutions.

Datadog has released new updates to its Apache Impala integration, CockroachDB integration and introduced Autodiscovery for SonarQube integration. The Apache Impala integration now includes out-of-the-box dashboards that help monitor key components of Impala clusters. Updates to the CockroachDB integration include additional metrics for measuring backup health and meaningful query latency. The new feature, Autodiscovery for SonarQube integration, enables users to quickly start gathering metrics from their projects with minimal setup. These updates aim to provide complete, end-to-end visibility into entire systems.

The latest releases from Datadog include new integrations with Apache Impala, CockroachDB, and SonarQube. The Apache Impala integration provides OOTB dashboards for monitoring key components of Impala clusters, enabling users to investigate slow queries and troubleshoot issues. The CockroachDB integration now includes additional metrics for measuring backup health and query latency, as well as an updated OOTB dashboard with visualizations for node performance and resource allocation. The SonarQube integration has been enhanced with Autodiscovery, allowing users to streamline their integration setup and start analyzing code immediately. Datadog also provides over 850 integrations to help users monitor their entire application, from hosts and containers to end-user sessions.

OpenAI, known for its GPT family of large language models, has seen widespread adoption across various fields since the release of GPT-3 in 2020. Its API allows developers to leverage these sophisticated AI models, and now, Datadog has introduced an integration to monitor and optimize the use of OpenAI's API within organizations. This integration helps track usage patterns, manage costs by analyzing token consumption, and monitor performance by assessing API response times. Users can gain insights into how different models are employed across teams, identify unauthorized usage, and adjust API parameters to optimize application engagement and costs. With the ability to collect detailed logs and traces, organizations can ensure they stay within rate limits and address potential performance issues effectively.

The Datadog Software Catalog has been enhanced with Scorecards, which serve as a tool for service owners, SREs, and stakeholders to assess gaps in observability and adherence to reliability best practices across their organization's services. These Scorecards evaluate services against criteria in three categories: Production Readiness, Ownership & Documentation, and Observability Best Practices. They provide a comprehensive overview of a service's health, helping teams identify and prioritize improvements, communicate effectively with stakeholders, and ensure adherence to development standards. The system allows for automated daily updates, recurring reports, and documentation access to assist teams in maintaining transparency and accountability. With Scorecards, organizations can proactively manage service performance, reliability, and availability, and address any deficiencies in real-time, fostering continuous improvement and collaboration across teams.

Datadog Network Performance Monitoring (NPM) offers enhanced search and mapping capabilities to help users quickly pinpoint critical metrics for network visibility. The new search experience allows for streamlined, unified filtering of both client and server data simultaneously. Clustered network maps organize large quantities of network data into neatly categorized visualizations for easy sorting through endpoints. NPM helps analyze client and server data together using a unified search bar and use clustered network maps to assess high-cardinality traffic, enabling users to find the information they need faster during time-sensitive investigations.

Datadog's Network Performance Monitoring (NPM) feature aims to simplify network troubleshooting by providing enhanced search and mapping capabilities that help users quickly identify critical metrics and visualize their network health in real-time. By analyzing client and server data together using a unified search bar, users can gain complete visibility into their network traffic flow, streamline their investigations, and pinpoint root causes of issues. Additionally, NPM's clustered network maps enable users to assess high-cardinality traffic, identify dependencies and bottlenecks, and visualize performance metrics for individual endpoints within clusters. Overall, NPM helps users find the data they need faster, making it easier to troubleshoot complex networks.

W3C Trace Context is a standardized format designed to unify trace data from distributed tracing solutions. It addresses the problem of broken traces when requests travel between services that have been instrumented with different tools by defining two headers, traceparent and tracestate. Datadog APM supports W3C Trace Context, allowing teams to capture complete traces from services that follow this standard. This improves observability in highly distributed applications and enables troubleshooting by visualizing the complete path of requests across multiple services.

W3C Trace Context aims to unify trace data from distributed tracing solutions by defining a standardized format. Datadog APM, Serverless, Synthetic Monitoring, and Real User Monitoring (RUM) now provide out-of-the-box support for W3C Trace Context, enabling complete trace propagation across services instrumented with different tools. This allows developers to view full traces as they propagate from their root to terminal service, improving the observability of applications and enhancing troubleshooting capabilities. The standardized format splits trace context data into two headers: `traceparent` and `tracestate`, which enable interoperability between tracing tools and support vendor-specific information propagation. With W3C Trace Context support, developers can track complete paths of requests, providing invaluable context for troubleshooting and end-to-end visibility across the Datadog platform.

Feature flag tracking is a new capability in Datadog RUM that enriches browser and mobile RUM data with feature flag tags, providing visibility into application performance and user experience. This helps teams release new features safely and reliably by isolating and measuring the impact of releases tagged with feature flags on user experience and ensuring they do not cause performance issues. Feature flag tracking surfaces flaws arising from your releases before they become widespread issues, enabling you to turn off or roll back problematic features as soon as an issue is identified. It also complements other Datadog features such as Deployment Tracking and Faulty Deployment Detection.

Feature flag tracking is a new capability in Datadog RUM that enriches browser and mobile RUM data with feature flag tags, providing visibility into application performance and user experience. This allows teams to release new features safely and reliably by testing their impact on users and easily rolling back if issues arise. With feature flag tracking, teams can identify problematic features during incident investigation, check the impact of newly released features, and monitor releases for user satisfaction. The capability provides out-of-the-box analysis and metrics, such as loading times and error rates by variant, to help pinpoint issues and ensure safe releases.

Threat modeling is crucial for building secure systems and involves system modeling to map out components and threat elicitation to identify vulnerabilities. However, organizations often face challenges such as lack of visibility into development, blind spots in data flows, discrepancies between design and implementation, and insufficient documentation. Datadog Application Security Management (ASM) addresses these gaps by providing comprehensive system mapping and real-time network data integration with distributed tracing tools like Service Map. This enables organizations to visualize their entire system, identify service owners, and mitigate threats by anticipating attack flows. ASM's deep integration with APM enhances threat modeling sessions, helping organizations develop valuable threat models for their systems with confidence.

Threat modeling is a critical component of building high-performing and secure systems. It involves analyzing representations of a system to highlight concerns about security and privacy characteristics, requiring two main steps: system modeling and threat elicitation. However, developing effective threat models can be challenging due to gaps in visibility into system components, data flows between systems, discrepancies between design and implementation, lack of documentation, and unclear ownership. Datadog Application Security Management (ASM) addresses these gaps by providing a comprehensive view of an organization's entire system, including new integrations, and enabling the identification of threats through distributed tracing, visualization of system dependencies, and service owners. By leveraging ASM, organizations can develop accurate threat models that anticipate the flow of potential attacks, identify vulnerabilities, and provide mitigation strategies to strengthen security for their systems as needed.

KubeCon + CloudNativeCon Europe 2023 was a sold-out event gathering 10,000 attendees, where Datadog participated as a platinum sponsor, delivering talks on various cloud-native topics such as Kubernetes, etcd, and security. The event started with colocated sessions like CiliumCon and Observability Day, highlighting advancements in eBPF and telemetry standards integration. Datadog's contributions included discussions on scaling Kubernetes, image signing, and combating supply chain attacks, while broader conference themes addressed security, cloud efficiency, and the resurgence of service meshes. The event underscored the growth of the cloud-native ecosystem, with a focus on sustainability and innovation as enterprises increasingly engage with these technologies.

Threat modeling is essential for creating secure systems, involving system modeling and threat elicitation to address security and privacy concerns. Challenges in developing effective threat models include limited visibility into system development, discrepancies between design and implementation, and data flow blind spots. Datadog's App and API Protection (AAP) tool addresses these challenges by providing comprehensive system mapping and real-time network data through distributed tracing with Datadog APM, enabling organizations to visualize complex infrastructures and anticipate potential threats. AAP facilitates threat modeling by identifying system components and dependencies, pinpointing service ownership for efficient communication, and generating security signals for detected vulnerabilities. This iterative process helps organizations adapt to evolving systems and threats, ensuring their threat models remain accurate and actionable.