July 2019 Summaries

Apache Hive is an open-source interface that enables users to query and analyze distributed datasets using SQL commands. It compiles SQL commands into an execution plan, which it then runs against your Hadoop deployment. Datadog's integration allows you to monitor Hive metrics and logs in context with the rest of your big data infrastructure. You can optimize Hive memory usage by tracking client sessions alongside memory usage from two Hive components: HiveServer2 and the Metastore. Additionally, you can troubleshoot slow queries by tracking the time SQL operations spend in different states and investigate execution errors in context using Datadog's log processing pipeline. The integration also provides visibility across your distributed big data architecture, including technologies like AWS Elastic MapReduce and ZooKeeper.

Segment, a customer data infrastructure company, has developed a metrics pipeline and API to provide users with real-time insights into the performance and reliability of their data pipelines. The metrics pipeline collects data for every Source-Destination pair, allowing users to track the unique delivery status of their data to Destinations. This information is exposed in the Segment Config API and can be monitored via a new Datadog integration. The architecture supporting this metrics pipeline includes Kafka for processing and collecting metrics, AWS Aurora MySQL for aggregating metrics, and a REST API service for exposing it as timeseries data. With these tools, users can visualize and analyze their data pipelines to gain deeper insights into their performance and reliability.

Segment is a customer data infrastructure that collects data from sources and routes it to destinations as quickly and reliably as possible. To provide real-time insights into the performance and reliability of their data pipelines, Segment developed a metrics pipeline and API. The pipeline measures, aggregates, and stores data about outbound events delivered every day, providing users with a global view of the real-time status of hundreds of partner APIs. With this data, users can quickly visualize if a Destination is having service problems, set up alerts, and investigate errors in just a few clicks, resulting in confidence in using Segment for customer data infrastructure. The pipeline also delivers data at scale, utilizing Kafka, AWS Aurora MySQL, and a REST API service to process and expose the data as timeseries data. The integration with Datadog allows users to visualize and alert on delivery metrics and errors, providing deeper insights into their data pipelines and enabling them to build custom dashboards and alerts to track real-time health.

Apache Hive is an open source interface that allows users to query and analyze distributed datasets using SQL commands. Hive compiles SQL commands into an execution plan, which it then runs against the Hadoop deployment. This integration with Datadog provides visibility into Hive metrics and logs, allowing users to monitor memory usage, troubleshoot slow queries, investigate execution errors in context, and gain insights across their entire big data infrastructure. The integration includes a log processing pipeline that parses Hive logs for key information, making it easier to identify issues and pinpoint root causes. With Datadog, users can track client sessions alongside memory usage from Hive components such as HiveServer2 and the Metastore, allowing them to determine when demand is likely to be high and optimize performance accordingly.

The Dashboard Details feature provides context around every dashboard in an organization's environment. It includes three parts: a Dashboard Description that explains the purpose of each dashboard, a Suggested Dashboards section with auto-generated recommendations based on usage patterns, and a Most Active Users section listing people who spend the most time viewing the dashboard. This feature aims to simplify cross-team collaboration and make it easier for users to understand the context of dashboards for troubleshooting purposes.

Miranda Kapin and Will To from Datadog have released the Dashboard Details feature to provide instant context around every dashboard, helping users understand how they're being used in their organization. This new functionality includes a Dashboard Description that explains the intended purpose of a particular dashboard, a Suggested Dashboards section with an auto-generated list of recommended dashboards based on usage patterns, and a Most Active Users section that identifies subject matter experts. With these features, users can simplify cross-team collaboration and get more out of their dashboards by making it easier to troubleshoot issues and understand the data shown. The Dashboard Details panel also includes support for Markdown, allowing users to customize descriptions and edit them directly in the Datadog UI or programmatically through Terraform.

Ansible is a versatile automation tool that facilitates the provisioning, management, and deployment of infrastructure and applications, particularly useful for large-scale systems. It enables users to create a consistent and repeatable process for configuring infrastructure across platforms such as AWS, including the installation and configuration of the Datadog Agent for monitoring. The process involves setting up a dynamic inventory of AWS EC2 instances, creating an Ansible playbook to deploy the Agent, and using Datadog to monitor the instances. Ansible's playbooks are idempotent, ensuring that systems achieve the desired state regardless of the number of executions. The guide also emphasizes the use of both dynamic and static inventory files, the importance of prerequisites like having an AWS IAM user with necessary permissions, and the role of Ansible vault in securing keys. The integration with Datadog provides real-time monitoring and reporting capabilities, helping users to efficiently manage and observe their dynamic infrastructure. Users can also explore Datadog's extensive integration options and leverage Ansible's callback plugin for detailed insights into deployment processes.

Apache Ambari is an open source management tool that helps organizations operate Hadoop clusters at scale. Datadog's new integration for Ambari allows users to monitor the performance of their Ambari servers along with other technologies in their stack, such as Apache Spark and PostgreSQL. The integration includes an out-of-the-box dashboard displaying key resource utilization metrics from Ambari servers, and automatically tags Ambari metrics by Ambari cluster, service, and component for easy drill down. Users can also customize dashboards to include metrics from related components of their infrastructure. Datadog's integration with all of the database backends Ambari supports makes it easy to monitor Ambari's backend database alongside the overall health of Ambari servers. Additionally, users can configure Datadog to collect Ambari-managed Hadoop metrics without installing the Datadog Agent directly on all of the servers Ambari manages. With this integration, users gain real-time visibility into Ambari's resource usage and availability, allowing them to troubleshoot performance issues across their entire big data architecture.

Log management solutions can help filter, aggregate, and analyze log data by standardizing attribute names across different technologies and data sources. This enables easier analysis of HTTP traffic and performance across the entire infrastructure. Datadog's common schema uses a prefix for similar attributes from various log integrations, allowing users to filter and aggregate logs by these terms. The schema is also extensible, enabling customization or addition of standard attributes for custom log data. Using a common schema enhances visibility into an organization's environment and can be easily implemented with Datadog's log processing features.

Datadog has released an integration for Apache Ambari, allowing users to monitor the performance of their Hadoop clusters, including resource utilization metrics and logs. The integration includes a dashboard that displays key resource utilization metrics from Ambari servers, as well as customizable dashboards to include metrics from related components. Users can also set up threshold or machine-learning-driven alerts on Ambari server resource availability to take preemptive action before issues arise. Additionally, the integration allows users to collect logs from Ambari, including audit logs, server logs, and alert logs, providing further granular insights into their Ambari deployment. The integration integrates with more than 850 technologies, allowing for unified insights across every component of a dynamic big data architecture.

A common schema for log attributes can unify data from different sources and technologies by standardizing attribute names, making it easier to analyze and search for data. This is particularly useful in complex environments where multiple components emit logs with varying naming conventions. By using a centralized platform like Datadog, teams can implement an automatic, standardized naming convention without modifying their code or logging strategies, enabling better visibility into their environment and facilitating the analysis of similar information from different technologies and types of technologies.

Datadog's JavaScript logging library helps detect and debug user-facing issues by enabling the monitoring of JavaScript console logs alongside backend logs, metrics, and distributed request traces from infrastructure and applications. This allows for real-time tracking of client-side errors and events to ensure timely issue resolution and maintain a positive user experience. By analyzing frontend error logs, users can identify patterns and trends, such as which browsers or regions are most affected by issues, and take corrective action to address these problems. Additionally, the library enables logging of custom user events, providing richer context around JavaScript console logs and enabling businesses to track key activities and improve their services. With this feature, developers can gain full-stack visibility across frontend and backend issues, allowing for continuous improvement of availability and performance.

At Dash 2019, Datadog announced several new features and products aimed at enhancing the coverage of modern application stacks. These include Network Performance Monitoring, Real User Monitoring, support for collecting browser logs, and single-pane-of-glass visibility for serverless environments. Other notable releases include Serverless view, SLOs and SLIs in Datadog, Browser Logs, Mobile Application, Real User Monitoring, Private locations for Datadog Synthetics, Log Rehydration™, Metrics from Logs, Watchdog for Infrastructure Metrics, Metric Correlation, Metrics without Limits™, Tracing without Limits™, and Trace Outliers. These features aim to provide comprehensive monitoring capabilities across various aspects of modern applications and infrastructure.

Datadog has announced the general availability of serverless monitoring for AWS Lambda functions. The Serverless view allows users to search, filter, and explore all their AWS Lambda functions in one place, providing detailed performance data for each function. Users can troubleshoot performance issues with distributed traces, collect custom metrics in real-time from serverless functions, and visualize Lambda functions along with their dependencies using the Service Map. Datadog's integration with AWS X-Ray enables users to trace Lambda functions and visualize how requests travel between various components of a serverless architecture. The addition of Lambda tracing complements existing metrics and logs provided by Datadog, allowing users to monitor key business metrics without adding latency to their applications.

Datadog has announced several new products and features to enhance its platform's capabilities, including Network Performance Monitoring, Real User Monitoring for mobile applications, and browser logs. The company also introduced a mobile app for easier issue triage on-the-go, as well as Log Rehydration to reload archived logs on demand. Additionally, Datadog added support for collecting browser logs, single-pane-of-glass visibility for serverless environments, metrics from logs, Watchdog for infrastructure metrics, metric correlation, and tracing without limits. These features are designed to provide broader coverage of the modern application stack and improve visibility into applications and infrastructure.

As applications grow, managing the volume and cost of log data becomes challenging, prompting organizations to limit or pre-filter logs and potentially miss crucial information. Datadog's Logging without Limits™ offers a solution by allowing all logs to be sent to Datadog cost-effectively, enabling real-time streaming and selective retention for analysis while archiving the rest in cloud storage for comprehensive historical access. New features, Metrics from Logs and Log Rehydration™, enhance this flexibility by enabling metric generation from logs to track trends and reloading logs from archives for in-depth analysis. These features support better log retention practices and cost reduction, allowing users to dynamically manage and analyze their log data.

Datadog has introduced Cloud Network Monitoring, a feature designed to provide comprehensive visibility into network flows across multi-cloud environments. This tool enables users to monitor network connections with granular detail and aggregate data using tags, allowing for insightful analysis of performance and dependencies. By leveraging eBPF technology, the feature offers extensive network observability with minimal performance impact. It supports dynamic infrastructure monitoring by focusing on longer-lived abstractions, such as services and availability zones, rather than transient components like individual hosts. This functionality helps identify misconfigured services and optimize network traffic patterns for performance and cost efficiency. Integrated with Datadog's platform, it allows users to correlate logs and request traces, offering a full-stack view of network dependencies and facilitating the identification of network issues.

Signal Sciences has integrated its next-gen WAF with the Datadog platform to provide users with visualization and analysis of web application activity on their Datadog dashboards. This integration allows teams to monitor their application security alongside infrastructure and applications, simplifying workflows. The Signal Sciences dashboard provides a high-level overview of key web layer attack metrics, enabling real-time identification of potential threats. With this integration, users can gain insights into malicious behavior against their applications and APIs using the most trusted web security solution on the market.

Signal Sciences has integrated its next-gen web application firewall (WAF) with the Datadog platform, allowing users to visualize and analyze web application activity in their Datadog dashboards and receive alerts about potential attacks. The integration enables teams to monitor their application security activity on the same platform they use for infrastructure and applications, simplifying workflows and providing critical visibility into malicious activity across web applications. With this integration, Signal Sciences provides powerful visualizations to aid in faster identification of potential threats, including total traffic allowed and blocked, OWASP Top 10 attacks, anomalous web requests, and more. The dashboard also streamlines the process of identifying potential attacks and accelerating remediation times by making it a cross-reference tool for high traffic activity or resource spikes. Overall, this integration empowers operations and development teams to proactively see and respond to web attacks in real-time, providing critical security monitoring with Signal Sciences.

Azure Lighthouse is a new feature that streamlines access management for users and applications across different Azure tenants. Managed service providers (MSPs) can now manage their customers' environments more efficiently with this feature. Datadog, a monitoring service, has announced support for Azure Lighthouse, allowing MSPs to implement a scalable approach to monitoring multiple customer Azure environments. Prior to Azure Lighthouse, managing multi-tenant environments was time-consuming due to the need for separate app registrations for each tenant ID. With Azure Lighthouse, a single app registration can be used to manage Datadog integration across multiple tenants while maintaining control over data access. Integrating Datadog with Azure Lighthouse is similar to the existing Azure integration setup process and provides deep visibility into customers' Azure environments. MSPs can start monitoring customer environments with Datadog's Azure integration today, or begin a free 14-day trial for comprehensive insights across multi-tenant Azure environments.

Azure Lighthouse is a new feature that provides improved access management for users and applications across different Azure tenants, allowing managed service providers (MSPs) to manage their customers' environments more easily. Prior to this feature, MSPs had to create separate app registrations for each tenant ID, which added considerable overhead. Now, with the integration of Datadog and Azure Lighthouse, MSPs can use a single app registration to manage the Datadog integration across multiple tenants, while maintaining control over data access. This setup enables deep visibility into customers' Azure environments, with automated enrichment of metrics with metadata from Azure, and allows for tailored dashboards and alerts based on individual customer subscriptions.

Amazon EventBridge is a serverless event bus that enables real-time data routing from applications to targets like AWS Lambda, facilitating event-driven application development with built-in security and error handling. The new integration allows Datadog alerts to be routed to EventBridge with minimal setup, enabling the configuration of EventBridge's targets to initiate auto-remediation pipelines and analytics queries. Built on CloudWatch Events, EventBridge supports routing rules for event buses, making it compatible with various AWS services, and it enhances Datadog alerts by allowing customizable event buses. Use cases include automatically allocating more memory to Lambda functions when memory limits are exceeded and auto-scaling EC2 environments based on custom business metrics. Users can start integrating EventBridge with Datadog to improve alert notifications and automate issue remediation across their application architecture.

The text provides an in-depth exploration of various methods to monitor and troubleshoot a Consul cluster by accessing metrics and logs. It outlines the use of the Consul CLI and API for on-demand data retrieval and troubleshooting, including commands like `consul debug` for detailed health and performance insights, and techniques to collect metrics over time using telemetry sinks. The document emphasizes the importance of comprehensive monitoring, suggesting configuring metrics sinks to retain data beyond default limits for long-term visibility. It also covers the utility of logging for identifying trends not captured by metrics, and how logs can be processed for deeper analysis. Additionally, it describes the role of Consul's browser UI and HTTP API in providing a high-level overview of cluster health, enabling users to monitor node and service availability through health checks. Finally, the text hints at future guidance on integrating Consul data with Datadog for enhanced visibility, while acknowledging HashiCorp's contribution to the content's accuracy.

ApacheBench (ab) is a command-line benchmarking tool designed to measure the performance of web servers, initially for Apache but applicable to any HTTP server. It evaluates server capabilities by sending a high volume of HTTP requests and recording metrics such as request latency and success rates. The tool allows users to configure parameters like the number of requests, timeout limits, and request headers, providing output in the form of detailed reports. These reports include metrics on request latency, connection times, and success rates, helping users identify areas for optimization in server performance. ApacheBench operates on a single thread and has certain constraints, such as requiring sufficient CPU and memory resources on the testing host and potentially not reflecting real-world request handling due to its send-everything-at-once approach. Despite its limitations, ApacheBench is useful for assessing server efficiency and optimizing server configurations, with its metrics offering insights into latency distribution and response success rates.

HashiCorp Consul is a software solution designed for managing clusters by facilitating service discovery and configuration across distributed systems. Released in 2014, it is widely adopted for its capabilities in service discovery, distributed key-value storage, automated health checks, and its recent addition of a service mesh. Consul operates by running an agent on each host, ensuring consistent network and configuration data is accessible across the cluster, using the Raft algorithm for data consistency and the Gossip protocol for membership information dissemination. Monitoring these components is crucial for maintaining cluster health; the article explains the importance of tracking various metrics related to Raft and Gossip protocols, resource utilization, and communication effectiveness to ensure optimal performance and reliability of the distributed system. Consul's metrics are accessible through the go-metrics library, offering insights into leadership transitions, Raft commits, and communication protocols, while also addressing how to manage memory usage effectively.