November 2019 Summaries

AWS re:Invent is an annual conference featuring keynote sessions, workshops, and customer case studies for tens of thousands of attendees. This year's event will include interesting sessions on deep learning applications, serverless architecture, chaos engineering, eBPF performance analysis, and more. Datadog has partnered with several companies to share their experiences using AWS services and modern observability tools. The conference offers a great opportunity for networking and learning about the latest developments in cloud computing and technology.

Datadog has introduced Cloud SIEM to provide comprehensive security visibility and detection capabilities across all layers of an environment, including infrastructure, network, and applications. This unified platform offers real-time threat detection, automatic correlation and triage of security signals, and customizable rule creation with simple and flexible editors. It integrates with over 850 tools and provides end-to-end visibility into the entire stack, making it easier for development, operations, and security teams to collaborate on security investigations.

At Datadog, we rely heavily on Kubernetes and have developed solutions to better control how our clusters scale and make it easier to deploy and manage the Datadog Agent. We've open sourced these solutions to share with the Kubernetes community. Additionally, we're announcing two new generally available features for monitoring Kubernetes in Datadog: using tags to analyze distributed traces across your clusters and viewing OpenMetrics data from Kubernetes as distribution metrics in Datadog.

Yair Cohen and David M. Lentz from Datadog are sharing their experiences with Kubernetes scaling challenges and introducing new features for monitoring Kubernetes in Datadog, including distributed tracing across clusters and distribution metrics. They're also announcing the open sourcing of Watermark Pod Autoscaler (WPA) to extend the capabilities of Horizontal Pod Autoscaling, ExtendedDaemonSet to improve deployment processes, and the Datadog Operator to simplify deploying and managing Agents at scale. These solutions aim to address challenges in running Kubernetes in production and give back to the Kubernetes community.

Datadog introduces Network Performance Monitoring to provide visibility into every component of an environment and all connections between them. The tool offers multi-cloud visibility into network flows in granular detail while enabling users to aggregate and monitor data using available tags. It is fully integrated with the rest of the Datadog platform, allowing for automatic correlation of logs and request traces. Network Performance Monitoring helps optimize network traffic patterns, identify misconfigured services, and provides full-stack dependency monitoring. Built on eBPF, it offers detailed visibility into network flows with extremely low overhead, ensuring performance without trade-offs.

You can use AWS CloudFormation to automate the process of building infrastructure and manage all resources for your stacks, such as EC2 instances, load balancers, and security groups. This ensures that configurations do not drift with each new environment you spin up. Datadog has collaborated with AWS to create resources available on the CloudFormation registry, enabling users to reference the entire schema of a Datadog resource and automatically update it as new versions become available. You can use these resources to automate tasks such as enabling Datadog's AWS integration, creating monitors for your services, scheduling downtime for monitors, managing users, creating dashboards, and more. With these tools, you can create repeatable steps for provisioning and setting up monitoring for all of your resources, automating the setup process, including installing the Datadog Agent to collect metrics and logs from your instances.

OpenTracing with Scala is a focus area, as Colisweb has developed a Scala wrapper around the OpenTracing library to ensure consistency in distributed tracing and context propagation. Further enhancing Scala's integration with Datadog, scaladog serves as a Datadog API client for Scala, inviting contributions from the community. In response to the retirement of Netflix’s Hystrix Dashboard, Pivotal's Spring Cloud Services team has adopted Micrometer to ship metrics to Datadog for circuit breaking. CircleCI users can utilize circle-dd-bench to time commands and report metrics to Datadog, while Intercom’s Datadog to Terraform Converter facilitates the conversion of monitor JSON into Terraform format for monitoring-as-code practices. Krishna Sharma's blog post highlights the importance of monitoring the Kubernetes Control Plane using Datadog and Helm, and datadog_nvml enables the monitoring of NVIDIA GPU statistics within Datadog, crucial for resource management during demanding processing tasks.

The text discusses various integrations and tools that enhance monitoring and performance in software development and deployment environments. It highlights the development of Scala OpenTracing by Colisweb, which wraps the OpenTracing library for Scala, and mentions scaladog, a Scala Datadog API client. Pivotal's Spring Cloud Services team transitioned from Netflix's Hystrix Dashboard to using Micrometer for circuit breaking metrics, sending them to Datadog for better monitoring. CircleCI users can benefit from the circle-dd-bench tool to report command execution times as custom metrics in Datadog. Intercom's Datadog to Terraform Converter facilitates converting monitor JSONs into Terraform alarm formats, supporting a monitoring-as-code approach. Krishna Sharma's blog post explains monitoring the Kubernetes Control Plane using Datadog and Helm to ensure efficient cluster operations. Additionally, datadog_nvml allows users to track NVIDIA GPU statistics within Datadog, useful for managing resources in intensive processing tasks.

AWS re:Invent is an annual gathering of tens of thousands of AWS staff, partners, and users for a full week of keynote sessions, feature announcements, customer case studies, hands-on workshops, and more. The sponsor hall offers a great way to learn about the newest AWS features and partner products, while conference sessions can be educational and inspiring. Some interesting sessions include "Using deep learning to track wildfires and air quality" which discusses using satellite imagery and meteorological data to predict wildfires and air quality in real time; "Refactoring a monolith to serverless in 8 steps", which outlines discrete steps for successfully refactoring a monolith to serverless; and "Scaling to billions of requests the serverless way at Capital One", which explores how Capital One used serverless streaming architecture to provide real-time insights for millions of customers. Additionally, there are sessions on chaos engineering in a serverless world, BPF performance analysis, and partnerships between AWS and Datadog, including sessions on using observability tooling to build confidence and accelerate migrations.

Datadog has introduced a new feature called "dark mode," which provides users with an alternative way of visualizing their data by using light-colored text on darker backgrounds and incorporating vibrant color palettes for host maps. This feature can be activated from anywhere within the app, allowing users to switch between themes easily. Dark mode is now generally available for all users, offering a fresh perspective on monitoring infrastructure.

Datadog has released a new feature called dark mode, which provides a striking way to visualize data by using light-colored text on darker backgrounds and graphical elements. The new color palettes for host maps, Viridis and Plasma, are also available in dark mode, providing a vibrant layout of hosts. Users can activate dark mode from anywhere in the Datadog app by hovering over their avatar and selecting the "Dark" theme, or cycle between themes using the shortcut Ctrl+Opt+D. The feature is now generally available for users to instantly start visualizing, troubleshooting, and exploring their infrastructure in a new light.

The MapR Data Platform is designed to manage, analyze, and store all types of data at scale across various infrastructures and locations. It leverages dataware, an abstraction layer that separates data from dependencies. The platform supports open source engines and tools such as Apache Hadoop, Hive, and HBase. Datadog's new integration with MapR 6.1+ provides comprehensive visibility into the distributed big data architecture, allowing users to quickly diagnose resource and performance-related issues. This includes monitoring key metrics from a user's MapR environment, tracking file system activity levels, detecting database query performance issues, and navigating changes in the flow of messages.

The MapR Data Platform is an enterprise software stack that enables organizations to manage, analyze, and store all their data at scale. It leverages the abstraction layer "dataware" to separate data from dependencies and supports open source engines and tools such as Apache Hadoop, Hive, and HBase. The platform provides comprehensive visibility into its deployment through its integration with Datadog, allowing users to monitor performance metrics and health status, track file system activity levels, detect database query performance issues, navigate changes in the flow of messages, and keep MapR components on course.

In this article, the author discusses how to use Datadog to monitor Amazon MQ metrics and logs alongside other AWS services. The integration of Datadog with AWS allows users to visualize performance data from their messaging infrastructure and create custom dashboards for monitoring various aspects of their applications and infrastructure. The author provides detailed instructions on setting up the AWS integration, enabling metric collection, tagging metrics, creating custom dashboards, collecting and analyzing Amazon MQ logs, and setting up alerts based on metrics and logs. The article emphasizes the importance of using tags to filter and aggregate data, allowing users to gain a deeper understanding of their messaging infrastructure's performance.

Amazon MQ is a managed messaging service hosted on the AWS cloud, designed to help users easily migrate ActiveMQ message brokers or any other supported messaging protocols to the cloud. It supports several ActiveMQ versions and can be used for various purposes such as integrating disparate systems, transitioning applications from monolithic architecture to microservices, and improving data reliability in dynamic environments. Amazon MQ uses resources to store and send messages between client programs through brokers that route messages between the nodes in a distributed application. Key metrics to monitor include MemoryUsage, ConsumerCount, ProducerCount, QueueSize, ExpiredCount, EnqueueCount, DequeueCount, HeapUsage, CPUUtilization, StorePercentUsage, and TotalMessageCount.

Datadog has announced native support for FireLens for Amazon ECS, allowing centralized logging from traditional infrastructure to serverless components. In partnership with AWS, built-in Fluent Bit support is provided for seamless routing of container logs from AWS Fargate. Users can configure Fluent Bit directly in their Fargate tasks and monitor the logs alongside other Fargate metrics for visibility into their containerized services. The integration also supports automatic ingestion of additional tags, enabling easy sifting, filtering, and analysis of logs. Additionally, Datadog's Autodiscovery feature automatically detects containerized services on Fargate and configures checks for 650+ integrations like Redis and NGINX.

Datadog is announcing native support for FireLens for Amazon ECS, which streamlines logging by enabling the configuration of a log collection and forwarding tool such as Fluent Bit directly in Fargate tasks. This allows users to centralize logging from their entire stack, including traditional infrastructure and serverless components. With built-in Fluent Bit support, users can now seamlessly route container logs from AWS Fargate to Datadog, enabling them to monitor these logs alongside other Fargate metrics. To start forwarding Fargate container logs to Datadog with Fluent Bit, users need to specify a new container definition for their Fargate task, which creates a FireLens container using Amazon's Fluent Bit image. The user can then create a new log configuration that uses AWS FireLens as the log driver and specifies output options for Fluent Bit, associating the logs with a specific service and source in Datadog. With this integration, users can monitor their Fargate tasks in Datadog, viewing logs alongside metrics from all of their Fargate containers, including CPU and memory usage. Additionally, Autodiscovery allows Datadog to automatically detect containerized services on Fargate and configure Datadog Agent checks for 850+ integrations like Redis and NGINX.

Amazon MQ is a managed messaging service hosted on the AWS cloud, providing a managed ActiveMQ messaging service that routes messages between nodes in a distributed application. It supports several ActiveMQ versions and allows for easy migration to the cloud. Amazon MQ can help solve problems like integrating disparate systems and improving data reliability in dynamic environments. The service provides metrics such as MemoryUsage, ConsumerCount, ProducerCount, QueueSize, ExpiredCount, EnqueueCount, DequeueCount, HeapUsage, CPUUtilization, StorePercentUsage, and TotalMessageCount to monitor the performance and resource usage of messaging infrastructure. These metrics help ensure that message brokers are operating efficiently with plenty of resources at all times, and can be used to identify potential issues such as running out of heap memory or disk space.

Datadog is a monitoring platform that integrates with AWS services, including Amazon MQ. To monitor Amazon MQ brokers and destinations alongside other AWS services, Datadog must be integrated into an AWS account. This integration enables the collection of Amazon MQ metrics, visualization through pre-built and custom dashboards, analysis of logs, creation of alerts based on metrics and logs, tagging of metrics for filtering and aggregation, and deployment of a log forwarder to collect and analyze Amazon MQ logs. Datadog's monitoring capabilities provide real-time insights into performance and potential issues with Amazon MQ infrastructure, allowing users to create customized dashboards, search and filter logs, tie logs to metrics, and set up alerts to stay informed. By integrating with more than 850 technologies, including AWS services such as EC2, Lambda, and S3, Datadog provides a single platform for monitoring all applications, services, and infrastructure.

The text discusses the use of Amazon CloudWatch and the ActiveMQ Web Console to monitor Amazon MQ metrics and logs. CloudWatch, a comprehensive monitoring solution, allows visualization and alerting on metrics from Amazon MQ and other AWS services, as well as collecting and analyzing logs, enabling users to set alarms for potential issues with the broker's performance. The ActiveMQ Web Console, built into each Amazon MQ broker, provides a basic view of certain metrics but lacks the ability to graph trends or create alerts. The text emphasizes the importance of a monitoring platform that offers complete visibility across the messaging infrastructure, hinting at the potential use of Datadog for in-depth analysis in a subsequent part of the series.