September 2016 Summaries

ActiveMQ is a message broker that facilitates message routing between different services using standard protocols, offering two versions—Classic and Artemis—that will eventually merge into a single version. Both versions support high-throughput, synchronous and asynchronous messaging, enabling connections between loosely coupled services in various programming languages. Datadog has enhanced its integration to monitor both ActiveMQ Classic and Artemis, allowing users to collect and analyze metrics, view logs, and use tags to correlate data, thus gaining comprehensive insight into messaging infrastructure. ActiveMQ supports point-to-point and publish/subscribe messaging, organizing messages in queues or addresses, and runs on the Java Virtual Machine, necessitating monitoring of JVM and host resources to ensure optimal performance. Datadog allows users to monitor memory usage, which is crucial for maintaining application health, and offers tools to collect logs and apply tags for correlating metrics, logs, and traces, enhancing troubleshooting capabilities. The integration simplifies monitoring all ActiveMQ components, providing a holistic view of system performance and contributing to the overall health of applications.

Microsoft Azure users can now monitor their Azure Redis Cache with Datadog, adding to the popular Azure services already supported by the platform. This integration allows access to over a dozen metrics from Azure Redis Cache, including commands processed, cache hits and misses, memory usage, CPU, server load, and more. With this new feature, users can create comprehensive dashboards that surface metrics from their caching layer as well as from the applications or services supported by the cache. Datadog will continue to add Azure integrations as additional Azure services are connected to the new Metrics API.

Datadog has announced the integration of Azure Redis Cache, allowing users to monitor its metrics alongside other Azure services and applications. This new integration provides access to over a dozen metrics, including commands processed, cache hits and misses, memory usage, CPU load, and server load. The integration utilizes Azure's new Metrics API, which was recently announced at Microsoft Ignite. With this addition, Datadog users can now monitor their entire application infrastructure in one place, including popular Azure services such as VMs, App Service, SQL Databases, and Logic Apps. This expansion enables users to gain a comprehensive view of their application infrastructure's performance and identify potential issues before they become major problems.

Azure Logic Apps is an automation service that allows users to connect different Azure services and other software-as-a-service (SaaS) products. It enables the creation of workflows using a visual interface with pre-built connectors, offering developer-focused features such as configurable retry logic and error handling. Datadog now supports monitoring for Azure Logic Apps, providing access to over 20 metrics related to overall workflows and granular actions. This integration allows users to create customizable dashboards, set up alerts, and easily monitor their Azure infrastructure in one place.

Azure Logic Apps is an integration service that enables users to automate responses to events by connecting different Azure services and other SaaS products. It provides a visual interface for building workflows, as well as developer-focused features such as configurable retry logic and error handling. The new integration with Datadog allows users to monitor all their Logic Apps workflows, providing access to over 20 metrics including latency, runs completed, and runs failed. This integration enables users to set up customizable dashboards, send alerts via various channels, and slice and dice metrics along any axis they want. Azure Logic Apps joins a growing list of Azure services supported in Datadog, which integrates with over 850 popular technologies.

This article discusses various tools and methods to monitor the performance of Elasticsearch clusters. It covers Elasticsearch's RESTful API + JSON, Node Stats API, Cluster Stats API, Index Stats API, Cluster Health HTTP API, Pending Tasks API, and cat API. The article also introduces dedicated monitoring tools such as ElasticHQ, Kopf, and Marvel. It concludes by suggesting the use of Datadog for comprehensive monitoring of Elasticsearch health and performance in context with metrics and events from other components of your stack.

In this article, we will explore the various metrics that can be monitored in Elasticsearch to ensure optimal performance and availability of your cluster. We will cover the following topics: 1. Search and indexing performance 2. Memory and garbage collection 3. Host-level system and network metrics 4. Cluster health and node availability 5. Resource saturation and errors By monitoring these key areas, you can gain valuable insights into how your Elasticsearch cluster is performing and identify potential issues before they become critical problems. 1. Search and Indexing Performance Search performance in Elasticsearch is determined by the time it takes to execute a search request on a given index or set of indices. The latency of search requests depends on several factors, including: - Query complexity: More complex queries with many filters and aggregations will take longer to process than simpler ones. - Number of shards: Each shard is searched in parallel, so having more shards can improve search performance. However, if there are too many shards, the overhead of coordinating between them may actually slow down search times. - Hardware resources: The amount of CPU and memory available on each node will also affect search performance. To monitor search performance, you should keep an eye on the following metrics: - Search request latency: This is the time it takes for a search request to be processed by Elasticsearch, from when it is received until the response is returned. You can use this metric to identify any sudden spikes in search times that may indicate a problem with your cluster. - Number of search requests per second: This metric provides an overall measure of how much search traffic your cluster is handling at any given time. If you see a steady increase in search request rates over time, it may be a sign that your cluster is becoming more popular and needs to be scaled up accordingly. Indexing performance in Elasticsearch is determined by the rate at which new documents can be added or updated within an index. The latency of indexing requests depends on several factors, including: - Bulk size: When adding or updating multiple documents at once, it is more efficient to use a bulk request rather than sending individual update requests for each document. However, if the bulk size is too large, it may cause performance issues due to increased memory usage and slower garbage collection times. - Number of shards: Each shard has its own independent indexing process, so having more shards can improve indexing performance by allowing multiple documents to be added or updated in parallel. However, if there are too many shards, the overhead of coordinating between them may actually slow down indexing times. - Hardware resources: The amount of CPU and memory available on each node will also affect indexing performance. To monitor indexing performance, you should keep an eye on the following metrics: - Index request latency: This is the time it takes for an index request to be processed by Elasticsearch, from when it is received until the response is returned. You can use this metric to identify any sudden spikes in indexing times that may indicate a problem with your cluster. - Number of index requests per second: This metric provides an overall measure of how much indexing traffic your cluster is handling at any given time. If you see a steady increase in index request rates over time, it may be a sign that your cluster is becoming more popular and needs to be scaled up accordingly. 2. Memory and Garbage Collection Elasticsearch uses the Java Virtual Machine (JVM) for its runtime environment, which means that it relies heavily on memory management techniques such as garbage collection. In order to ensure optimal performance of your Elasticsearch cluster, you should monitor both the amount of available heap memory and the frequency and duration of garbage collection events. Heap Memory Usage: By default, Elasticsearch allocates 1GB of JVM heap memory per data node. However, depending on the size and complexity of your dataset, you may need to adjust this setting up or down based on your specific needs. To monitor heap memory usage in Elasticsearch, you should keep an eye on the following metrics: - Heap used percentage: This metric shows what percentage of total allocated heap memory is currently being used by Elasticsearch processes. If this value consistently stays above 85%, it may be a sign that your cluster is running out of available heap space and needs to be scaled up accordingly. - Heap committed percentage: In addition to tracking how much heap memory is currently in use, you should also monitor the amount of heap memory that has been reserved for future use by Elasticsearch processes. If this value consistently stays above 95%, it may be a sign that your cluster is nearing its maximum capacity and needs to be scaled up accordingly. Garbage Collection: The JVM performs garbage collection at regular intervals in order to free up unused heap memory. However, if there are too many objects being allocated or deallocated within the same time period, it can lead to increased CPU usage and longer response times for user queries. To monitor garbage collection performance in Elasticsearch, you should keep an eye on the following metrics: - Total time spent on garbage collections: This metric shows how much total CPU time has been consumed by all garbage collection events across all nodes within your cluster over a given period of time (e.g., past hour). If this value consistently stays above 10%, it may be a sign that your cluster is experiencing performance issues due to excessive amounts of garbage collection activity. - Number of minor and major garbage collections: Minor garbage collections occur more frequently than major ones, but they typically only free up small amounts of heap memory at a time. Major garbage collections, on the other hand, are much less frequent but can potentially free up large amounts of heap memory all at once. By tracking both types of garbage collection events separately, you can gain valuable insights into how efficiently your Elasticsearch cluster is utilizing its available resources. 3. Host-Level System and Network Metrics In addition to monitoring application-specific metrics within Elasticsearch itself, it is also important to collect and analyze various host-level system and network metrics from each of your nodes. This will help you gain a more comprehensive understanding of the overall health and performance characteristics of your entire infrastructure stack. Some key areas to focus on when monitoring host-level system and network metrics include: - Disk space usage: Running out of available disk space can lead to severe performance degradation or even complete data loss in extreme cases. To prevent this from happening, you should monitor both the total amount of free space remaining on each node as well as the rate at which it is being consumed over time. - CPU utilization: High levels of CPU usage can indicate that one or more processes within your Elasticsearch cluster are consuming an excessive amount of processing power and may need to be optimized further in order to reduce their overall resource consumption footprint. - Network throughput: Monitoring network traffic patterns across all nodes within your cluster can help you identify any potential bottlenecks or capacity constraints that might be limiting the scalability and performance capabilities of your Elasticsearch deployment. 4. Cluster Health and Node Availability Ensuring high levels of availability and resiliency for your Elasticsearch cluster is critical in order to minimize downtime and maximize user satisfaction with your search application. To achieve this goal, you should monitor both the overall health status of your entire cluster as well as the individual availability states of each individual node within it. Some key areas to focus on when monitoring cluster health and node availability include: - Cluster status color (green, yellow, red): Elasticsearch uses a simple traffic light system to indicate whether or not all primary shards and replica shards are currently assigned to nodes within your cluster. A green status indicates that all shards are fully available and functioning properly, while a yellow status means that some replica shards may be missing or unassigned but searches will still return complete results. Finally, a red status signifies that at least one primary shard is missing or unavailable, which means that searches will only return partial results and new indexing operations will be blocked until the issue has been resolved. - Number of nodes: In order to ensure optimal performance and availability for your Elasticsearch cluster, you should always strive to maintain an even distribution of primary shards and replica shards across all available nodes within it. By keeping track of how many nodes are currently online at any given time, you can quickly identify if there has been a sudden decrease in the total number of active nodes that could potentially indicate a problem with one or more individual nodes. - Number of initializing/unassigned shards: When new indices are created or existing ones are updated, their corresponding primary and replica shards will initially be placed into an "initializing" state while Elasticsearch attempts to assign them to suitable target nodes within your cluster. Once this process has been completed successfully, the shards will then transition into either a "started" or "unassigned" state depending on whether they have been assigned to primary-eligible nodes or not. By keeping track of how many shards are currently in each of these different states, you can gain valuable insights into how efficiently your Elasticsearch cluster is able to handle ongoing changes and updates to its underlying dataset. 5. Resource Saturation and Errors In addition to monitoring various performance-related metrics within your Elasticsearch cluster, it is also important to keep an eye on any potential resource saturation or error conditions that might arise over time. This will help you quickly identify and address any emerging issues before they have a chance to escalate into more serious problems down the line. Some key areas to focus on when monitoring for signs of resource saturation or errors within your Elasticsearch cluster include: - Thread pool queues and rejections: Each node in an Elasticsearch cluster maintains several different types of thread pools, which are used to manage how threads consume memory and CPU resources. By keeping track of both the current number of queued threads as well as any recent instances where new threads were unable to be allocated due to queue overflows or other capacity constraints, you can gain valuable insights into whether or not your nodes are currently able to keep up with their ongoing workloads without running out of available resources. - Cache usage metrics: Elasticsearch uses two main types of caches (fielddata and filter) in order to speed up response times for certain types of search queries by reducing the amount of data that needs to be read from disk during each query execution cycle. However, if these caches become too large or consume an excessive amount of heap memory, they may actually end up slowing down overall system performance instead of improving it. To prevent this from happening, you should monitor both the current size as well as any recent instances where evictions were performed in order to free up space within each cache instance. - Pending tasks: Primary nodes within an Elasticsearch cluster are responsible for handling various types of administrative tasks such as creating new indices or assigning shards to target nodes. If these primary nodes become overloaded with too many pending tasks, it can lead to significant delays in processing time and may even result in some critical operations being dropped altogether due to resource contention issues. To prevent this from happening, you should monitor both the current number of pending tasks as well as any recent instances where new tasks were unable to be assigned due to queue overflows or other capacity constraints. By monitoring these key areas within your Elasticsearch cluster on a regular basis, you can gain valuable insights into how efficiently it is currently able to process incoming search requests and handle ongoing changes and updates to its underlying dataset. This will enable you to quickly identify any potential performance bottlenecks or resource saturation issues that might be limiting the scalability and overall effectiveness of your search application, allowing you to take proactive measures in order to address these problems before they have a chance to escalate into more serious issues down the line.

The Elasticsearch RESTful API provides a powerful toolset for collecting performance metrics about your clusters, nodes, and indices. The Node Stats API offers detailed information about nearly every metric from the previous Part 1, except for overall cluster health and pending tasks. This section covers how to query various APIs, including Cluster Health, Pending Tasks, cat API, dedicated monitoring tools such as ElasticsearchHQ, Kopf, and Marvel, which provide valuable insights into your cluster's performance and help you identify potential issues. These tools can be used individually or in combination to create a comprehensive monitoring solution for your Elasticsearch cluster.

Elasticsearch is an open source distributed document store and search engine that stores and retrieves data structures in near real-time. It relies heavily on Apache Lucene, a full-text search engine written in Java. Elasticsearch represents data in the form of structured JSON documents, making full-text search accessible via RESTful API and web clients for languages like PHP, Python, and Ruby. The cluster is made up of one or more nodes, with each node being a single running instance of Elasticsearch that designates which cluster it belongs to and what type of node it can be. There are three common types of nodes in Elasticsearch: primary-eligible nodes, data nodes, and client nodes. Primary-eligible nodes are eligible to become the primary node if the current primary node fails, while data nodes store data in the form of shards and perform actions related to indexing, searching, and aggregating data. Client nodes act as load balancers that help route indexing and search requests. Elasticsearch organizes data by storing related documents in the same index, which can be thought of as a logical wrapper of configuration. Each index contains a set of related documents in JSON format, with each shard being a complete instance of Lucene, like a mini search engine. The primary node coordinates cluster tasks, and any primary-eligible node is also able to function as a data node. Elasticsearch provides several key metrics that can help detect signs of trouble and take action when faced with problems like unreliable nodes, out-of-memory errors, and long garbage collection times. These include search and indexing performance, memory and garbage collection, host-level system and network metrics, cluster health and node availability, resource saturation and errors, and pending tasks. Monitoring these metrics can help detect potential issues before they become major problems and ensure the smooth operation of an Elasticsearch cluster.

The text discusses how to set up Datadog to collect and monitor Elasticsearch performance metrics, distributed request traces, logs, and other data. The process involves installing the Datadog Agent, configuring the integration with Elasticsearch, tagging metrics for better analysis, setting up alerts for potential issues, and deploying APM to track application performance and optimize code. The setup allows users to gain comprehensive insights into their Elasticsearch clusters' health and performance, enabling them to troubleshoot issues, optimize their infrastructure, and make data-driven decisions.

Emily Chang's concluding post in a series on monitoring Elasticsearch performance delves into addressing common challenges users face with the system. The article outlines five key issues: managing cluster status when it turns red or yellow, handling data nodes running out of disk space, optimizing slow search execution times, enhancing index-heavy workloads, and dealing with bulk thread pool rejections. It provides detailed strategies for each problem, such as using Elasticsearch's snapshot and restore module for data recovery, scaling nodes to manage disk space, employing custom routing and force merging for better search performance, adjusting shard allocation and disabling merge throttling for improved indexing, and scaling the cluster to handle high request rates. The piece emphasizes the importance of monitoring key performance metrics with tools like Datadog, which offers real-time insights and integrations to optimize Elasticsearch's functionality.

Datadog is primarily used for infrastructure and application monitoring, but its "data-driven" philosophy has influenced various aspects of users' lives. People are using Datadog to monitor beer consumption, sleep patterns, bathroom availability, energy use, and more. The company encourages users to instrument everything and collect as many metrics as possible to gain better visibility into their environment and make data-informed decisions for improvement.

Datadog users are applying the "data-driven" philosophy beyond just infrastructure and applications, using monitoring to gain visibility into various aspects of their lives, such as beer consumption, sleeping, restroom availability, and energy use. For example, a Datadog user set up a system to monitor the temperature and quantity of beer in his office kegerator, while another group of engineers built a bathroom monitoring system using door sensors and Raspberry Pis. Some users are also using monitoring to track their sleep patterns and air quality, with one user leveraging DataDog's integration with the Sense from Hello to combat mold in his bedroom. Additionally, Datadog users have created voice interfaces like Alexa integrations to interact with devices without needing dashboards, demonstrating the versatility of the platform beyond traditional use cases.

Google Compute Engine (GCE), part of the Google Cloud Platform (GCP), offers scalable virtual machines as an alternative to AWS EC2. Datadog is a tool that helps monitor GCE instances' activity, health, and performance by collecting necessary metrics. These include status checks, CPU usage percentage, I/O throughput, network throughput, and saturation. Datadog automatically tags these metrics with instance type, host name, availability zone, and more for better organization and analysis. Users can also leverage advanced features like alerting mechanisms, host maps, and outlier detection to monitor GCE effectively.

Google Compute Engine (GCE) is a cloud platform service that provides on-demand virtual machines, allowing for easily scalable infrastructure. Launched in 2013, GCE has become a serious alternative to AWS EC2 and has been adopted by major companies such as Spotify. To effectively monitor GCE instances, Datadog collects performance metrics, providing targeted alerts, correlated events, and insights into instance activity, health, and performance. The integration offers status checks, CPU usage tracking, throughput metrics for I/O, network, and saturation, and automatically tags metrics with instance type, host name, availability zone, and more. With this integration, users can slice and dice their metrics, filter and group hosts, and utilize advanced features like host maps, outlier detection, and alerting mechanisms to investigate performance issues. The integration is available for existing Datadog users and can be signed up for a free trial to start monitoring GCE instances immediately.

The company has a growing number of women in its sales team, including the director of sales. They have partnered with AcknowledgeHer to help more women enter the world of sales and tech. Their partnership includes "Wine & Wisdom" events where sales experts from Datadog and AcknowledgeHer provide insights on effective sales practices and career navigation. College students and recent graduates can network, attend panel interviews, and participate in Q&A sessions with these experts. Additionally, interns from the AcknowledgeHer program will have on-site learning opportunities at Datadog, and some of its graduates may be interviewed for sales development roles.

Marissa Parillo, a recruiter, emphasizes the importance of diversity in sales teams. She notes that while her company has made progress in increasing female representation, there is still work to be done. To address this issue, she partnered with AcknowledgeHer, an organization that aims to empower women in tech sales, through a series of events called "Wine & Wisdom". These events will provide college students and recent graduates with insights into effective sales practices and career navigation, as well as on-site learning opportunities at Datadog. The partnership is designed to help women recognize their potential in sales and prepare for careers in the field.

Datadog is hosting a contest for users to showcase their best dashboards, with prizes including Swag Packages and an Apple Watch Series 2. The contest runs from September 19th to September 23rd at 5pm ET. To participate, submit a tweet containing an image of the dashboard, the #DashingDashboards hashtag, and mention @DatadogHQ. Make sure not to disclose confidential information and ensure it's okay with your organization before submitting. One submission per Twitter handle per day is allowed. Winners will be retweeted daily and entered into a raffle for the grand prize.

The Datadog team is hosting a contest to showcase the best dashboards built with their platform. The goal is to highlight both beautiful and practical dashboards, with an emphasis on interesting use cases. Participants can submit one dashboard per day using Twitter, with specific requirements including including an image of the dashboard, the #DashingDashboards hashtag, and ending with cc: @DatadogHQ. The contest runs from September 19 to 23, with prizes including a Swag Package featuring Datadog merchandise and an entry into a raffle for an Apple Watch Series 2.

PagerDuty introduces Workflow Extensions, allowing customers to configure custom workflows with add-ons like Datadog dashboards and graphs for dynamic incident response. Embedded Datadog dashboards provide real-time alert context, while inline metric graphs offer detailed information on metrics that triggered the alert. These features can be accessed through PagerDuty's mobile interface for convenient alert investigation on the go. By integrating these tools, users can experience improved incident handling and faster recovery times.

The new Workflow Extensions feature in PagerDuty allows customers to embed custom workflows with add-ons, collaboration tools, and more. This enables the integration of Datadog dashboards and graphs directly within PagerDuty incidents for dynamic, contextualized incident response. Users can now easily embed real-time Datadog dashboards as top-level menu options and inline metric graphs into PagerDuty to gain important context for investigation. The feature is accessible through PagerDuty's Extensions tab and can be used with a free trial of Datadog.

The author shares their experience of transitioning to remote work after moving across the country. They provide nine tips for healthy, sane, and productive remote work based on personal experience and stories from colleagues at Datadog. These include having a dedicated workspace, dressing for the day, showing face in person when possible, taking breaks, increasing face-to-face communication, investing in a comfortable office setup, organizing with fellow remote workers, training oneself to focus, and finding a suitable satellite office. The author also mentions that Datadog is currently hiring for remote-friendly positions.

As John Matson, a remote worker at Datadog, discovered the challenges of transitioning to a new work environment. He consulted with colleagues who were veterans of remote work and identified key steps to ease the transition. These include building walls between professional and personal life, dressing for the day, showing your face through regular in-person visits, getting out of the house to take breaks, carving out time for face-to-face communication, investing in a comfortable office setup, organizing with a community of remote workers, training yourself to buckle down on focus, and finding your own satellite office. By implementing these strategies, remote workers can maintain productivity, stay connected with colleagues, and thrive in their work environments.

John Keats once said, "Nothing ever becomes real till it is experienced." This quote aptly applies to dynamic infrastructure and distributed applications where understanding a technology's theoretical functionality often differs from its practical implementation. Modern computer-science programs primarily focus on architecture and design fundamentals, leaving students with little knowledge of monitoring best practices or experience with monitoring tools. To address this gap, Datadog has partnered with GitHub's Student Developer Pack to offer free Datadog Pro accounts for up to 10 hosts to eligible students. These accounts provide the same features as paid versions, including data storage, alerting capabilities, collaboration options, and access to Datadog Support Team assistance. For more information on this initiative and how to obtain a free account, visit the provided link.

To address the knowledge gap in monitoring best practices among computer-science students, a partnership has been established between Datadog and GitHub's Student Developer Pack. This collaboration aims to provide students with a free Datadog Pro account for up to 10 hosts, offering them access to key features such as data storage, alerts, collaboration tools, and support from the Datadog Support Team. The goal is to eliminate the financial barrier to learning about monitoring and give students hands-on experience with modern monitoring tools.

Datadog has introduced Alert Rollup, which automatically rolls up alerts from the same triggered monitor to help users quickly identify underlying issues. With this feature enabled, users receive two notifications for triggering or recovering conditions, with subsequent messages rolled up in 20-second intervals to reduce noise. This feature is now available on all alerts and can be tested by creating an Ops channel in a preferred ChatOps integration.

Ceph is an open-source, distributed object store and file system designed for scalability, reliability, and performance, providing compatibility with S3 and Openstack Swift APIs. While Ceph includes native CLI tools for cluster health checks, production environments require comprehensive monitoring to provide historical and infrastructure-wide context. Datadog offers the ability to correlate Ceph's throughput and latency with other system components, aiding in identifying performance bottlenecks and resource provisioning. With Datadog, users can monitor Ceph at various levels, including cluster-wide metrics, pool-level details, and individual node statuses, helping to prevent issues such as monitor-quorum deadlocks and overloaded or nearly full Object Storage Daemons (OSDs). Additionally, Datadog assists in monitoring apply and commit latencies, ensuring system performance and robustness in data handling. Users can start monitoring Ceph with Datadog quickly, leveraging its features for enhanced operational insights and management.