July 2014 Summaries

Datadog's webhooks integration allows teams to automatically send SMS messages via Twilio when an alert triggers in their application or environment, providing real-time communication and facilitating quick troubleshooting. To create a webhook that uses the Twilio SMS API to send a message, users navigate to the webhooks integration tile in the Datadog app, give the webhook a name, enter the Twilio SMS API URL, and define the payload which includes the number the message should be delivered to, the number it will come from, and the body of the message. The webhook can be triggered by various means such as Synthetic tests, event monitors, and other types of alerts, allowing teams to automatically interact with third-party and internal services as changes occur in their environment. To make the most of the webhooks integration, users can enrich their webhook's payload by using links and variables to create a message that can help reduce mean time to resolution (MTTR) by giving incident responders a head start on investigating issues.

Ashwin Jiwane, Software Engineer at PagerDuty, explains how the company uses data to improve its notification pipeline reliability and ensure timely delivery of alerts to customers. By combining Datadog and PagerDuty, they have created an End-to-End Third-Party Provider testing practice that proactively identifies outages in their third-party providers' systems and quickly finds a replacement to minimize or avoid customer impact. This involves setting up three phones with different mobile carrier networks and using an internally-built mobile app to send SMS alerts. The time taken for each SMS to reach the designated phone is measured, and if it exceeds acceptable thresholds, a PagerDuty alert is sent to the on-call engineer who switches priority levels of providers accordingly. This approach helps limit the impact of failures on customers and ensures that third-party vendors are consistently tested for reliability.

At PagerDuty, we use Datadog to measure the effectiveness of third-party services that deliver SMS alerts to our customers. We can't control these providers, but by combining Datadog and PagerDuty, we've created a practice that proactively discovers outages in one of our provider's systems, quickly finds a replacement, and minimizes customer impact. This process involves setting up multiple phones with different mobile carrier networks, using an internally-built mobile app to send SMS alerts, and analyzing the time taken for each SMS to reach the designated phone and how long it takes to reply back. When a provider exceeds our acceptable thresholds, we consider it downgraded and take action to switch priority levels of each provider to ensure the most functional one is used first.

Jason Moiron, a Datadog Software Engineer, discusses the inefficacy of using Go's `ioutil.ReadAll` function, arguing that relying on this function is often a programming mistake due to its inflexibility and inefficient memory usage. He highlights the advantages of using `io.Reader`, which offers more flexibility and better memory control, especially important in environments with limited memory and highly concurrent processes. Moiron points out that many data sources exceed the memory capacity of most computers and that streaming data can be processed more efficiently without waiting for entire datasets to load. He observes that the programming community tends to overuse `ioutil.ReadAll` despite the superior capabilities of streaming APIs, attributing this to the simplicity of byte slices compared to abstract streams. Moiron encourages the Go community to adopt better practices by identifying and replacing unnecessary uses of `ioutil.ReadAll` on platforms like GitHub, which would enhance code quality and benefit the overall ecosystem.

To celebrate Batman's 75th Anniversary and honor Batman Day, a custom Agent check was developed to graph the Batman Curve in Datadog. Users can install this check and send Bat metrics on a schedule. By combining these metrics in one graph using timeseries visualization and plotting the average of dd.batman.top and dd.batman.bottom, users can create their own Bat-signal like Commissioner Gordon. This feature allows for quick system monitoring and performance analysis, with Datadog offering a 14-day free trial to try it out. Happy Batman Day!

To celebrate Batman's 75th Anniversary and provide a fun challenge, the team created a custom Agent check for Datadog to visualize the Batman Curve. The Bat-signal can be sent to Datadog by plotting an equation resembling Batman and submitting coordinates on a schedule. Users can replicate this by installing the custom Agent check and combining metrics in one graph, averaging `dd.batman.top` and `dd.batman.bottom` metrics for a signal. By using Datadog's free trial, users can create cool graphs and gain actionable insights into their infrastructure performance. The team invites users to try it out on Batman Day.

The database market is rapidly evolving, driven by the explosion of big data applications. Fractal Tree indexing is a hidden innovation that replaces traditional B-tree indexing in many NoSQL databases, including Tokutek's TokuMX and TokuDB products. These indexes are designed to handle large volumes of unstructured data efficiently. To monitor and analyze performance, Tokutek has developed an integration with Datadog's monitoring service, offering a comprehensive framework for enterprise-grade monitoring, analysis, alerting, and reporting. Key metrics monitored include opcounters, replication lag, cache table utilization, and storage size, providing actionable insights into cluster health and infrastructure optimization opportunities. The Datadog-TokuMX integration also allows for customization of dashboards to fit specific needs, enabling users to create tailored visualizations and alerts based on their unique requirements.