January 2019 Summaries

The text highlights several key achievements of the Datadog team in 2018, including the release of a new monitoring agent written in Go, expanded support for Java and Node.js, improved log management features such as Logging without Limits and Log Patterns, introduction of App Analytics, Serverless view, Watchdog, and numerous other feature updates. The company also opened a European region, launched the Datadog Learning Center, and made significant improvements to its Host Map, dashboard lists, and API endpoints. Additionally, the team expanded its ecosystem with over 250 new integrations, including support for cloud platforms, databases, and security services. With these achievements, the company is poised for continued growth and innovation in the monitoring space.

Continuous integration (CI) has become an essential practice in software development, and Datadog offers out-of-the-box support for several CI tools to provide visibility into these pipelines. Monitoring CI servers helps identify bottlenecks and correlating CI activity with application performance metrics can pinpoint problematic deployments. Datadog's integrations with popular CI tools, such as Jenkins, GitLab, CircleCI, Travis CI, TeamCity, and Azure DevOps, enable users to watch for problems in their pipelines and troubleshoot issues by comparing events in CI pipelines with trends in metrics, logs, or request traces. Datadog also allows users to set up alerts based on specific information from their CI infrastructure.

Datadog provides out-of-the-box support for several continuous integration tools, including Jenkins, GitLab, CircleCI, and Travis CI, making it straightforward to monitor and troubleshoot CI pipelines. With Datadog's integrations, users can gain full visibility into their CI pipelines, identify bottlenecks, and pinpoint deployments that have degraded performance. Additionally, Datadog's dashboards allow users to visualize data from their own CI infrastructure, combine system metrics with status metrics or events from pipelines, and monitor job duration over time to see how changes in the CI workflow affect performance. The platform also enables users to troubleshoot issues by comparing events in CI pipelines with trends in metrics, logs, or request traces, and alert on worrisome trends such as unexpected declines in completed builds during busy periods. Integrations with popular CI tools are straightforward, and Datadog can be used with a wide range of CI tools, including GitLab, Jenkins, CircleCI, Travis CI, TeamCity, and Azure DevOps.

Dash 2019, Datadog's second annual conference, is set to take place in New York City on July 16 and 17 at Pier 60, Chelsea Piers, offering attendees stunning views of the Hudson River and NYC landmarks. The event will gather software developers and operations engineers to explore advancements in application and infrastructure scaling through a variety of presentations, workshops, and networking opportunities. Last year's highlights included keynotes from industry leaders like Datadog CEO Olivier Pomel and former NASA Flight Director Paul Hill, alongside breakout sessions on topics such as microservices, chaos engineering, and scaling for AAA game releases. The 2019 conference promises an expanded selection of keynote sessions, breakout talks, and hands-on workshops, complemented by networking activities and dining experiences, with early-bird pricing and lodging discounts available for early registrants. Attendees are also encouraged to propose talks to share their own experiences and insights at the event.

The article discusses how advanced statistical methods are reshaping the user experience (UX) of alerts in monitoring tools. It explains that nearly all alerts today are defined using four dimensions: scope, metric, threshold(s), and time. However, these constraints have limitations such as static thresholds not adapting to changing conditions, warning thresholds being a crutch, and scope having to be defined upfront. The article then introduces algorithmic alerting methods like forecasting, anomaly detection, and outlier detection that offer more flexibility in tracking thresholds and utilizing time. It also talks about algorithmic feeds that don't require upfront configuration and can watch things not explicitly configured. The author predicts that the future of alerting UX will involve matching patterns to people using supervised algorithmic feeds, making manual definition of alerts unnecessary.

Steve Boak discusses how advanced statistical methods are transforming the user experience (UX) of alerts in monitoring tools. He highlights four main dimensions used to define alerts: scope, metric, threshold, and time. However, these constraints have limitations that can lead to static thresholds not adapting to changing conditions, warning thresholds being a crutch for false positives, and scope needing to be defined upfront. This is where algorithmic alerting comes in, introducing three main varieties: forecasting, anomaly detection, and outlier detection. These methods offer flexibility in tracking thresholds and utilizing time, but still require upfront configuration. Algorithmic feeds may use some of these methods with loose constraints, allowing for a more dynamic experience. However, the future of alerting UX relies on matching patterns to people, which will determine the quality of the experience and potentially make manually defined alerts unnecessary.

This post demonstrates how to use AWS CloudFormation to automatically deploy infrastructure that is preconfigured to send metrics and logs to Datadog. The process involves creating an EC2 instance running Apache web server, installing the Datadog Agent, configuring it with your API key, and enabling Datadog's integration with Apache. A sample CloudFormation template is used for this purpose. AWS CloudFormation is a configuration management service that allows you to create and update infrastructure programmatically using templates. The demonstration also includes the anatomy of the sample CloudFormation template and how it creates two AWS resources and a Datadog resource.

CloudFormation is a configuration management service from AWS that allows users to create and update infrastructure programmatically using JSON or YAML templates. These templates define the properties of each resource, such as an EC2 instance, and specify the desired state of the infrastructure. CloudFormation can deploy infrastructure that's readily observable, including the Datadog Agent, which collects and forwards metrics, logs, and distributed request traces to a user's Datadog account. The template defines two AWS resources - an EC2 instance running Apache web server, and a security group allowing traffic on port 80. It also includes a Datadog resource that creates a monitor alerting when the EC2 instance becomes unavailable. The template accepts parameters for the API key and application key, which are used to configure the Agent and create the monitor. The configuration is organized into three configsets: Install, Configure, and Restart, each defining the commands and packages to be run on the instance. CloudFormation triggers a script that reads all the config keys to determine how to achieve the desired configuration. The template defines an EC2 instance with an Amazon Linux 2 AMI from the us-west-1 region, a custom security group exposing port 80 externally, and output variables for URL and InstanceDashboard values returned when the template is executed.

In a Community Interview series, Brett Langdon, a software engineer and DevOps consultant, discusses his extensive contributions to the Datadog ecosystem, including maintaining the open-source Node package node-dogapi. Brett highlights the evolution of Datadog's open-source experience, noting that the community and the company have significantly contributed to the availability of tools, reducing the need for custom solutions. He shares insights into the challenges of maintaining long-standing projects and emphasizes the importance of community feedback in sustaining software projects. Despite not actively using node-dogapi himself, he manages its upkeep, driven primarily by community contributions. Brett, who values Datadog's receptiveness to his contributions, will soon join the company as an Open Source Software Engineer, focusing on Python APM, a decision strongly motivated by his passion for open-source work and his admiration for Datadog.

In this interview, Zach Daniel, a software engineer with a passion for Elixir, discusses his journey from using Node.js and RethinkDB to adopting Elixir, particularly for high-performance applications at Albert.io, now continuing his work at DockYard. Zach highlights the benefits of Elixir's tools, such as Ecto and Phoenix, which offered the robust database interaction needed for their projects. He describes his role in developing Spandex, the Datadog Elixir APM library, detailing the challenges of integrating effective monitoring and tracing capabilities to ensure seamless application performance without introducing bottlenecks. Zach emphasizes the simplicity and efficiency of Elixir compared to other languages and shares his experiences in managing state and concurrency. As Spandex gained popularity, Zach's engagement with the Elixir community grew, leading to further development and improvements. He continues to actively contribute to Spandex and is working on new projects that will utilize its core functionalities, with plans for native Datadog integration. The interview concludes with an invitation for developers to explore Datadog's platform and participate in community events.

Brett Langdon, a seasoned software engineer and DevOps consultant, recently joined Datadog as an Open Source Software Engineer, bringing with him a rich history of contributions to the Datadog ecosystem, including maintaining the node-dogapi library. In a conversation with Daniel Maher from the Datadog Community team, Brett highlights the evolution of Datadog's open-source experience, noting the substantial growth and support within the community that makes it easier for new contributors to get involved. He discusses the challenges of maintaining backward compatibility while refactoring code and the importance of community feedback in improving Datadog's tools and APIs. Although he does not actively use the node-dogapi package he maintains, Brett ensures its alignment with the latest documentation and relies on community contributions for ongoing maintenance. His new role at Datadog reflects his passion for the company and his desire to re-engage with the open-source community, particularly in developing the Python APM, which was a significant factor in his decision to join the team.

In a detailed interview, Zach Daniel, an Elixir enthusiast and maintainer of the Spandex library for Datadog, discusses his journey with Elixir and its application in observability. Initially working at Albert.io, Zach transitioned the company's tech stack from Node.js and RethinkDB to Elixir with Phoenix and Ecto for handling large-scale statistics. This switch was driven by Elixir's superior database interaction capabilities. While Elixir's built-in monitoring tools initially sufficed, Zach realized the need for more advanced observability, leading to the development of Spandex, a Datadog Elixir APM library. He describes the challenges of creating a non-intrusive monitoring tool that doesn't hinder application performance, emphasizing the importance of ease of use and effective state management in tracing services. Despite initial difficulties due to limited documentation, Zach leveraged existing resources like Python and Ruby clients and the Datadog public Slack channel to develop the library. The project gained traction, leading to community involvement and further development. Zach continues to actively contribute to Spandex, enhancing its capabilities for better integration with Datadog.

Datadog has been actively supporting inclusiveness and diversity in the tech industry by hosting events with organizations such as She+ Geeks Out, Tech Ladies, and Lesbians Who Tech. These events have included panel discussions on topics like "The Only Experience" and "Making the Leap to Manager," a happy hour, and a holiday party. Datadog has also partnered with other groups like Out in Tech, Grace Hopper Celebration, and Write/Speak/Code for conferences and events aimed at promoting diversity in tech. The company is committed to fostering an inclusive work environment and encourages underrepresented groups to share their perspectives while connecting with others in the industry.

Datadog has been actively supporting inclusiveness and diversity in tech by partnering with organizations such as She+ Geeks Out, Tech Ladies, and Lesbians Who Tech to host panel discussions, happy hours, and holiday parties. These events provide opportunities for underrepresented groups to share their perspectives and connect with each other and the broader tech community. The company's recruiting team has also benefited from these partnerships, engaging with individuals from diverse backgrounds at conferences such as Out in Tech and the Grace Hopper Celebration. By fostering strong, diverse partnerships, Datadog aims to build a more inclusive work environment and continue growing its team of talented individuals.

In Elasticsearch, unassigned shards can indicate issues with data availability or cluster reliability. To address this issue, first verify the worth of the data in the unassigned shards. If it's valuable, consider solutions such as purposefully delayed shard allocation, too many shards and not enough nodes, re-enabling shard allocation, missing shard data, low disk watermark, or multiple Elasticsearch versions. Monitoring for unassigned shards is crucial to maintain optimal cluster health and performance.