March 2019 Summaries

The Datadog Summit, scheduled for April 16 in Seattle, aims to bring together engineers, developers, and other professionals from the community. Attendees will have the opportunity to learn about building cultures of observability, improving system performance and reliability, and getting hands-on experience with workshops on infrastructure monitoring, distributed tracing, log analytics, and creating Datadog integrations. The event is free but requires RSVP due to limited space.

The Datadog Summit will be held on April 16 in Seattle, where attendees can meet and learn from the community, as well as from Datadog staff. The event features sharing of knowledge from engineers and developers who have transformed their organizations by building cultures of observability, practical advice from community members, and hands-on workshops covering infrastructure monitoring, distributed tracing, and log analytics. These workshops include training sessions, demonstrations, and even a workshop where attendees can build their own Datadog integration. The summit is free but space is limited, so it's recommended to RSVP in advance to reserve a seat at the Loews Hotel near Pier 55.

Nils Bunge and Maxim Brown discuss the importance of logging in Java applications to gain visibility into performance and troubleshoot issues. They cover various Java logging frameworks, including JUL, Log4J, and Logback, as well as their respective facades like SLF4J. The authors explain how to create, configure, and format loggers, log levels, appenders, and layouts, providing examples for each. They also discuss the benefits of writing logs to a file, streaming logs to an endpoint, and logging to JSON, which can be used to add custom information like Mapped Diagnostic Context (MDC) and enrich logs with extra context and metadata. Additionally, they cover best practices for getting the most out of Java logs, including using centralized logging solutions and monitoring services like Datadog to collect, parse, and visualize logs alongside metrics and distributed request traces.

At Datadog, engineers developed an MTA Service Tracker dashboard to provide real-time insight into New York's subway system. Recently, a similar dashboard was created for the MBTA in Boston. The dashboard uses AWS Lambda function to pull real-time vehicle and station data from the MBTA's GTFS feed, which provides location information and expected arrival times at every station. The data is then visualized on a shareable screenboard showing the next scheduled trains for nearby stops and calculating average wait times and total number of vehicles across each line. Additionally, performance data about service-level indicators (SLIs) is provided by the MBTA, offering insight into severe service interruptions and recovery status. The Lambda function used to create this dashboard is publicly available on GitHub for others to use and customize based on their preferences.

Datadog engineers created an MTA Service Tracker dashboard to provide insight into New York's subway system, and a similar MBTA version was built for the local transit system in Boston. To access the data, they wrote an AWS Lambda function that pulls real-time vehicle and station information from the MBTA's GTFS feed into Datadog. The function pairs raw wait time data with route and vehicle metadata as meaningful tags, sends it to Datadog, and creates events based on MBTA service alerts. A dashboard was designed to visualize this data, including the next scheduled trains for six stops near the office, average wait times, and total vehicles across the system for each line. The MBTA provides performance data via a separate API, which can be used with Datadog to create custom SLI indicators. The Lambda function is publicly available on GitHub, allowing users to use it and create their own custom dashboards based on their interests.

HashiCorp Vault is a secrets management tool that securely stores and tightly controls access to sensitive data such as tokens, passwords, certificates, encryption keys, etc. Datadog's new integration allows users to monitor Vault and ensure its proper functioning along with the rest of the development architecture that relies on it. The integration includes alerts for unexpected sealing of Vault servers, leader changes in high-availability mode, and verification of Vault initialization. By integrating HashiCorp Vault with Datadog, users can monitor their secrets management system alongside other integrated services like Consul and Kubernetes in one place, providing visibility into Vault along with the rest of the technologies in their stack.

HashiCorp Vault is a secrets management tool that securely stores and controls access to sensitive data. It provides a centralized storage solution, robust auditing, and encryption services to safeguard against secret sprawl. With Datadog's integration, you can monitor Vault and receive alerts on unusual activity, ensuring your secret management service is running properly along with the rest of your development architecture. Vaults can exist in two states: sealed or unsealed, with sealed servers becoming operational again once unsealed. A standalone Vault server will become sealed if it restarts unintentionally, but you can set up alerts to notify you if a Vault intended to be unsealed has become sealed. Monitoring leader changes and startup issues is crucial to ensure the stability of your Vault cluster. Integrating HashiCorp Vault with Datadog provides visibility into Vault's status and monitoring of its performance along with other integrated services like Consul and Kubernetes.

Datadog has announced a new integration with Alibaba Cloud, now generally available, which enables monitoring of Alibaba Cloud services such as load balancers, managed databases, and more. The integration automatically collects key metrics and metadata from various Alibaba Cloud services, including ECS, SLB, ApsaraDB RDS, and CDN, and provides real-time visibility into these services in Datadog. Additionally, the Datadog Agent can be installed on Alibaba Cloud VMs to get even richer context around the collected metrics and tags, allowing for a unified view of dynamic infrastructure and applications. The integration also includes support for over 850 built-in integrations with Datadog APM, enabling tracing requests across distributed services and instances, as well as log collection from each VM for troubleshooting purposes.

Go's built-in logging library is suitable for local development and simple applications. logrus is recommended for structured logging in JSON format, which can be parsed by machines and easily added to with new fields. glog allows for logging level control through command line flags, reducing log volume in production. To write Golang logs effectively, make calls to the logger from within the main application process, standardize logs with predefined messages, send logs to a central platform, use HTTP headers and unique IDs to log user behavior across microservices, avoid declaring goroutines for logging, write logs to a file, implement a standard logging interface, and centralize Golang logs using solutions like syslog or Datadog.

IBM MQ is an enterprise-grade messaging middleware used by various industries for its stability and reliability. Datadog's new IBM MQ integration allows users to collect key metrics and logs from their instances, providing deeper visibility into the health and performance of their messaging infrastructure. The integration tracks important metrics such as queue depth, message age, and dead letter queue alerting. With Datadog, users can monitor and alert on important metrics and logs from their messaging infrastructure, along with over 650 other technologies and services.

IBM MQ is an enterprise-grade message-oriented middleware (MOM) that provides stability and reliability to companies in various industries, including financial services, retail, and aviation. The new IBM MQ integration with Datadog enables users to collect key metrics and logs from their IBM MQ instances, visualize them with customizable dashboards, and gain deeper visibility into the health and performance of their messaging infrastructure. Data integrity is a core principle of IBM MQ, guaranteeing no loss of messages, and the integration tracks dozens of key metrics, including queue depth, message age, and dead-letter queue alerts. With Datadog's system-level checks and powerful monitoring and alerting capabilities, users can monitor the overall health of their queues and queue managers, scale their instances accordingly, and take action immediately when issues arise.

Amazon DocumentDB is an AWS-managed document database service compatible with MongoDB 3.6 API. It handles tasks such as database management and autoscaling, while also backing up data to S3. Datadog's Amazon DocumentDB integration provides visibility into key metrics and logs from your clusters, ensuring proper provisioning, meeting performance expectations, and managing costs. The service consists of EC2 instances and a cluster storage volume that replicates data across three availability zones. Monitoring resource metrics like CPU utilization, memory, network throughput, and I/O helps determine if the cluster is properly scaled and using the best-suited instance class. Audit logging in DocumentDB can be enabled for events such as authentication attempts, database creation or deletion, and collection creation or deletion. Datadog's integration allows automatic parsing of audit logs into a human-readable format, enabling search and filtering on important attributes.

Amazon DocumentDB is an AWS-managed document database service compatible with the MongoDB 3.6 API, allowing users to leverage automated database management and scaling by AWS. The service provides visibility into key metrics and logs from database clusters through integration with Datadog, enabling monitoring of cluster health, performance, and costs. A DocumentDB cluster consists of EC2 instances and a storage volume that replicates data across availability zones, ensuring high availability and security features such as instance running in Amazon VPC environments. The service offers various metrics for tracking resource utilization, volume usage, and throughput to optimize cluster scaling and cost management. Additionally, Datadog's integration provides audit logging capabilities to monitor cluster events and automate log sending from CloudWatch Logs to Datadog.

Datadog offers App Analytics for PHP to help developers monitor performance, identify bottlenecks and optimize their users' experience. The tool provides distributed tracing, detailed performance metrics, and auto-instrumented frameworks and libraries. It also includes a Service Map that visualizes the application's component services and request pathways between them. Datadog APM features such as high-level service overview, span summary, Watchdog, traces and flame graphs, correlating traces and logs, App Analytics for customer-level insight, and flexible dashboards can be used to optimize PHP performance.

Datadog offers App Analytics for PHP to help monitor performance, identify bottlenecks, and optimize user experience. The service provides distributed tracing, detailed performance metrics, and a Service Map to visualize application architecture and dependencies. It also includes auto-instrumented frameworks and libraries, open source tracing library, and support for various databases and operating systems. Datadog's APM features include Watchdog, traces, flame graphs, span summary, and correlating traces with logs. Additionally, App Analytics allows users to translate application insights into customer-level insights, creating flexible dashboards that combine app performance, infrastructure, and logs.

C# logging is crucial for optimizing application performance and debugging errors, but effective logging requires context and a readable format. This guide outlines best practices for C# logging, emphasizing the benefits of using JSON formatting for easier log analysis and recommending specialized frameworks like Apache log4net over built-in providers like TraceSource for enriched logging capabilities. It walks through the installation and configuration of logging libraries, the addition of metadata, and the standardization of logs with naming conventions. The guide also covers logging both handled and unhandled exceptions and suggests streaming logs to a centralized log management service such as Datadog for comprehensive monitoring and analysis. By following these practices, developers can gain deeper insights into application health and performance, leveraging logs to track trends and troubleshoot issues effectively.