April 2018 Summaries

Dash, a new conference focusing on building and scaling applications, infrastructure, and technical teams, is set to launch in New York City on July 11-12. The event will feature keynotes, breakout sessions, hands-on labs, trainings, and networking opportunities. With over 1,000 engineers expected to attend, Dash aims to bring together professionals from dev and ops backgrounds. The conference will be held at Spring Studios in Tribeca and will follow three themed tracks: Performance, Scalability, and Teams. Notable speakers from companies such as Google, Shopify, and Airbnb are set to share their insights on developing scalable, resilient, high-performance applications and teams. Registration is now open with early bird pricing available.

As a Software Engineering Intern at Datadog, the author contributed to open source projects using Application Performance Monitoring (APM) tools to identify performance bottlenecks and compare different implementations. They initially worked on optimizing Homebrew, a package management system for macOS extensively used by Datadog's engineering teams. The author identified the slowest part of the 'brew linkage' command and implemented caching using SQLite3, which significantly improved performance. They also tried pstore and DBM as alternative caching mechanisms but found SQLite3 to be faster. Their optimizations met the requirements specification by processing over 50 packages in under 5 seconds. The author recommended proceeding with the DBM implementation due to its benefits such as not requiring a third-party gem, cleaner code structure, and Ruby's native Hash get/set conventions.

Datadog provides a comprehensive guide for integrating and monitoring Redis using its platform, emphasizing a shift from traditional "master-slave" terminology to "primary-replica." The article details the steps to verify Redis functionality, install and configure the Datadog Agent, and ensure proper data reporting. It highlights the creation of custom dashboards to visualize Redis metrics alongside other infrastructure data, using features like Application Performance Monitoring (APM) for detailed request tracing and logging. The guide outlines how to enable log collection, customize configurations, and set up alerts for proactive system monitoring. By following these instructions, users can achieve enhanced visibility and automated alerting for their Redis environments, with the option to start with a free Datadog trial if not already subscribed.

The author of the text is Andrew Robert McBurney, a software engineering intern at Datadog. He contributed to the open source project Homebrew, which is a popular package management system for macOS. To improve the performance of Homebrew, Andrew used Application Performance Monitoring (APM) and instrumented its code using Datadog's ddtrace gem. He identified the bottleneck in the `check_dylibs` function and implemented caching mechanisms using SQLite3 and DBM to speed up the processing of package links. After comparing the performance of the two implementations, Andrew recommended using the DBM implementation due to its faster build time, cleaner code, and ability to fetch data from the cache quickly. The project's requirements were met, with the command processing more than 50 packages in under 5 seconds.

The new Datadog Agent is written in Go, with Python used for checks. This is possible because the Datadog Agent embeds a CPython interpreter that can be called whenever it needs to execute Python code. Embedding Python in a Go application has several benefits, such as gradually moving portions of an existing Python project to a new language without losing any functionality during the process and reusing existing Python software or libraries without re-implementing them in the new language. The Global Interpreter Lock (GIL) poses a challenge when it comes to embedding Python, but solutions like explicitly invoking the GIL and using the LockOSThread function from the runtime package can help manage this issue.

The text discusses the challenges and benefits of embedding Python in a Go application, specifically highlighting the use of cgo as a Foreign Function Interface (FFI) to invoke CPython functions. The Datadog Agent is mentioned as an example project that uses this approach to embed Python for executing custom checks without recompiling the agent. The text covers various tradeoffs and limitations, including the overhead introduced by cgo, manual handling of the Global Interpreter Lock (GIL), and limitations on goroutine scheduling. However, it also provides solutions and workarounds to mitigate these issues, such as using go-python for a thin wrapper around the C API and forcing goroutines to run on the same thread using the LockOSThread function from the runtime package. The text concludes that embedding Python in Go is worth the tradeoffs for the convenience of running Python checks in Go.

Amazon RDS provides a managed service that enables you to set up, operate, and scale PostgreSQL deployments in the cloud. Monitoring your Amazon RDS PostgreSQL instances is crucial for ensuring optimal performance and availability of your databases. In this series, we’ll cover an overview of key metrics to monitor when running PostgreSQL on AWS RDS, including replication delay, resource utilization (CPU, memory, disk I/O, network), connections, and more. In the first part of this series, we discussed how to monitor replication delay in Amazon RDS PostgreSQL instances. In this second part, we’ll cover how to track key system-level metrics like CPU, disk, memory, and network usage, as well as other important metrics related to storage, connections, and more. Resource utilization Monitoring key system-level metrics like CPU, disk, memory, and network can help you investigate PostgreSQL performance issues and ensure that the database has enough resources to complete its work. RDS also provides the option to enable enhanced monitoring on your instances (excluding the db.m1.small instance class), which can help you gain more visibility into resource usage. Although CloudWatch also provides basic system-level metrics for RDS instances, enhanced metrics are collected by an agent that runs directly on the instance, rather than via the hypervisor, which results in higher-granularity data (collected as frequently as once a second). Disk I/O metrics |Metric description |CloudWatch name |Enhanced monitoring name |Metric type |Read I/O operations per second |ReadIOPS |diskIO.readIOsPS |Resource: Utilization |Write I/O operations per second |WriteIOPS |diskIO.writeIOsPS |Resource: Utilization |I/O operations waiting for disk access |DiskQueueDepth |diskIO.avgQueueLen |Resource: Saturation |Average amount of time per read I/O operation (ms) |ReadLatency |- |Resource: Other |Average amount of time per write I/O operation (ms) |WriteLatency |- |Resource: Other RDS PostgreSQL users can select from three types of storage (provided through Amazon Elastic Block Store) that cater to varying levels of desired performance: SSD, provisioned IOPS, and magnetic. Provisioned IOPS is the highest-performance option, and delivers speeds of up to 40,000 I/O operations per second. RDS provides CloudWatch metrics, ReadIOPS and WriteIOPS, that correspond to the average number of read and write I/O operations completed per second over each 1-minute interval. However, DiskQueueDepth, which tracks the number of I/O requests waiting in the queue, can often be more informative. If you see a consistently high value for this metric, you may need to convert to another storage type, or scale your storage to help ease the workload. Adding read replica instances may also help reduce I/O pressure on read-heavy database workloads. Memory metrics |Metric description |CloudWatch name |Enhanced monitoring name |Metric type |Available RAM (bytes) |FreeableMemory |memory.free |Resource: Utilization |Swap usage (bytes) |SwapUsage |swap.cached |Resource: Saturation When PostgreSQL reads or updates data, it checks for the block in the shared buffer cache first, and also in the OS cache, to see if it can serve the request without having to read from disk. If the block is not cached, it will need to access the data from disk. However, it will also cache it in memory so that the next time that data is queried, it won’t need to access the disk. PostgreSQL query performance relies heavily on caching data in the in-memory shared buffer cache, so AWS recommends providing your database instances with enough memory to store all of your most commonly accessed data. Monitoring FreeableMemory and SwapUsage can help ensure that your database has enough RAM to serve queries from memory rather than disk. AWS recommends tracking ReadIOPS to determine if your data is stored mostly in memory—ideally, it should be a low, steady value. If you suspect that your instance needs more memory, you can try scaling up its RAM and observing the ensuing effect on ReadIOPS. If it drops drastically, this indicates that your data was previously being accessed mostly from disk rather than memory, and that you may need to continue allocating more RAM to your instance to optimize query performance. The shared_buffers parameter determines how much memory the database can use for the shared buffer cache. In PostgreSQL, this value is usually about 128 MB, but in RDS, the default value of shared_buffers is calculated as a proportion of your database instance’s available memory using the following formula: DBInstanceClassMemory / 32768. Note that RDS requires you to specify shared_buffers in terms of 8-KB units, while the DBInstanceClassMemory parameter variable is provided in bytes, not kilobytes. Storage metrics |Metric description |Name |Metric type |Availability |Available storage space (bytes) |FreeStorageSpace |Resource: Utilization |CloudWatch |Disk space used by each table (excluding indexes) |pg_table_size |Resource: Utilization |PostgreSQL: Database object management functions |Disk space used by indexes in the table |pg_indexes_size |Resource: Utilization |PostgreSQL: Database object management functions |Number of index scans initiated on this table or index |idx_scan |Resource: Utilization |PostgreSQL: pg_stat_user_tables or pg_stat_user_indexes When you create an RDS PostgreSQL database instance, you must provide it with a certain amount of storage (usually in the form of EBS volumes). Monitoring FreeStorageSpace can help you determine if you’re running out of space, meaning that you either need to scale up your storage or delete unused or outdated data/logs. Note that you can increase, but not decrease, the amount of storage allocated to an RDS instance. In addition to tracking available storage space on your database instances, you can also track how much storage space is being utilized by various tables and indexes in your database. PostgreSQL collects statistics internally to help you track the size of tables and indexes over time, which is helpful for gauging future changes in query performance. As your tables and indexes grow in size, queries will take longer, and indexes will require more disk space—so eventually, you will either need to scale up the instance’s storage, partition your data, or rethink your indexing strategy. If you see any unexpected growth in table or index size, it may also point to problems with VACUUMs not running properly, so you should also inspect VACUUM-related metrics to see if they provide other insights. In the next part of this series, we’ll show you how to query pg_stat_user_indexes to see if there are any underutilized indexes that you could remove in order to free up storage space and decrease unnecessary load on the database. Indexes can be increasingly difficult to maintain as they grow in size, so it may not be worth applying resources to data that isn’t queried very often. Network and CPU metrics |Metric description |CloudWatch name |Enhanced monitoring name |Metric type |Network traffic to RDS PostgreSQL instance |NetworkReceive Throughput (bytes/sec) |network.rx (packets) |Resource: Utilization |Network traffic from RDS PostgreSQL instance |NetworkTransmit Throughput (bytes/sec) |network.tx (packets) |Resource: Utilization |CPU utilization (percent) |CPUUtilization |cpuUtilization.total |Resource: Utilization Keeping an eye on the CloudWatch metrics NetworkReceiveThroughput and NetworkTransmitThroughput will help you determine if your instances have enough network bandwidth to serve queries and replicate updates to standby and/or replica instances. This is particularly important if your replica instances are located in a different region. High CPU utilization is usually not a cause for concern. However, If you notice that CPU has increased significantly without any obvious reason, you can try querying pg_stat_activity to see if long-running queries may be the source of the issue. Resource utilization metrics to alert on FreeStorageSpace: This is a critical metric to monitor on your database instances. If you run out of storage, you will not be able to connect to the database instance. As such, AWS recommends setting up an alert to get notified when this metric reaches 85 percent or higher. This will give you enough time to take action by deleting outdated data/logs, removing unused indexes or tables, or adding more storage to the instance. DiskQueueDepth: High-traffic databases can expect to see queued I/O operations. However, if you see this metric increasing along with any noticeable spikes in read or write latency, you may need to upgrade your storage type to keep up with demand. ReadLatencyand WriteLatency: These two metrics help you track the latency of I/O read and write operations, and can help you determine if your allocated storage is able to handle the database workload. If latency continues to degrade, you can consult the RDS documentation to see you could improve performance by upgrading to a higher-performance storage option like provisioned IOPS, which enables RDS instances to process more I/O requests concurrently. Connections |Metric description |Name |Metric type |Availability |Number of open connections |DatabaseConnections |Resource: Utilization |CloudWatch |Number of open connections |numbackends |Resource: Utilization |PostgreSQL (pg_stat_database) |Percentage of max connections in use |numbackends as percentage of max_connections in pg_settings |Resource: Utilization |PostgreSQL (pg_stat_database, pg_settings) |Client connections waiting on a server connection (PgBouncer) |cl_waiting |Resource: Saturation |PgBouncer |Max time a client connection has been waiting to be served (PgBouncer) |maxwait |Resource: Saturation |PgBouncer The PostgreSQL primary server process forks a new process every time a client requests a connection. PostgreSQL sets a max_connections limit, which determines the maximum number of connections that can be opened to the server at any one time. By default, RDS will set this parameter in proportion to your database instance class’s available memory. The formula varies according to the version of PostgreSQL you’re running. RDS also reserves up to three of these connections for system maintenance. If you see the number of open connections consistently approaching the number of maximum connections, this can indicate that applications are issuing long-running queries, and constantly creating new connections to send other requests, instead of reusing existing connections. Using a connection pool can help ensure that connections are consistently reused whenever they go idle, instead of requiring the primary/source instance to frequently open and close connections. In high-concurrency environments, using a connection pool like PgBouncer can help distribute requests made to your primary instance. The pool serves as a proxy between your applications and RDS PostgreSQL instances. In versions 9.6+, you can also set an idle_in_transaction_session_timeout, which instructs PostgreSQL to close any connections that remain idle for longer than this period of time. By default, this value is 0, which means that it is disabled. Next steps in Amazon RDS PostgreSQL monitoring In this post, we’ve covered an overview of the key metrics to monitor when running PostgreSQL on RDS. As you scale your Amazon RDS PostgreSQL deployment over time, keeping an eye on these metrics will help you detect and troubleshoot potential issues and keep database operations running smoothly and efficiently. Read the next part of this series to learn how to collect all of these metrics from AWS CloudWatch and from PostgreSQL itself. Source Markdown for this post is available on GitHub. Questions, corrections, additions, etc.? Please let us know.``` SUMMARY:

To effectively monitor the performance of PostgreSQL on Amazon RDS, it is essential to collect metrics from both Amazon CloudWatch and directly from each PostgreSQL database instance. This post provides detailed instructions on accessing RDS metrics via AWS Management Console, AWS CLI, or CloudWatch-compatible tools, and demonstrates how to visualize these metrics and set alerts. It also explains how to connect to an RDS PostgreSQL instance to query native metrics using PostgreSQL’s statistics views, such as pg_stat_database and pg_stat_user_tables, which are not available on CloudWatch. The post highlights the importance of regularly collecting these metrics to track changes over time, offering insights into replication delay, connection metrics, locks, and disk usage. Additionally, it introduces the next part of the series, which will guide readers on using Datadog for automated metric collection and performance analysis across multiple technologies.

In this guide, we learn how to set up Datadog to automatically collect key metrics from Amazon RDS PostgreSQL instances and provide more meaningful insights into their performance over time. We cover two main steps: accessing CloudWatch metrics with Datadog's Amazon RDS integration and querying metrics directly from the database instance using the PostgreSQL integration. Additionally, we explore how to enhance visibility by setting up RDS enhanced metrics with Datadog and deploying distributed tracing and APM to monitor applications that rely on RDS PostgreSQL.

Datadog provides a comprehensive PostgreSQL RDS monitoring solution that integrates with Amazon RDS and allows users to collect key metrics, query metrics directly from the database, and gain visibility into application-level performance. The integration also enables users to set up distributed tracing and APM to monitor requests to their application and forward to Datadog, providing detailed insights into system performance. With this solution, users can create custom dashboards that combine key metrics from all sources, set up alerts for potential issues, and leverage machine learning-powered features like forecasting and anomaly detection to identify potential problems before they impact users.

Amazon's built-in CloudWatch monitoring service can be used to collect Amazon EBS metrics and status checks. This can be done through the CloudWatch or EC2 web consoles, AWS command line tool, or a program or third-party monitoring service that connects to the CloudWatch API. To monitor CloudWatch metrics, users need access to a user account or role whose security policy grants minimal permissions needed to manage the CloudWatch and EBS APIs. CloudWatch collects metrics through a hypervisor from any AWS services used in an infrastructure. It provides two complementary ways to aggregate data: periods and statistics. The period sets the timespan, in seconds, over which CloudWatch will aggregate a metric into data points. Statistics are different ways to aggregate the data over the collection period. There are three primary ways to view EBS metrics from CloudWatch: via the AWS web console, via the AWS CLI, and via a third-party tool that integrates with the CloudWatch API.

Amazon Elastic Block Storage (EBS) is a cloud-based storage service that provides block-level storage volumes for use with Amazon EC2 instances. Unlike instance store volumes, EBS volumes maintain their state when stopped or detached from an instance and can be connected to multiple instances within the same availability zone. EBS volumes provide redundancy and data backup through snapshots, which are stored in S3 buckets and can be transferred across AWS regions. EBS-optimized instances offer dedicated bandwidth for EBS volumes, separate from other network activity, with varying levels of performance depending on the instance type. Understanding the network configuration of your EC2 instances and attached volumes is crucial to ensure optimal disk performance. EBS volumes come in different types, including SSD (gp2 and io1) and HDD (st1 and sc1), each with distinct characteristics suited for specific workloads. Mixing and matching volume types can open up various use cases, such as running a high-transaction database on gp2 volumes while storing logs on sc1 volumes. Monitoring EBS volumes is essential to ensure application health and optimal configuration. Key metrics to track include IOPS, throughput, block size, latency, disk activity, status checks, and scheduled events. Amazon CloudWatch provides a convenient way to view these metrics for your EBS volumes and other AWS services.

Datadog's AWS integration allows users to connect CloudWatch with Datadog in order to automatically collect metrics from various AWS services, including EBS. With over 650 integrations, Datadog enables the correlation of AWS metrics with those from other services within an environment. The data is accessible for 15 months at full granularity. To enhance visibility into Amazon EBS volumes and overall infrastructure, users can install the Datadog Agent on their instances. This allows for the collection of system-level metrics from volumes at a 15-second resolution. Additionally, with Datadog APM and logging, the installation of the Datadog Agent provides a fully unified monitoring platform. There are two ways to start using Datadog to monitor EBS volumes: enabling the AWS integration or installing the Datadog Agent on EC2 instances. The former approach allows for immediate pulling of all metrics outlined in the first part of this series, while the latter provides detailed monitoring of applications and infrastructure. Both methods can be used in a complementary fashion. Enabling the AWS integration requires correctly delegating AWS IAM roles and granting the Datadog role read-only access. Once set up, EBS metrics (along with EC2 and other AWS services) will flow into Datadog for visualization and monitoring on custom dashboards. Installing the Datadog Agent enables users to monitor applications and infrastructure in greater detail and depth. The Agent is installed on the root volume of an instance and can collect and forward metrics, logs, and request traces from instances. It automatically reports system-level metrics for instances and any EBS volumes mounted to them. Datadog's Live Container view provides complete coverage of container fleets with metrics reported at two-second resolution, while Live Process monitoring offers the same level of visibility into all processes running across a distributed architecture. Tags enable users to easily slice their hosts and drill down into particular problem areas in their infrastructure. Advanced alerting features like anomaly and outlier detection, forecasting, and tag-based alerting allow for effective identification of issues in dynamic environments. Datadog alerts help users stay ahead of future problems in their infrastructure and applications by notifying them of unexpected instance behavior or predicting when a certain threshold may be crossed. Users can sign up for a free 14-day trial to start monitoring their cloud infrastructure, applications, and services with Datadog.

Amazon EBS (Elastic Block Storage) is a persistent block-level storage as a service that works in conjunction with EC2 instances. It provides redundancy and data backup capabilities through snapshotting, allowing for easy creation of duplicate datasets or migration of application infrastructure to multiple locations. EBS volumes are virtualized, networked drives that share infrastructure with other users, so disk performance is tightly linked to network performance and throughput. Many EC2 instance types are EBS-optimized, providing dedicated bandwidth to EBS volumes separate from the instance's other network activity. EBS volumes come in different flavors, including solid-state drives (SSD) and hard disk drives (HDD), each with unique characteristics that affect their performance and usage charges. It is essential to monitor EBS volume metrics, such as IOPS, throughput, latency, disk activity, status checks, events, and disk space metrics, to ensure optimal performance, optimize resource utilization, and troubleshoot issues with applications running on these volumes.

Datadog's AWS integration allows users to automatically collect metrics from multiple AWS services, including EBS, and correlate them with data from other services in their environment. Users can also install the Datadog Agent on their instances to gather system-level metrics from volumes, including disk usage, at 15-second resolution. This enables a fully unified monitoring platform for applications and infrastructure. The integration allows users to create customized dashboards that meet specific monitoring needs, visualize key metrics, and set up alerts for potential issues such as increased disk read operations or unexpected instance behavior. Additionally, the Datadog Agent provides benefits like higher resolution metrics, device naming consistency, and advanced alerting features like anomaly detection and forecasting. Users can start monitoring their EBS volumes with Datadog by enabling the AWS integration or installing the Agent on their instances.

CloudWatch monitoring service provides a simple way to begin collecting Amazon EBS metrics and status checks. The service can be accessed through the CloudWatch console, the AWS CLI, or via a third-party tool that integrates with the CloudWatch API. Additionally, an agent can be installed on EC2 instances to collect system-level information such as disk utilization metrics, providing more visibility into infrastructure components like EBS volumes. While CloudWatch provides general overview of volume fleet, it may not collect all EBS metrics, and custom metrics incur additional charges.

Vacuuming is an essential aspect of maintaining a healthy PostgreSQL database. It helps optimize database performance and resource usage by marking dead rows as available to store new data, updating the visibility map, and preventing transaction ID wraparound failure. Monitoring key metrics such as dead rows, table disk usage, and last time vacuum processes ran can help ensure that VACUUMs are running smoothly across your databases. Common issues related to VACUUM-related problems include autovacuum being disabled or not configured properly, lock conflicts, and long-running open transactions. Monitoring PostgreSQL VACUUMs and other metrics can provide a comprehensive view of database health and performance.

PostgreSQL vacuuming is essential for maintaining the health and efficiency of your database. Vacuuming helps optimize performance and resource usage by marking dead rows as available to store new data, updating a visibility map, preventing transaction ID wraparound failure, and more. Monitoring key metrics and events, such as `pg_stat_user_tables`, can help ensure that vacuum processes are running smoothly across your databases. Investigating common issues, including autovacuuming problems, lock conflicts, and long-running open transactions, is crucial to resolving vacuuming-related issues. Correlating vacuuming activity with performance indicators and infrastructure metrics using tools like Datadog can provide deep visibility into database health and overall application performance.