December 2017 Summaries

Google Kubernetes Engine (GKE) is a managed containerized application service on Google Cloud Platform, allowing cluster operators to focus on running applications without managing the Kubernetes control plane. GKE comes in two modes: Standard and Autopilot, with the latter providing automated node management. To effectively monitor GKE clusters, it's essential to collect metrics and performance data from across the cluster, including CPU and memory usage, container and pod events, network throughput, and individual request traces. Datadog is a key tool for monitoring GKE, providing components such as the Datadog Agent and Cluster Agent that can be deployed to monitor the cluster. The Datadog Agent collects metrics and logs from pods in the cluster, while the Cluster Agent acts as a proxy for node-based Agents and provides additional features like Kubernetes Admission Controller. To deploy the Datadog Agent and Cluster Agent, prerequisites include enabling the Datadog GCP integration and collecting GKE control plane metrics. Once deployed, the Datadog Agent provides access to various dashboards and visualizations, including the GKE Standard and Enhanced dashboards, as well as the Kubernetes Overview Page and pods view. These tools offer features like Watchdog Insights and flame graphs to help troubleshoot issues in the cluster. Additionally, Datadog provides application-level metrics through its APM feature, allowing users to correlate performance data across infrastructure components and gain deeper insights into their system's health and performance.

PostgreSQL is a powerful open-source relational database management system (RDBMS) that enables you to store and manage large volumes of data in tables with rows and columns. In order to ensure that your applications can consistently access the data they need, it’s crucial to monitor key performance metrics for your PostgreSQL databases. In this series, we will cover an overview of PostgreSQL monitoring and its key performance metrics. We will also explain how you can collect these metrics from your PostgreSQL instances using a combination of built-in statistics collector functions and external tools like Prometheus and Grafana. This first part of the series focuses on explaining what each metric means, why it’s important to monitor them, and which specific queries or configuration settings you should use to collect these metrics from your PostgreSQL databases. We will also provide some examples of how you can interpret these metrics in order to detect potential issues with your database performance. In the next part of this series, we will explain how to configure Prometheus and Grafana to collect and visualize these key performance metrics for your PostgreSQL instances. In Part 3, we will show you how to use distributed tracing tools like Jaeger or Zipkin in conjunction with your PostgreSQL monitoring data to help pinpoint the root cause of any potential issues that arise. By the end of this series, you should have a solid understanding of what each key performance metric means for PostgreSQL, and how you can use them together to effectively monitor and troubleshoot your databases in real time. Let’s get started! ## Part 1: Key Performance Metrics for PostgreSQL Monitoring In this section, we will cover an overview of the key performance metrics that are most important to monitor when it comes to ensuring optimal performance for your PostgreSQL databases. We will explain what each metric means, why it’s important to monitor them, and which specific queries or configuration settings you should use to collect these metrics from your PostgreSQL databases. We have organized this section into three main categories: query throughput & performance metrics, concurrent operations performance metrics, and replication & reliability metrics. We will also provide some examples of how you can interpret these metrics in order to detect potential issues with your database performance. ### Query Throughput & Performance Metrics One of the most important aspects of PostgreSQL monitoring is keeping an eye on query throughput and performance metrics, which help give you an idea of what types of queries your database is serving. By tracking these metrics over time, you can identify trends or patterns that may indicate potential issues with your database performance. Here are some key performance metrics related to query throughput & performance: - Rows inserted, updated, deleted by queries (per database) - Tuples updated vs. heap-only tuples (HOT) updated - Total number of transactions executed (commits + rollbacks) We will explain each one in more detail below. #### Rows Inserted, Updated, Deleted by Queries (Per Database) PostgreSQL tracks several key statistics related to the number of rows that are inserted, updated, and deleted by queries on a per-database basis. These statistics can be accessed using the following SQL query: ```sql SELECT * FROM pg_stat_database; ``` This will return a table with one row for each database in your PostgreSQL instance, showing you the total number of rows that have been inserted (tup_inserted), updated (tup_updated), and deleted (tup_deleted) by queries on that database. Monitoring the number of rows inserted, updated, and deleted can help give you an idea of what types of write queries your database is serving. If you see a high rate of updated and deleted rows, you should also keep a close eye on the number of dead rows, since an increase in dead rows indicates a problem with VACUUM processes, which can slow down your queries. A sudden drop in throughput is concerning and could be due to issues like locks on tables and/or rows that need to be accessed in order to make updates. Monitoring write activity along with other database metrics like locks can help you pinpoint the potential source of the throughput issue. #### Tuples Updated vs. Heap-Only Tuples (HOT) Updated PostgreSQL will try to optimize updates when it is feasible to do so, through what’s known as a Heap-Only Tuple (HOT) update. A HOT update is possible when the transaction does not change any columns that are currently indexed (for example, if you created an index on the column age, but the update only affects the name column, which is not indexed). In comparison with normal updates, a HOT update introduces less I/O load on the database, since it can update the row without having to update its associated index. In general, you want to see more HOT updates over regular updates because they produce less load on the database. If you see a significantly higher number of updates than HOT updates, it may be due to frequent data updates in indexed columns. This issue will only continue to increase as your indexes grow in size and become more difficult to maintain. #### Total Number of Transactions Executed (Commits + Rollbacks) PostgreSQL tracks the total number of transactions executed on a per-database basis, which includes both commits and rollbacks. These statistics can be accessed using the following SQL query: ```sql SELECT * FROM pg_stat_database; ``` This will return a table with one row for each database in your PostgreSQL instance, showing you the total number of transactions executed (xact_commit + xact_rollback). Monitoring the total number of transactions executed can help give you an idea of how much write activity is occurring on your databases. A high rate of commits and/or rollbacks could indicate that there are issues with data consistency or integrity, which may require further investigation. ### Concurrent Operations Performance Metrics Another important aspect of PostgreSQL monitoring is keeping an eye on concurrent operations performance metrics, which help ensure that the database can scale sufficiently to be able to fulfill a high rate of queries. The VACUUM process is one of the most important maintenance tasks related to ensuring successful concurrent operations. Here are some key performance metrics related to concurrent operations: - Locks - Deadlocks (v. 9.2+) - Dead rows We will explain each one in more detail below. #### Locks PostgreSQL grants locks to certain transactions in order to ensure that data remains consistent across concurrent queries. You can also query the pg_locks view to see the active locks on the database, which objects have locks, and which processes are waiting to place locks on objects. Viewing the number of locks per table, categorized by lock mode, can help ensure that you are able to access data consistently. Some types of lock modes, such as ACCESS SHARE, are less restrictive than others, like ACCESS EXCLUSIVE (which conflicts with every other type of lock), so it can be helpful to focus on monitoring the more restrictive lock modes. A high rate of locks in your database indicates that active connections could be building up from long-running queries, which will result in queries timing out. #### Deadlocks A deadlock occurs when one or more transactions holds exclusive lock(s) on the same rows/tables that other transactions need in order to proceed. Let’s say that transaction A has a row-level lock on row 1, and transaction B has a row-level lock on row 2. Transaction A then tries to update row 2, while transaction B requests a lock on row 1 to update a column value. Each transaction is forced to wait for the other transaction to release its lock before it can proceed. In order for either transaction to complete, one of the transactions must be rolled back in order to release a lock on an object that the other transaction needs. PostgreSQL uses a deadlock_timeout setting to determine how long it should wait for a lock before checking if there is a deadlock. The default is one second, but it’s generally not advised to lower this, because checking for deadlocks uses up resources. The documentation advises that you should aim to avoid deadlocks by ensuring that your applications acquire locks in the same order all the time, to avoid conflicts. #### Dead Rows If you have a vacuuming schedule in place (either through autovacuum or some other means), the number of dead rows should not be steadily increasing over time—this indicates that something is interfering with your VACUUM process. VACUUM processes can get blocked if there is a lock on the table/row that needs to be vacuumed. If you suspect that a VACUUM is stuck, you will need to investigate to see what is causing this slowdown, as it can lead to slower queries and increase the amount of disk space that PostgreSQL uses. Therefore, it’s crucial to monitor the number of dead rows to ensure that your tables are being maintained with regular, periodic VACUUM processes. ### Replication & Reliability Metrics Many users set up PostgreSQL to replicate WAL changes from each primary server to one or more standby servers, in order to improve performance by directing queries to specific pools of read-only standbys. Replication also makes the database highly available—if the primary server experiences a failure, the database will always be prepared to failover to a standby. Here are some key performance metrics related to replication & reliability: - Number of checkpoints requested & scheduled - Buffers written by checkpoints as percentage of total buffers written - Replication delay (seconds)

PostgreSQL offers a built-in statistics collector to help users monitor database health and performance by querying predefined statistics views such as pg_stat_database, pg_stat_user_tables, and pg_stat_user_indexes, which aggregate data on database activity, table usage, and index operations. Users can configure the collector in the postgresql.conf file to track additional metrics like disk I/O latency and user-defined functions, although some metrics require querying system administration functions or other native sources. The statistics collector provides snapshots of database activity, but for real-time insights and historical analysis, dedicated monitoring tools like PgHero offer a more user-friendly interface. PgHero, an open-source tool, aggregates key database metrics into a visual dashboard, highlighting long-running queries, unused indexes, and disk usage. For a more comprehensive monitoring solution, users can integrate PostgreSQL metrics with platforms like Datadog to visualize and alert on potential issues in real-time, offering a holistic view of database performance in the context of the entire system infrastructure.

PostgreSQL is an open-source object-relational database system that ensures data integrity and reliability with features like Multi-Version Concurrency Control (MVCC) and write-ahead logging. It has a query planner/optimizer that determines the most efficient way to execute queries, accounting for factors such as index usage and internal statistics about the database. The database uses MVCC to ensure that concurrent transactions do not block each other, and it periodically runs checkpoint processes to flush dirty pages from memory to disk. PostgreSQL maintains data reliability by logging each transaction in the write-ahead log (WAL) on the primary and writing it to disk periodically. It also collects metrics about its own resource usage, including connections, shared buffer usage, and disk utilization. The database's performance can be monitored through various key metrics, such as read query throughput and performance, write query throughput and performance, replication and reliability, and resource utilization. These metrics are accessible through PostgreSQL's statistics collector and other native sources, and they provide insights into the health and availability of the database.

Out In Tech's Digital Corps initiative aims to help build websites for the underrepresented and persecuted in places where openly expressing one's true identity is punishable. Datadog, a company committed to diversity, has partnered with Out In Tech as part of their strategy. They have hosted two events: Demos & Drinks night on October 17 and Job Networking Mixer in Boston on October 18. These events aim to increase the visibility and growth of the underrepresented community within tech while also supporting Datadog's own diverse workforce. Employees who attended these events found them inspiring, educational, and a chance to meet others from similar backgrounds. Datadog is committed to continue supporting Out In Tech in their future endeavors.

Datadog has partnered with Out In Tech, a non-profit organization that helps LGBTQ communities build websites for their own communities. The partnership was initiated by Datadog's employees who are part of the LGBTQ community and wanted to support an underrepresented group in tech. Through this collaboration, Datadog has hosted two events: a Demos & Drinks night in NYC and a Job Networking Mixer in Boston. These events aimed to increase visibility and growth for the LGBTQ community within tech, providing opportunities for networking and recruiting efforts. Employees who participated in these events reported being inspired by the passion and diversity of attendees, and expressed gratitude for Datadog's investment in their personal development. The partnership is part of Datadog's diversity strategy, which aims to build a diverse workforce and affect change in the broader tech community.

Datadog has introduced forecasting algorithms that use machine learning to predict future values of metrics and alert users in advance about potential issues. Forecasts can account for seasonality and adapt to baseline shifts, allowing users to visualize expected trends and set up alerts accordingly. The feature is useful not only for tracking infrastructure and application metrics but also for forecasting critical business metrics. By combining historical trends with future insights, forecasts in Datadog's dashboards provide more visibility into the health and performance of services.

During the holiday season, Datadog Cares, the social impact group at Datadog, organized a company-wide food drive to support local communities, addressing challenges such as accommodating diverse nationalities and enabling participation from remote employees. The initiative, which was first launched last year, involved a friendly competition among teams from different offices in New York, Boston, Paris, and remote locations, with team captains facilitating donation logistics. To track progress, Datadog used its dashboards to display participation metrics, encouraging employees to contribute even minimally, which often led to larger donations. The drive achieved a 91 percent participation rate, highlighting employee engagement with Datadog Cares initiatives. Building on this success, the company plans to expand the effort this year, forming new teams and incorporating smaller offices, while maintaining the spirit of friendly competition and community support.

Datadog has introduced a Live Process view to enhance full-stack monitoring by enabling users to explore, inspect, and monitor every process across distributed infrastructures in a centralized manner. This new feature allows users to query and filter running processes using tags, providing detailed process-level system metrics at two-second intervals, which is crucial for understanding resource constraints and identifying problematic processes. The tool addresses the challenge of high cardinality in process monitoring by offering intelligent aggregation and filtering, allowing users to efficiently manage and navigate through massive amounts of process data. Additionally, the Live Process view integrates with the Live Container view to offer insights into process trees within containers, aiding in the management and tuning of containerized applications. This enhancement also supports inventory management by allowing users to run reports on process trees to understand software usage, manage licenses, and ensure version compatibility across environments.

Datadog has announced support for OpenTracing, an open standard for distributed tracing. The company's Go tracer now supports OpenTracing, with other tracing clients to follow soon. As part of its commitment to open source, Datadog is joining the Cloud Native Computing Foundation and the OpenTracing Specification Council. OpenTracing provides APIs in several languages that can be used to log request spans to a number of pluggable backends. By adopting the OpenTracing standard, customers can benefit from vendor-neutral instrumentation without concern for vendor lock-in or costly code changes in the future.

Datadog has announced support for OpenTracing, a vendor-neutral standard for distributed tracing. The company's Go tracer now supports OpenTracing, and other tracing clients will follow soon. Datadog is also joining the Cloud Native Computing Foundation and the OpenTracing Specification Council to contribute to the evolution of the standard. Adopting OpenTracing enables customers to tap into deep request-level visibility without vendor lock-in or costly code changes. Datadog's support for OpenTracing allows users to easily integrate with other tracing backends, while joining the CNCF and OpenTracing Specification Council ensures customer voices are heard as the standard evolves. With this announcement, Datadog invites users to start a free trial to explore full-stack monitoring with distributed tracing and APM built on open instrumentation.

The Last Pickle, an Apache Cassandra consultant firm, has released a new set of dashboards for Datadog users to monitor their Apache Cassandra clusters. These dashboards are divided into two categories: Overview and Themed. The Overview dashboard is designed to easily detect any unexpected behavior in the Cassandra cluster without attempting to troubleshoot at this stage. On the other hand, the Themed dashboards are meant for efficient troubleshooting, identifying bottlenecks, and fixing issues. These themed dashboards focus on specific aspects of Cassandra's internal processes such as Read path, Write path, and SSTable management. The metrics and aggregations used in these dashboards have been optimized based on years of experience diagnosing Cassandra issues in production environments.

The Datadog team, in collaboration with Alain Rodriguez and The Last Pickle, has released a new set of dashboards for monitoring Apache Cassandra. These dashboards aim to provide clear, detailed, out-of-the-box metrics for Cassandra users to detect any unexpected behavior and troubleshoot issues efficiently. The dashboards are categorized into two types: an overview dashboard that verifies the cluster's health and themed dashboards that target specific areas such as read path, write path, and SSTable management. The new dashboards feature a three-column layout with optimized aggregations and metrics for effective troubleshooting. They are now available to all Datadog users who have enabled the Cassandra integration in their account.