May 2020 Summaries

Thanos, a CNCF sandbox project, is used to create a highly available metrics system with unlimited storage capacity on top of existing Prometheus deployments. It allows querying multiple Prometheus instances at once and merges data for the same metric across multiple instances to produce a continuous stream of monitoring logs. Thanos can be used to monitor a YugabyteDB cluster that is running on Google Kubernetes Engine (GKE), ensuring the continuous availability of YugabyteDB metric data by aggregating data from multiple Prometheus instances into a single view. The setup requires installing YugabyteDB, Prometheus Operator, and Thanos using Helm 3, and configuring them to work together seamlessly.

YugabyteDB and CockroachDB are both distributed SQL databases that offer scalability, high availability, and geographic distribution. YugabyteDB was architected to support all the above features, including high performance in terms of providing massive throughput and extremely low latency, being cloud native, open source, and supporting a wide range of PostgreSQL-compatible features. In contrast, CockroachDB has rewritten its SQL layer in Go, which may result in less advanced feature set compared to YugabyteDB. The two databases were evaluated using the YCSB benchmark at scale, with YugabyteDB showing on average 3x higher throughput than CockroachDB and outperforming it in all YCSB workloads in terms of throughput achieved whether using hash or range sharding. Additionally, YugabyteDB was found to have better write/update latency and read latency compared to CockroachDB. The performance differences were observed due to the use of RocksDB by CockroachDB, uneven compaction effects on performance, and backpressure issues with CockroachDB's lack of throttle for write requests. Despite these differences, both databases are capable of handling large amounts of data and can be used in real-world enterprise scenarios.

This Apache Airflow DAG is designed to migrate data between PostgreSQL and YugabyteDB, leveraging the SQL API (YSQL) compatibility between the two databases. The DAG detects new records in PostgreSQL and migrates them to YugabyteDB, updating the product and order tables with rows greater than a specified maximum ID. The task is scheduled to run every minute starting on today's date, allowing for real-time data migration and tracking through the Airflow UI. By following this tutorial, users can deploy a functional Airflow workflow that integrates with PostgreSQL and YugabyteDB, enabling seamless data migration between these two databases.

Deploying YugabyteDB for Apache Airflow on Google Cloud involves deploying Airflow on a Google Compute Engine VM, configuring firewall rules to expose the Airflow web UI, installing Apache Airflow and setting up a GKE cluster for YugabyteDB. The Airflow backend is built on top of YugabyteDB, which provides high availability, support for multiple cloud deployment options, and high performance. The process includes building the Airflow database objects and restarting the Airflow scheduler and web server with the new YugabyteDB backend configuration. This setup enables Airflow to interact with a PostgreSQL-compatible database while avoiding single points of failure and providing improved data distribution and geo-redundancy capabilities.

Welcome Bill Cook as Yugabyte’s New CEO Bill Cook joins Yugabyte as its new CEO, bringing an exemplary record of building and scaling enterprise infrastructure companies from his previous roles at Pivotal Software and EMC's Greenplum division. He is excited to lead the company on its mission to make YugabyteDB the default cloud native database. The appointment comes after a year of significant momentum for the company, including major product updates, increased adoption, and community engagement. With Bill at the helm, the team expects to continue driving forward momentum and achieving success in their goals.

As CEO of Yugabyte, I am committed to building a talented team, accelerating the adoption of Yugabyte Platform in the enterprise, and remaining committed to open source software while collaborating with our community and supporting our customers' expansion of use cases for YugabyteDB. The company has built credibility among its developer community and enterprise customers, and we have successfully developed an enterprise-grade cloud native relational database that powers highly resilient applications at internet scale. I am excited to work with the team to meet our goals and build a strong company culture that supports top talent in the industry.

YugabyteDB is a PostgreSQL-compatible, distributed SQL database that supports many features similar to Oracle, including string concatenation, ROWNUM and ROWID pseudocolumns, IF EXISTS for DDL operations, outer joins using (+), UPDATE and DELETE with another table, MERGE (UPSERT) statements, TRUNCATE operations, hierarchical queries, partition handling, block structure, and packages. YugabyteDB is designed to handle automatic geo-distribution, continuous availability, and open source licensing while maintaining SQL feature depth, performance, and ACID compliance. Most of the features highlighted in Roland's post are supported by YugabyteDB, with a few exceptions that will be addressed in future releases.

The latest 451 Research report, Distributed SQL, An Enabler for Globally Consistent Transactional Workloads, explores how database systems are playing a key role in digital transformation initiatives. The report highlights that 70% of enterprises are currently undergoing digital transformation activities and 55% expect to gain operational efficiency from their data platforms as a result. It also notes that the relational data model, which requires SQL, is widely adopted for transactional operational workloads now and in two years' time, with 70% running on-premises and 30% in the cloud. The report suggests that multi-cloud and hybrid cloud will be a significant investment focus going forward, with companies looking to match their data platform systems with cloud-native infrastructure to achieve optimal operational efficiency.

Colocated tables are a feature in YugabyteDB that allows storing small datasets on the same tablet, reducing query fan-out and boosting performance. By default, all tables in a database are stored on the same tablet, but large tables or tables with high write throughput can be opted out of colocation. This results in improved performance for queries involving joins and subqueries across related tables. The feature is particularly useful for small datasets that require high availability or geo-distribution, as well as for applications with a large number of databases and inter-related tables. By colocating these datasets on the same tablet, YugabyteDB reduces the overhead on clusters and improves query performance. Overall, colocated tables provide a convenient way to optimize database performance without compromising scalability.

YugabyteDB is a high-performance, open-source distributed SQL database designed to support cloud-native environments. It's built on a scalable and fault-tolerant design inspired by Google Spanner, with a PostgreSQL wire-compatible SQL API. YugabyteDB is well-suited for applications requiring Kubernetes support, geo-distributed data, high performance, and availability, as well as the familiarity of SQL. Microsoft Azure Kubernetes Service (AKS) provides a highly available, secure, and fully managed Kubernetes service for developers to host their applications on containers in the cloud. AKS is a natural fit for YugabyteDB due to its design and features. To get started with YugabyteDB on AKS, developers can install the Azure CLI, create a resource group, create a Kubernetes cluster, install YugabyteDB using Helm chart, and deploy the Northwind sample database.

YugabyteDB's distributed SQL APIs, YSQL and YCQL, have different load balancer requirements due to their underlying client driver implementations, with YSQL relying on PostgreSQL drivers that are not cluster aware or shard aware, while YCQL relies on Cassandra Query Language's smart client drivers. YugabyteDB provides a Spark connector library for YSQL and a JDBC API for YSQL, as well as support for ordering rows by the IN list of values using the VALUES clause or WITH ORDINALITY syntax. To backup a single table in a YSQL database, use the ysql_dump tool with the --table argument, while to list all table names in a YugabyteDB database, query the information_schema.tables view. New documentation and blog posts have been published on various topics, including array data types and functionality, upcoming events, and getting started with YugabyteDB.

YugabyteDB has reached a significant milestone with the release of version 2.1.5, which includes over 40 new enhancements and fixes. This release builds upon the foundation inspired by Google Spanner and is designed to be an open-source, high-performance distributed SQL database. The update includes improvements to tablet splitting, colocated tables functionality, and snapshot backups, as well as support for TLS encryption and a share folder containing sample databases. Additionally, YugabyteDB has welcomed its 1000th member to the community Slack channel, marking a notable achievement in the growth of the community.

YugabyteDB is a full-featured, ACID-compliant distributed SQL database that not only scales writes horizontally and self-heals from failures but also performs distributed query execution through "code shipping" (aka pushdowns). It supports range sharding along with the ability to pre-split a range into multiple shards, but hash sharding is the default option for horizontal scaling. YugabyteDB offers online index rebuilds and schema changes in the context of the Yugabyte Cloud QL (YCQL) API today, while these features are a work in progress for the YSQL API. With YugabyteDB v2.1, YSQL outperforms CockroachDB v19.2 in the YCSB benchmark especially as data volume stored per node grows into TBs. The database is described as a monolithic SQL database on top of a distributed KV database rather than a "distributed SQL database" as claimed by Cockroach Labs. YugabyteDB supports pessimistic locking for pessimistic concurrency control and has made significant performance improvements in YCSB, sysbench, and TPC-C benchmarks.

YugabyteDB outperforms CockroachDB in a comparison of their performance using the YCSB benchmark, achieving 3x higher throughput and 4.5x lower latency on average. YugabyteDB's storage layer, DocDB, is engineered for high performance at scale, with smart load balancing across multiple disks, separating compaction queues to reduce read amplification, and enabling multiple RocksDB instances to work together efficiently on a single node. In contrast, CockroachDB's storage layer is simplistic, leading to poor performance in large-scale workloads. YugabyteDB's query layer reuses PostgreSQL's native query layer, retaining advanced RDBMS features and allowing application development velocity to remain uncompromised. The comparison highlights the importance of considering the design principles and architectural differences between distributed SQL databases when evaluating their performance for specific use cases.