October 2020 Summaries

In this podcast episode, Scott McCarty from Red Hat discusses the evolution of automation and how it has led to the development of Kubernetes and OpenShift. He explains that while automation was already known in the early 2000s, there was no standard way to do it across different companies. With the advent of Kubernetes and OpenShift, developers can now build applications once and deploy them anywhere without having to worry about the underlying infrastructure. Scott also talks about how Red Hat Marketplace is a culmination of 20 years of work in automating software delivery supply chains. It allows companies to buy and sell pre-built applications that are already containerized and can be easily deployed on any Kubernetes platform. This has led to the creation of an ecosystem where developers can focus on building their applications while leaving the infrastructure management to others. Finally, Scott shares his belief that we live in a multi-cloud world, with most companies already using multiple cloud platforms without realizing it. He believes that OpenShift is well-suited for managing applications across different clouds and can help organizations get a better handle on their costs and usage.

CockroachDB is a distributed SQL database that offers users the ability to optimize for low latency or resilience. Data topology patterns are crucial in multi-region deployments, as they map the shape of a cluster and determine how data is replicated and geographically distributed across nodes. There are three common topology patterns: Follow the Workload (default pattern), Geo-Partitioned Replicas, and Geo-Partitioned Leaseholders. Each pattern has different survivability capabilities and can be chosen based on specific latency and resilience requirements.

The GigaOm Radar Report for Cloud Databases analyzes how enterprise companies are approaching the DBaaS market, and provides a detailed breakdown of top players. Key criteria used by enterprises to compare DBaaS platforms include support for relational model, multi-model support, and managed services automation offerings. The report offers in-depth analyses of various vendors such as Amazon Aurora, Cockroach Cloud, Couchbase, Google Cloud Spanner, Microsoft Azure CosmosDB, and MongoDB Atlas. GigaOm's findings indicate that cloud-native database platforms are the natural evolution of on-premises workhorse databases, with their inherent architectural and operational advantages attracting vast interest from enterprises.

CockroachDB is a distributed SQL database designed with an architecture that converts SQL statements into key-value data. The bottom layer of this architecture is a distributed, replicated, transactional key-value store. This enables efficient distribution of data within the database and supports tables and indexes. The key-value store facilitates dynamic schema change and allows CockroachDB to retain the efficiency of a KV store while still speaking SQL. Under the hood, it stores tabular data in a monolithic sorted map of KV pairs, where every table has a primary key, there is one key/value pair per column, and keys and values are all strings. The architecture also supports efficient range scans and automated sharding through range splitting.

Cockroach Labs recently introduced a new algorithmic approach to organizing files in their storage system, Pebble, which led to an 80+% reduction in ingestion time for the standard TPC-C benchmark dataset. The team initially replaced the implementation of their IMPORT bulk-loading feature with a simpler and faster data ingestion pipeline but later faced issues where some IMPORTs were much slower or even stuck. They discovered that directly sending out-of-order data to the KV storage layer was causing these problems, as LSMs like RocksDB store data in order. The solution hinged upon their recent switch from RocksDB as their key-value store to Pebble, where they were able to add a new algorithmic approach to organizing files that led to the massive improvement.

MyMahi, a New Zealand-based digital education company, built their student platform using CockroachDB Core embedded in a technology stack that includes spot instances (e.g., AWS Fargate), serverless functions (e.g., AWS Lambda), and GraphQL (e.g., GraphQL.js, GraphQL Tools). They chose the open source version of CockroachDB as their database because it met their requirements for scalability, ACID transactions, and ease of use. MyMahi's application architecture includes components such as Route53 DNS resolution, Cloudfront CDN, API Gateway, Lambda, SQS queue, S3 bucket, and a CockroachDB cluster. They use AWS Fargate spot instances to lower costs for their database cluster and handle bursts of traffic using AWS Lambda with Node.js. MyMahi also uses GraphQL to fetch exact data from CockroachDB and has abstracted transaction retry logic into a single function.

Geo-partitioning is a technique used by CockroachDB to minimize latency in distributed databases. It allows control of data location at the row level, enabling users to keep customer data close to them and reducing the distance it needs to travel, thereby improving user experience. By applying geo-partitioning rules, P99 latency can be reduced from 400ms to under well under 40ms, a significant improvement that meets the 100ms rule for instantaneous interactions. This technique is particularly useful when dealing with applications running across multiple regions and users worldwide.

Cockroach Labs is participating in Hacktoberfest, an annual event by DigitalOcean celebrating open-source contributions. The month-long event encourages developers of all skill levels to contribute positively to the community. Cockroach Labs has already seen improvements in their documentation and appreciates the support from this community. Open source initiatives often build active communities that foster lifelong friendships, but they can also be taken advantage of by bad actors. To keep open-source alive and well, it's crucial to evolve how we think about contributions, licensing, and community participation rules. Cockroach Labs invites everyone to contribute to their projects during Hacktoberfest, offering various resources and support for new contributors.

This tutorial by Cockroach Labs enterprise architect Fabio Ghirardello demonstrates how to simulate a multi-region CockroachDB cluster on localhost using Docker. The instructions assume you are running Linux or macOS, and have Docker installed. The process involves creating the required networks, configuring the HAProxy in each region, creating Docker containers for nodes in each region, initializing the cluster, attaching nodes to inter-regional networks, and finally, setting up a license and configuration for your Cockroach Cluster. Once completed, you can view the CockroachDB Admin UI at localhost:8080 and start your development work on a simulated multi-region deployment.