November 2023 Summaries

In "The Engineer's Survival Guide," Derrick Miller recounts his experience during Superstorm Sandy while working at Fog Creek Software in Lower Manhattan. The company had just decommissioned its disaster recovery site to transition entirely to the cloud, leaving them vulnerable as the hurricane approached. Despite having backup generators, the flooding rendered them useless, but the data center survived due to its independent generator. A dramatic effort ensued, with staff manually transporting diesel fuel up 18 flights of stairs to keep the generator running. This incident highlighted the critical need for robust disaster recovery plans, as the real risk was not immediate revenue loss but long-term damage to reputation and customer trust. The experience underscored the importance of integrating availability and resilience into systems from the start, as unforeseen disasters can strike at any time.

A survey of 300 technical decision-makers revealed that around half of the companies in the US, UK, and Germany are already utilizing multi-cloud strategies, with some even running workloads across multiple clouds. However, the transition to a multi-cloud setup isn't universally beneficial and often isn't necessary for achieving high availability since single-cloud solutions can suffice for most applications. The complexity and cost associated with multi-cloud deployments, especially for relational databases, require a compelling use case to justify the transition. Companies often opt for multi-cloud for business reasons like regulatory compliance and avoiding vendor lock-in rather than purely technical benefits. The modernization journey from legacy databases to a cloud-native, multi-cloud environment is challenging and should proceed gradually, focusing on incremental benefits at each stage. CockroachDB is highlighted as a versatile tool that can support companies throughout their cloud journey, from on-premises to multi-cloud deployments, offering compatibility and migration tools to ease transitions.

Alessandro Salvatori, Principal Engineer at DoorDash, discussed the company's transition from Aurora Postgres to CockroachDB in response to critical database failures during peak order times. Initially, DoorDash's monolithic Python application relied on a single Aurora Postgres cluster, leading to performance bottlenecks, particularly with the primary node's write capacity. The rapidly growing user base and increased demand during the pandemic exacerbated these issues, culminating in a major outage on April 17, 2020. To address this, DoorDash began using CockroachDB, which allows all nodes to serve writes and facilitates horizontal scaling, thereby eliminating the write bottleneck. Salvatori detailed the measured approach to this transition, highlighting the development of a tool to extract tables from the main database for easier scaling and the eventual automated migration to CockroachDB, which aligned with DoorDash's scaling needs and performance goals.

At RoachFest 2023, Joe Rizzo and James Lupolt from Hard Rock Digital presented how their selection of CockroachDB as a data platform has enabled them to streamline operations by consolidating their data infrastructure while adhering to the complex regulatory landscape of the online gambling industry. Initially considering separate databases for each state to comply with diverse regulations, they opted for a unified CockroachDB system, significantly reducing their total cost of ownership. By deploying CockroachDB nodes in strategic AWS regions and utilizing Amazon outposts where necessary, they ensured compliance with data residency requirements. Their experience highlighted the importance of network considerations in distributed systems and the need for a centralized yet flexible approach to data management. CockroachDB allowed Hard Rock Digital to achieve faster market entry, lower costs, and enhanced user experience by avoiding service duplication across jurisdictions.

Enterprises are increasingly adopting multi-cloud strategies for reasons such as operational resilience, compliance, and avoiding vendor lock-in, as well as for some unexpected non-technical reasons. At RoachFest23, Fortinet's Director of Software Development, Louis Jia, shared FortiSASE's journey from a self-hosted CockroachDB to CockroachDB's managed services, highlighting the challenges and benefits of such a migration. Initially launched in two AWS regions, the self-hosted setup posed availability and management challenges, leading to the decision to switch to a fully managed service. This transition involved collaboration with Cockroach Labs' engineers and was completed smoothly, even before the introduction of dedicated migration tools. The managed service offered FortiSASE enhanced performance, simplified upgrades, seamless regional expansion, and improved database reliability, ultimately allowing the team to focus on their core security offerings without being bogged down by database management. This migration has led to zero downtime since the switch, providing Jia and his team with peace of mind and better operational efficiency.

The "State of Multi-cloud 2024" report explores the current landscape and future of multi-cloud adoption, highlighting insights from 300 architects and engineering directors across the US, UK, and Germany. The report identifies that while most public cloud providers are similar, subtle differences can create significant challenges for companies adopting multi-cloud strategies. Despite these hurdles, multi-cloud adoption is increasing, driven by business goals such as regulatory compliance and the desire to avoid vendor lock-in. Approximately half of the surveyed companies are already utilizing multi-cloud, with more than half running at least one workload across multiple clouds. The trend is expected to continue, especially with the EU's Digital Operational Resilience Act (DORA) influencing preparations. The report underscores the technical complexity of multi-cloud deployment but suggests that economic factors and regulatory needs will further drive its adoption.

CockroachDB Cloud has introduced a new Browser SQL Shell feature that allows users to quickly write and execute SQL queries directly from their browser, enhancing the ease of database management. This tool aims to minimize the friction typically associated with database interactions by eliminating the need for generating and storing credentials, thereby streamlining tasks like schema iteration and syntax testing. The SQL shell, which is built using Codemirror 6, offers syntax highlighting, autocomplete, and other functionalities reminiscent of Jupyter notebooks and advanced terminals, making it easier to perform and share database operations. Additionally, the SQL shell facilitates a more efficient workflow with features such as persistent queries, history navigation, and CSV export, while maintaining a seamless user experience consistent with CockroachDB's terminal interface. As the platform continues to evolve, feedback from users is encouraged to further refine the tool's capabilities.

CockroachDB, a distributed SQL database, offers a robust solution for performing online schema changes without the need for downtime, maintaining data consistency and application performance. Unlike traditional methods that require scheduled downtime, CockroachDB's schema change engine operates in the background, leveraging a consistent distributed schema cache that allows multiple schema versions to be used concurrently. This system, inspired by Google's F1 team, facilitates schema updates by using a staged process that sequentially grants and revokes capabilities, ensuring that changes such as adding or dropping indexes are executed smoothly across the entire cluster. By employing a lease mechanism for schema versions, CockroachDB accelerates the convergence to the latest schema version, allowing schema changes to be conducted during normal business hours, even during peak usage times. This approach not only eliminates the need for off-peak deployment but also aligns with modern operational practices, making it a popular choice for enterprises aiming for minimal disruption and maximum uptime.

City Storage Systems, a startup in the real-estate and foodtech space, has successfully implemented a scalable global infrastructure using CockroachDB and Kubernetes, as discussed by engineering manager Rasmus Bach Krabbe and software engineer Frederick Stenum Mogensen at RoachFest. By leveraging CockroachDB for their OLTP workloads, the company has minimized manual labor and reduced human error, despite initial challenges with manual cluster provisioning and scaling. To overcome these issues, they developed a custom Kubernetes operator, streamlining the deployment and management of resources, which significantly simplified operations such as cloud migrations. This approach has provided City Storage Systems with a highly available and consistent database solution capable of supporting multi-region and multi-cloud environments, ultimately aligning with their global ambitions.

CockroachDB is a highly scalable, distributed SQL database that enables global data distribution with resilience against outages, allowing data to be pinned to specific locations to optimize performance and comply with data residency regulations. The concept of node locality is crucial in CockroachDB, as it involves assigning key-value pairs to each node to describe its geographic location, aiding in the efficient management of multi-region capabilities. This system allows CockroachDB to optimize data access and improve performance by reducing network round trips through mechanisms like "follow-the-workload," which adaptively moves data closer to active usage areas. For Kubernetes deployments, CockroachDB utilizes a tool called locality-checker to manage locality by detecting region and availability zone information, which it writes to configuration files for use during node startup. Adjustments in Kubernetes manifests, such as adding init containers and specific volume mounts, help automate the deployment process, ensuring that CockroachDB nodes are aware of their geographic positioning to leverage multi-region SQL capabilities effectively.

The blog post outlines a method for companies using Azure to enable access to CockroachDB for approved users by dynamically mapping Azure Active Directory users to corresponding SQL users in CockroachDB. This is achieved using JSON Web Tokens (JWTs), the Azure CLI, and CockroachDB’s SQL client. The process involves setting up prerequisite resources, securely connecting to the database using certificates, and configuring CockroachDB with specific cluster settings, particularly in version 23.1.5 and later. The use of the "upn" key for dynamic user mapping is emphasized to facilitate multiple user connections. The post provides guidance on obtaining necessary JWTs, configuring cluster settings, and securely connecting to CockroachDB using SQL commands, while also highlighting the importance of avoiding insecure connections in production environments.

CockroachDB: The Definitive Guide emphasizes the importance of architecting applications for scalability, resilience, and low-latency performance, especially in the face of disasters such as fires and cloud provider outages. At RoachFest23, Thomas Boltze from Santander highlighted that human error is often the primary cause of system failures, indicating the need for robust resiliency practices. He shared insights from Santander's journey to achieving a resilient payments system by continuously testing, identifying, and addressing failures, which eventually enabled the system to withstand data center outages and process payments without interruption. The key to their success lay in a culture shift towards curiosity, shared responsibility, and automation, resulting in a system designed to handle multi-region and multi-cloud failures, ensuring uninterrupted service even during significant outages.