September 2020 Summaries

The concept of consensus systems is explored, with a focus on durability as the primary reason for using such systems. The original properties of a consensus system are modified to make them more practical and usable in real-world scenarios. A new set of rules is introduced that are agnostic of durability requirements, allowing for flexibility in deploying nodes and making durability decisions. Various use cases are presented, including large numbers of replicas spread across multiple data centers, four zones with one node in each zone, and six nodes spread over three zones. The configurations in these scenarios are uncomfortable for traditional majority-based consensus systems but allow for creative combinations that achieve better trade-offs. A way to reason about flexibility is proposed, which involves a common view shared by the Request and Election algorithms. The Worst Case Scenario highlights the potential for stall or loss of data if two nodes fail without being known, leading to compromise and mitigation strategies. Practical Balance emphasizes the low likelihood of certain failure modes, such as a replica node failing after an acknowledgment, and orders of magnitude are discussed in terms of request completion versus leader election frequency.

Vitess has integrated Orchestrator, an open-source MySQL replication topology management and high availability solution, as a native component to achieve reliable failover, availability, and topology resolution of its clusters. Orchestrator uses a holistic approach to failure detection, triangulating checks with the servers' checks, whereas traditional methods rely on a single observation to act. This approach allows orchestrator to detect failures more reliably and make goal-driven decisions to ensure a cluster is always in a state compatible with Vitess' expectations. The integrated Orchestrator can run Vitess-native functions, such as locking shards or fetching tablet information, and is aware of the replication tree's health and status. It promotes a server based on multiple limiting factors and non-server-specific rules, allowing users to choose a failover and recovery policy described in code. This integration enables more reliable and efficient management of MySQL replication clusters, reducing the risk of cluster corruption and improving overall availability.

A consensus system aims to ensure that a single value or series of values are accepted and kept consistent across multiple nodes, with the primary guarantee being that once a value is acknowledged as accepted, all other values must be rejected. The goal is to achieve durability at scale without giving absolute guarantees, as no system can provide 100% failure tolerance. A single leader consensus system is used, where the leader accepts requests and makes them durable, and a new leader can be elected to resume requests without divergence or loss of data. The rules governing such systems include accepting a series of requests in strict order, limiting complexity by using a single leader design, and ensuring that at most one leader prevails during concurrent attempts to elect a leader. These rules are intentionally generic to allow for creativity and flexibility in achieving the goals of durability and safety without requiring specific parameters like majority quorums or proposal numbers.

The author, a software engineer with two decades of experience working with MySQL, has become frustrated with the lack of integration between open source tools in the MySQL ecosystem. Despite efforts to collaborate on projects, they found that deploying and automating third-party tools was often challenging due to differences in infrastructure, deployment mechanisms, and network setups. The author believes Vitess can address these issues by providing a full-blown infrastructure framework for MySQL, which would enable online schema migrations, high availability, consistency, disaster recovery, and more with minimal setup required. The author's vision is compatible with the PlanetScale team's plan to simplify Vitess installation, improve its ease of use, and bring in more infrastructure solutions, ultimately creating a "Grand Unified Theory" for database infrastructure that integrates existing tools and knowledge.