May 2019 Summaries

Spring Data REST Services Powered By Distributed SQL – A Hands-on Lab` is a tutorial that guides users through the process of creating a Spring Boot REST service using YugabyteDB as its distributed SQL database. The tutorial assumes users have installed JDK version 1.8+ and Maven 3.3+, and provides instructions on installing YugabyteDB, cloning the sample app code repository, customizing the properties file, building and running the application, sending requests to create users, products, and orders, querying results using YSQL shell and REST API, and finally, suggests what's next for further learning and exploration with YugabyteDB.

YugabyteDB has released version 1.2.9 with over 20 new features, enhancements and bug fixes. The release includes updates to the YSQL API, such as support for default values, overriding, and comment on statements. Other improvements include RPC heartbeats for faster failovers after network partitions and enhanced documentation and community engagement. The company is also hiring and invites users to compare its capabilities with other databases like CockroachDB and Google Cloud Spanner.

The problem of runaway queries in distributed SQL databases arises when queries consume excessive I/O and CPU resources, impacting other clients even after the query has timed out. In YugabyteDB, this issue can be caused by a SELECT * command without a LIMIT clause executing on multiple tablets, leading to high disk utilization and CPU usage. To mitigate this problem, the YQL timeout is propagated to DocDB through an RPC, where it is converted to a deadline, allowing the iterator to build responses while preventing resource monopolization. This solution ensures that both YQL and DocDB have matching deadlines, reducing the impact of runaway queries on the cluster's performance.

This application is a blueprint for building IoT applications using Confluent Kafka, KSQL, Spring Boot, and YugabyteDB. It provides a real-time dashboard to track fleet management data from IoT-enabled vehicles, including overall distribution of vehicle types per delivery route, most recent subset of these vehicle types per shipment delivery route, and list of vehicles near road closures. The application architecture consists of Confluent Kafka as the streaming platform, YugabyteDB as the distributed SQL database, KSQL for stream processing on Kafka, Spring Boot for the data dashboard, and a program to simulate vehicle events being written into the Kafka stream. The application uses three tables in YugabyteDB to store aggregated data: Total_Traffic, Window_Traffic, and poi_traffic. The data dashboard queries these tables using Web Sockets and jQuery to push data to the webpage in fixed intervals.

I worked as an intern at YugaByte, a startup that offers a distributed database product called YugabyteDB. I was drawn to the company due to its recent funding round and the fact that it targeted the enterprise database space. The application and interview process were smooth, with two 45-minute interviews that convinced me to take the opportunity despite my initial skepticism about startups. Upon starting work, I found a well-organized development cycle with minimal testing and documentation, but extensive onboarding documentation and support from coworkers helped me get settled quickly. My first substantial task was fixing an issue with node blacklisting, which left me responsible for solving the problem, unlike in my previous internships where I was usually told how to go about it. Throughout my 4-month internship, I worked across the entire stack, contributing to various projects such as improving the timestamp datatype and adding features to the full-stack management platform. I also learned about distributed systems and considered CDC a potential future career path. The experience helped me gain technical skills, but also taught me valuable lessons about entrepreneurship, investing, and personal development.

Apache Kafka is a distributed event streaming platform that has gained massive adoption since its open-sourcing in 2011. It serves as a publish-subscribe messaging system for distributed applications and avoids tightly coupled app-to-app interactions. However, it lacks the persistence and consistency guarantees of a traditional database, making it necessary to integrate with a distributed SQL database like YugabyteDB to support business-critical event-driven applications. A distributed SQL database provides horizontal write scalability, native failover and repair, global consistency and low latency reads, as well as SQL's query flexibility, which are essential for modern enterprise applications. The integration of Kafka with a distributed SQL database enables developer agility by providing the benefits of both systems, including scalability, fault tolerance, and high performance.

YugabyteDB is a Consistent and Partition-tolerant (CP) database that handles network partitions and failovers in a way that ensures High Availability (HA) for most practical situations, even while remaining strongly consistent. During network partitions or node failures, the replicas of impacted tablets form two groups: a majority partition that can establish Raft consensus and a minority partition that cannot. The majority partition elects a new leader among themselves, and the DB accepts new writes after the leader election completes. Minority partitions are available for reads only, but not for writes. YugabyteDB's sharding, replication, and transactions architecture is similar to Google Cloud Spanner, which also handles network partitions and failovers in a CP database with high write availability. After identifying a 25-second recovery time issue during testing, the team dropped and recreated TCP connections, resulting in predictable 5-second failovers after a network partition. The solution allows for tuning of rpc_connection_timeout_ms to enable faster recovery times.

YugabyteDB 2.0 is expected in Summer 2019, promising significant advancements in distributed SQL capabilities, including support for all data types, window functions, and various query types such as joins and upserts. The release will also introduce enhancements to core database features like serializable isolation, change data capture, and master-follower & multi-master global deployments. Additionally, improvements are planned for secondary indexes and documents, with added security features including encryption at rest and support for popular Key Management Systems. A reimagined user experience is also being developed to simplify deployment and management of distributed SQL databases. The release will be production-ready and feature certified integrations with major ORMs and web frameworks.

I've summarized the text for you: As an experienced developer who spent almost thirty years at Oracle, I'm excited to join YugaByteDB as a Developer Advocate, where I'll work with a modern, cloud-native, and open-source distributed SQL database. The key differences between Oracle Database and YugabyteDB lie in their architecture, scalability, and fault tolerance, which are designed for the cloud era. Distributed SQL is a new paradigm that allows for scaling out and gaining fault tolerance through table shards automatically replicated across multiple machines. Cloud native means agility, freedom to change infrastructure choices, and relief from tedious tasks. Open source enables developers to choose software without licensing costs or financial commitment. I'm drawn to YugaByteDB's modernity, its promise to restore SQL's popularity in distributed architecture, and the opportunity to work with a high-growth startup team where I'll play multiple roles during a single day.

Distributed SQL offers an alternative approach to polyglot persistence that doesn't compromise agility in microservices development and operations. Unlike polyglot persistence, distributed SQL databases provide horizontal write scalability, high fault tolerance, and geo-distribution without compromising on low latency and high throughput. This allows for simplified development and operations, open source principles, and cloud native infrastructure, making them perfect for today's multi-cloud era. By adopting distributed SQL, developers can avoid the complexity of operating multiple databases and instead focus on building scalable and resilient data services that can be deployed across different cloud platforms with ease.

YugabyteDB 1.2 was recently Jepsen certified, marking a significant milestone for the database system. The certification process involved testing the database's consistency and transactional claims, with results indicating that YugabyteDB meets these requirements. A deep-dive into the testing approach and research methodology used by Kyle Kingsbury, creator of Jepsen, provided valuable insights into how to assess a database's reliability and performance. The webinar also addressed questions from attendees on various aspects of Jepsen testing, including heuristic approaches for evaluating internal tests, detecting cluster failures, and extending blackbox testing to include metrics and endpoints. With the certification in place, users can now expect to try out new generators implementations, compare YugabyteDB with other databases, and explore deployment options like macOS, Linux, Docker, and Kubernetes.