May 2022 Summaries

Change Data Capture (CDC) is a technique used to capture changes in a source database system in real-time, enabling the streaming of those changes as events through a data processing pipeline for further processing. This technique is particularly useful in modern microservices-based architectures that involve many bounded services, and can be applied to various use cases such as search indexes, in-memory data cache, real-time notifications, data sync between sources, and fraud detection. CDC enables the creation of a data processing pipeline with components including a source database system, a supplier service to capture changes through CDC, a processor service to apply business rules, a consumer service to generate notification events, and a target database system to store notification events. The use case described leverages YugabyteDB's pull-based CDC API alongside a practical business case using a Spring Data processing pipeline for flight operations, capturing flight schedule change events in real-time from the source system and sending them to a processing pipeline that applies specific business rules to generate notifications. The pipeline is built using Spring Cloud Stream with message binders such as Apache Kafka, and the components are designed to work together seamlessly.

Temenos has successfully scaled its cloud platform to 100,000 business transactions per second by leveraging a greener and more scalable platform with YugabyteDB as a key component. The highwater benchmark utilizes three different databases - YugabyteDB for the manufacturing element, MongoDB for the distribution, and Redis for the session management - each serving a specific role in meeting functional requirements. Temenos chose YugabyteDB due to its ability to provide transactional consistency, which is crucial for banking applications. The platform's performance was demonstrated through complex transactions such as reservations, bookings, payments, and transfers, with YugabyteDB exceeding 430,000 database read/writes in an OLTP mode.

This blog post demonstrates how to stream data from YugabyteDB to downstream databases like MySQL, PostgreSQL, Elasticsearch, using YugabyteDB's Change Data Capture (CDC) feature. The CDC feature is introduced in YugabyteDB 2.13 and allows for the capture of changes made to the database, which can then be published to Kafka and streamed to other databases. The post provides a detailed example of how to deploy the necessary connectors to achieve this, including the Debezium Connector for YugabyteDB, JDBC Sink connector for PostgreSQL, and other sink connectors for MySQL and Elasticsearch. The example shows how to configure the connectors to replicate data from YugabyteDB to the downstream databases, and demonstrates how to insert data into YugabyteDB to trigger the replication process.

YugabyteDB's Change Data Capture (CDC) feature allows data to be sent from the database to Elasticsearch, enabling near-real-time search capabilities for applications that require it. This integration uses Logstash as a bridge between YugabyteDB and Elasticsearch, allowing data to be streamed into Elasticsearch in JSON format. The CDC architecture consists of YugabyteDB reporting changes to its write-ahead-log (WAL), which are then pulled by the Debezium Kafka connector and ingested through Logstash into Elasticsearch. This enables applications to search the data based on different filtering criteria using Kibana or directly from Elasticsearch, making it suitable for use cases such as building near-real-time search engines or upstreaming data for analytical or data science applications.

The YugabyteDB community has reached significant growth milestones, including over 5,500 members in their Slack group and two million clusters created by users. The community is actively engaged, with thousands of active contributors asking questions, offering suggestions, and providing expert advice. The company has also launched a revamped Yugabyte University program, which offers training and certification opportunities for distributed SQL, attracting over 5,000 students. The community is invited to join in making YugabyteDB better for everyone through various programs, including the contributor program, Slack channel participation, newsletter sign-up, and social media following.

YugabyteDB's Change Data Capture (CDC) feature can be used to send data from YugabyteDB to ClickHouse, an open-source column-oriented Online Analytical Processing (OLAP) database. This integration allows users to offload and retain the data for multi-year retention of legal or fiscal compliance, while also enabling real-time dashboards or batch-based reports and feeds for external applications or systems. The CDC architecture is a pull-based approach that reports changes from the database's write-ahead-log (WAL), and it can be integrated with ClickHouse through a Kafka configuration. To configure the sink in ClickHouse, users need to follow specific steps, including installing the server, client, and Kafka table engine, and creating a MergeTree table to store the CDC data ingested from the Kafka topic.

GORM is an Object Relational Mapping library for Golang that converts data between incompatible type systems using object-oriented programming languages. It provides an object-oriented layer between relational databases and object-oriented programming languages. YugabyteDB is a distributed SQL database built on a scalable and fault-tolerant design inspired by Google Spanner, offering deep integration with GraphQL projects and advanced RDBMS features like stored procedures and triggers. The example application demonstrates how to start a YugabyteDB cluster, download GORM, configure the sample project, create a table Employee, insert data into it, query the table, and verify the results using the Go code.

YugabyteDB is an open-source distributed SQL database designed for cloud-native applications that aims to eliminate the shortcomings of legacy databases by combining consistency and performance with horizontal scalability and native resiliency. It has helped Fortune 500 companies like Wells Fargo, Kroger, and General Motors modernize their data layers by simplifying developers' lives, lowering costs, and reducing risks. VMware Tanzu is a portfolio of products that enables customers to make the most of modern applications on any cloud, providing a foundation for a modern application platform with scale across multi-cloud and Kubernetes operations. The combination of YugabyteDB and VMware Tanzu offers end-to-end scalability, performance, and resiliency from the application to the data layer down to the cloud of choice, making it suitable for mission-critical microservices, edge and streaming applications, and geo-distributed workloads.

At Yugabyte, we are always on the lookout for ways to make your experience better. We are quite jazzed to announce a number of updates to YugabyteDB Managed that will make your life easier and help you reduce costs. Yugabyte Cloud is now called YugabyteDB Managed! Your YugabyteDB Managed clusters can now take a nap, allowing you to pause them without incurring cost for vCPU capacity, while disk and backup storage are charged at standard rates. You can also choose a maintenance window for patches and DB upgrades, review and customize it, and enjoy Cloud Shell enhancements with faster load times, longer sessions, and support for multi-tasking.

As an intern at DSS Asia 2022, Lakshmi gained valuable experience helping to run the Distributed SQL Summit event, which brought together developers, architects, and business leaders from across Asia and beyond. She learned that sometimes it's good to push beyond perceived limitations to innovate and explore new ideas, such as purchasing Amazon gift cards for a lucky draw instead of traditional prizes. Lakshmi also had to adapt to unique cultural nuances and time zone differences, taking mistakes as opportunities for growth and learning. Additionally, she witnessed the importance of effective communication in teamwork, combining strengths and resources to efficiently pull off the event. Overall, the experience was enriching and has given her a desire to be part of DSS Asia 2023, with plans to complete school and other projects before then.