August 2022 Summaries

I'm Basava Kolagani, a first-year Masters student at the University of Wisconsin-Madison, who worked as an intern on the Distributed Storage and Transactions team at Yugabyte, a company that has developed a PostgreSQL-compatible, distributed SQL database. During my 13-week internship, I was amazed by the quality of the code and had the opportunity to work on two tasks that contributed to enhancing the performance of YugabyteDB, persisting Raft log entries asynchronously and expediting the bootstrapping of new nodes. I collaborated closely with my manager and mentor, receiving continuous feedback and guidance throughout the project. The entire interview process was seamless and fast, and I appreciated the opportunity to present my approach at design review sessions and receive suggestions from engineers. Overall, my internship has been awesome, and I would highly recommend Yugabyte and YugabyteDB as a great place to learn, contribute, and have fun.

The fourth annual Distributed SQL Summit 2022 will take place on September 14, 2022, with a speaker line-up that includes world-class keynotes, fireside chats with Fortune 500 leaders, and technical presentations by Yugabyte and industry experts. The event aims to share use cases, best practices, and real-world successes and challenges around database modernization with distributed SQL architecture. Attendees can gain insight from key brands such as Wells Fargo, Fiserv, Kroger, and Bed Bath & Beyond Inc., as well as learn about the latest developments in YugabyteDB, YugabyteDB Anywhere, and YugabyteDB Managed. The event also offers live hands-on workshops, 90-minute courses, and certification offerings from Yugabyte University to expand skills and knowledge.

YugabyteDB and Oracle offer different approaches to achieving high availability (HA) and scalability in databases. YugabyteDB is a single, coherent database distributed across multiple servers, built on the Raft protocol for resilience and scalability. It provides cloud-native resiliency, backup, bronze-level HA, silver-level DR, gold-level multi-master capabilities, and platinum-level protection with minimal complexity and cost. In contrast, Oracle offers various MAA levels (bronze to platinum) with different features such as RMAN, RAC, Data Guard, and GoldenGate for achieving HA and scalability. When comparing the two, YugabyteDB's built-in replication, sharding, and clustering capabilities simplify setup and provide high availability without additional complexity or cost. Oracle's MAA levels require more configuration and licensing, with some features only available in specific scenarios. Ultimately, YugabyteDB provides PostgreSQL compatibility and all of Oracle's HA features as part of its core, making it a viable alternative for cloud-native applications requiring high availability and scalability.

The YugabyteDB on Tanzu Reference Architecture is a modern approach to building, running, and managing business-critical workloads, designed to support flexibility, scale, and resiliency for organizations adopting cloud-native data layers. This reference architecture provides guidance on deploying YugabyteDB on Tanzu Kubernetes Grid, covering various deployment options and integrating with VMware SDDC, Backup and Recovery, Monitoring, and Logging. It aims to help developers unlock end-to-end scalability, performance, and resiliency by combining a cloud-native distributed database with a cloud-native application platform.

YugabyteDB is a high-performance, transactional distributed database for cloud-native applications, offering change data capture (CDC) capabilities that enable the automatic application of data changes to another data repository or consumption by applications and tools. Snowflake is a cloud-based data warehousing and analytics service, used in conjunction with Kafka Connect's Snowflake Sink Connector to stream data from YugabyteDB's CDC feature to Snowflake. The process involves setting up Confluent Cloud, creating a Docker image with the Debezium connector, configuring Kafka Connect, generating keys for authentication, and deploying the SnowflakeSinkConnector. Once set up, the pipeline can be tested by sending records to the Kafka topic and verifying that they are successfully replicated in Snowflake. Troubleshooting steps are also provided to address common issues such as insufficient privileges or failed record processing.

MinIO is being used for YugabyteDB backup and restore operations as part of the YugabyteDB Anywhere solution, which delivers a streamlined operational experience for enterprises using cloud-native environments at scale. This allows customers to configure backup destinations using Network File System (NFS) or MinIO, a high-performance, S3-compatible object storage solution that is native to Kubernetes and widely adopted by commercial and community users. With its 100% open source status under GNU AGPL v3 and strict adherence to the API, MinIO provides a reliable and secure option for storing backups in MinIO buckets, which can also be used for restoring databases when required.

This summary explores how to leverage the Change Data Capture (CDC) feature of YugabyteDB to transfer data from the database to Azure Blob Storage or Azure Data Lake Storage using Azure EventHub as an intermediary. The process utilizes YBDB's CDC, Kafka MirrorMaker, and Azure EventHub to stream changes from the YBDB write-ahead-log into Azure EventHub, where they are then stored in various formats such as Avro, Parquet, JSON, and CSV. The integration enables real-time data processing and analytics capabilities for large-scale enterprise workloads.

YugabyteDB provides default memory configuration values that should be evaluated for specific use cases and workloads. The three main buckets of memory utilization are the tserver process, the master process, and the postgres process. The tserver process is configured to use up to 85% of available RAM, while the master process uses 10%. Tuning memory settings requires observing memory utilization through YugabyteDB's observability windows and sizing memory correctly for each process. A starting rule of thumb is to set the tserver default_memory_limit_to_ram_ratio to 50% of total available RAM, but this should be adjusted based on actual memory usage. The block cache, memtables, log cache, read buffers, master process, and ysql processes all require tuning to optimize memory utilization. Increasing available memory for critical processes like tserver and master can improve performance, while leaving too little memory can lead to out-of-memory events.