July 2022 Summaries

Redpanda, a data streaming platform optimized for modern hardware, requires careful cluster sizing for effective production-level performance. While its default settings suffice for basic use, production environments benefit from fine-tuning to avoid bottlenecks or underutilization. Key considerations for sizing include I/O performance, with a recommended 2:1 core-to-disk ratio using NVMe SSDs for optimal throughput; memory allocation of at least 2GB per core to enhance processing efficiency via its thread-per-core model; network bandwidth management, crucial in cloud deployments for predictable performance; and strategic storage solutions, like Redpanda's Tiered Storage, to balance high throughput with extensive data retention capabilities. These factors ensure Redpanda efficiently scales within infrastructure limits, minimizing operational costs.

Building a cloud platform from scratch at Redpanda has been both a challenging and rewarding experience, as it involves making numerous quick decisions and adapting to the dynamic nature of an early-stage startup. The team has developed a cloud service offering that provides customers with the choice of deploying Redpanda clusters either within Redpanda's cloud environment or their own virtual private cloud (VPC), while managing the provisioning and maintenance. Redpanda Cloud is distinguished by features like Dedicated and Bring Your Own Cloud (BYOC) clusters, which cater to data sovereignty requirements by ensuring data or cloud credentials remain within the customer's environment. The platform leverages modern technologies such as Kubernetes, Terraform, and Prometheus, allowing engineers to work with cutting-edge tools and experiment with their projects. As the Cloud and Site Reliability Engineering (SRE) teams have evolved, they have focused on improving monitoring, incident response processes, and platform reliability, making it a thrilling space for engineers interested in data streaming and cloud technologies.

Developers face challenges in meeting stringent privacy and compliance regulations, such as GDPR, when handling personal information in applications, particularly in continuous data streams. This has led to the adoption of sophisticated data streaming and storage infrastructures. One such solution is Redpanda, a Kafka API-compatible, Zookeeper-free, JVM-free streaming data platform, which, when integrated with Delta Lake, enhances compliance capabilities by managing malformed data, auditing, and purging relevant data. A practical application of this integration is demonstrated through an eCommerce app that streams clickstream user data, using Redpanda for data streaming and Delta Lake for structured storage and compliance tracking. The setup involves using tools like Apache Spark for data processing, with the entire process detailed in a GitHub repository, offering a scalable and compliant framework for various industries beyond eCommerce, such as financial services and industrial automation.

A conversation unfolds between the narrator and Karl, a DevOps Manager eager to run Apache Kafka on Kubernetes, where the narrator challenges the practicality of such a move. While Kubernetes offers benefits like process isolation and resource optimization, its integration with Kafka poses challenges, especially in networking and caching, since Kafka's architecture doesn't align seamlessly with Kubernetes' operational model. The discussion highlights the complexities of using Kafka on Kubernetes, such as handling external connections and cache management, and suggests Redpanda as an alternative that better suits Kubernetes environments due to its architecture and operational efficiencies. The narrator emphasizes that while Kubernetes is a powerful tool, it is not always the best fit for every application, urging consideration of the operational context when deciding on deployment strategies.

Out of Memory (OOM) events in Linux can significantly affect programs like Redpanda, which uses the Seastar library to optimize hardware utilization, making it vulnerable to the OOM Killer—a kernel function designed to terminate the largest process with the lowest priority when memory is scarce. To address these issues in a Kubernetes (K8s) environment, the Redpanda team developed a custom operator that automates deployment and optimizes memory allocation by leveraging cgroups, adjusting memory reservations, and isolating processes within containers. By reserving 10% of memory for the operating system and reducing the memory allocated to Redpanda, the team mitigated memory pressure and OOM events, ensuring the stability of long-running clusters and improving resource management for larger deployments. These adjustments allow for better observability and debugging, enhancing the overall experience for Redpanda users in a containerized cloud environment.