April 2022 Summaries

The tutorial discusses the process of streaming data from Redpanda, a Kafka API-compatible platform, to Apache Pinot, an open-source software for online analytical processing (OLAP) with low latency. By using Docker, users can containerize both Redpanda and Pinot, making it easier to manage and integrate the two systems for efficient data streaming. The guide walks through setting up a Redpanda cluster, creating a topic, and then running Pinot, followed by configuring a real-time table in Pinot to receive data from Redpanda. It also includes steps for schema and table creation in Pinot, as well as ingesting JSON flight data from 2014 into Pinot via Redpanda, allowing for SQL queries that analyze flight patterns. This integration demonstrates the capability to handle and analyze large volumes of data quickly, highlighting Pinot's strengths in real-time data querying and Redpanda's reliability and speed as a streaming platform.

Securing a Redpanda cluster on Kubernetes involves configuring authentication to ensure that the users and clients accessing the API endpoints are verified. The blog outlines various methods to achieve this, such as using SASL, TLS, and mTLS, each offering different levels of security. SASL provides authentication without encryption, while TLS authenticates the server and encrypts communications. mTLS adds an extra layer by requiring mutual authentication between the client and server, making it the most secure but also the most complex to implement. The Redpanda custom resource definition (CRD) in Kubernetes includes four APIs—Kafka API, HTTP Proxy, Schema Registry, and Admin API—each of which can be individually configured for authentication to provide flexibility in securing the cluster. TLS is available for all APIs, offering server authentication and communication encryption, while SASL supports the SCRAM method for the Kafka API. The Redpanda operator utilizes cert-manager to generate necessary certificates for TLS and mTLS, with options for trusted certificate issuers through configuration parameters. Detailed documentation and tutorials are available to guide users through the configuration process, and feedback is encouraged via Slack or GitHub.

Redpanda offers a user-friendly solution for creating a streaming data environment with compatibility to Apache Kafka, and can be quickly set up on macOS using Homebrew and Docker Desktop. By executing a simple command line, developers can install Redpanda and its command line interface, rpk, to create a single-node cluster for interaction. For those without Docker or Homebrew, the installation process is straightforward, requiring the download of Docker Desktop and the installation of Homebrew through a terminal script. Once set up, users can experiment with Redpanda by utilizing various rpk commands to manage clusters, such as creating topics and streaming data. The guide provides details on scaling from a single-node to a three-node cluster and includes steps for producing and consuming messages. Additional resources, including community support and command documentation, are available for users seeking further assistance or exploration with Redpanda.

Amazon's introduction of the Arm-based Graviton processors in their EC2 service has presented a competitive alternative to the traditional x86 architecture for data-intensive applications, like those using Redpanda. Launched in 2018, the initial Graviton chips offered a promising price-to-performance ratio, but it was the release of Graviton 2 in 2020 that marked a significant leap in performance with up to 64 cores and 512 GiB of RAM. These Arm-based instances demonstrated a 20% edge in price/performance compared to Intel instances for high-throughput tasks, such as message ingestion in Redpanda, while also offering smoother regulation of SSD throughput. Benchmarks showed that Graviton 2 instances maintain consistent performance without the throughput oscillation observed in some Intel instances, making them a viable option for storage-intensive workloads. Furthermore, the process of porting applications to Arm, while potentially challenging for compiled languages, proved manageable for Redpanda, which benefited from thorough build process adjustments. Overall, Amazon's Graviton 2 instances provide a strong consideration for organizations seeking cost-effective and efficient cloud computing solutions.

Redpanda announced its acquisition of CloudHut, the creators of Kowl, a popular web UI for streaming data platforms, and plans to rename it Redpanda Console, integrating it deeply into Redpanda's ecosystem. This acquisition aims to enhance the developer experience by providing a user-friendly interface for managing Apache Kafka topics, consumer groups, and real-time data exploration. Kowl will remain open-source with a Business Source License, while Kowl Business as a standalone product will be discontinued. The Redpanda Console will support all existing Apache Kafka installations, and users will not experience disruption during the transition. Redpanda also acquired Kminion, a Kafka-to-Prometheus metric exporter, which will continue under the Apache 2.0 license. Redpanda is inviting the community to join its Slack channel and explore job opportunities as it expands its platform capabilities.