December 2025 Summaries
181 posts from Redpanda
Filter
Month:
Year:
Post Summaries
Back to Blog
In this third installment of a series on high availability (HA) and disaster recovery (DR) with Redpanda, the focus is on deploying Redpanda clusters across multiple availability zones (multi-AZ) to enhance system resilience. The post explains that by distributing brokers over three or more zones, the architecture minimizes single points of failure, as the Raft consensus algorithm ensures that the cluster can still function if one zone fails. While a multi-AZ setup provides increased availability, it also incurs higher costs and potential latency due to the network distance between zones and additional data transfer charges. The introduction of follower fetching, a feature in Redpanda 23.2, allows consumers to read data from the closest replica, which reduces cross-zone traffic and associated costs. The article discusses deployment options using Redpanda Cloud or self-hosted solutions with Terraform and Ansible, emphasizing the balance between cost, complexity, and the need for extreme availability, with a promise of exploring multi-region deployments in the next post.
Dec 31, 2025
2,332 words in the original blog post.
The Medallion Architecture, originally popularized by Databricks, is a structured approach to organizing data storage into three layers—Bronze, Silver, and Gold—each representing increasing levels of data refinement and quality. This architecture has been widely adopted by major companies, including Microsoft, as it effectively manages data progression from raw ingestion to analytics-ready datasets, enhancing data quality and accessibility. Redpanda, integrated with Apache Iceberg, facilitates the implementation of the Medallion Architecture by allowing seamless integration of batch and real-time analytics, eliminating the need for complex configurations, and supporting schema evolution and ACID compliance. By utilizing open file formats like Parquet and ORC, the architecture ensures efficient data processing across layers, while Redpanda's integration with Iceberg enables near real-time data transformations, streamlining the transition from raw to refined data. This multi-layered approach is especially beneficial for companies seeking to unify historical and streaming data for rapid, insightful analysis, aligning with Redpanda's vision for universal analytics.
Dec 30, 2025
2,098 words in the original blog post.
Redpanda has introduced three major new features: Tiered Storage, Protobuf support in the Schema Registry, and a tech preview for Wasm Data Transforms, aimed at enhancing capabilities for developers building real-time applications. Tiered Storage allows for seamless unification of real-time and historical data by archiving log segments to cloud storage and replaying them as needed, while Protobuf support expands schema management options, complementing existing Avro support and catering to diverse programming languages. The Wasm Data Transforms tech preview enables the creation of WebAssembly-based scripts for data transformation tasks directly on topics, reducing the need for external stream processors and facilitating tasks like data scrubbing and normalization. These features, while currently available in tech preview and not yet recommended for production use, demonstrate Redpanda's commitment to innovation and improving data handling processes in real-time applications.
Dec 30, 2025
967 words in the original blog post.
Change Data Capture (CDC) is a software design technique that enables the identification and tracking of changes in data across systems, facilitating efficient and real-time data transfer from source to target systems. Traditional batch data capture methods often suffer from inefficiencies and high latency, as they do not account for changes in data between intervals, whereas CDC improves performance by capturing changes continuously and in real-time, which is crucial for systems requiring up-to-date data insights. CDC is particularly beneficial for preventing dual-writes in microservices architectures, ensuring regulatory compliance, enabling real-time data streaming, and facilitating asynchronous data replication. Common CDC methods include Date Column Differences, Table Differences, Trigger-Based, and Log-Based, with the latter being favored for its reliability, scalability, and minimal impact on source systems, despite the complexities of dealing with various database log formats. Log-Based CDC is generally recommended for large-scale operations due to its ability to decouple source and target systems and its robust community support, making it an ideal solution for constructing accurate and actionable data pipelines.
Dec 30, 2025
2,261 words in the original blog post.
Businesses aiming to analyze large-scale advertising campaigns in real time can leverage Amazon Redshift, a fully managed petabyte-scale data warehouse known for its speed, scalability, and cost-effectiveness, particularly when integrated with platforms like Apache Kafka or its modern alternative, Redpanda. Redshift's architecture supports the rapid ingestion and processing of streaming ad event data, which is critical for advertisers needing real-time insights into metrics such as impressions, clicks, and conversions. By integrating Redpanda with Amazon Redshift, companies can simplify and scale their data storage, reporting, and analytics processes, enabling agile decision-making and campaign optimization. The integration process involves creating a synthetic dataset, setting up Redpanda for data ingestion, and using a JDBC sink connector to feed streaming data into Redshift, allowing for sophisticated data querying and analysis. This setup ensures consistent performance as data volumes grow, making it particularly beneficial for ad tech platforms handling data from multiple sources. The process also emphasizes the importance of optimizing query performance, maintaining data consistency, and managing system performance through strategies like autoscaling and monitoring alerts.
Dec 30, 2025
3,131 words in the original blog post.
Neo4j, a graph database management system, is designed for efficiently storing, querying, and managing complex networks of data compared to traditional relational databases, which rely on table structures and can struggle with complex queries. This tutorial demonstrates how to integrate Neo4j with Redpanda, a streaming data platform similar to Apache Kafka, to perform event-driven graph analysis. By creating a streaming application, users can handle data such as movies and their directors, mapping this information into Neo4j's nodes, relationships, and properties. The tutorial guides the setup process, including configuring Docker, creating a Redpanda topic, setting up Neo4j, and building a Java application to stream data into the Neo4j database. Users learn to execute Cypher queries in Neo4j's browser interface for data analysis, exemplifying Neo4j's applicability in areas like social networks, recommendation engines, fraud detection, and knowledge graphs. The tutorial also encourages exploring Redpanda's Community edition and provides additional resources for learning and community support.
Dec 30, 2025
2,646 words in the original blog post.
The tutorial explores the integration of Hazelcast and Redpanda to create scalable, fault-tolerant applications that handle real-time data, using a pizza delivery service as an example. Redpanda, a Kafka-compatible streaming platform, efficiently manages high-throughput data streams, while Hazelcast enables real-time processing and enrichment of these streams with contextual data. The guide details the setup of both technologies using Docker and Python, demonstrating how to generate and process pizza order streams, enrich them with recommendations, and return the results to Redpanda. The collaborative use of Hazelcast and Redpanda showcases a robust framework for building resource-efficient, real-time applications capable of scaling dynamically to handle increased user activity, such as during high-demand events. The tutorial encourages feedback and engagement through community platforms and highlights further resources for exploring these technologies.
Dec 30, 2025
2,005 words in the original blog post.
Building future-proof and scalable data systems is a challenging endeavor that requires careful tool selection, involving considerations such as data warehouses, object storage, cloud providers, protocols, and programming languages. To aid in this decision-making process, several best practices can be employed. These include understanding the problem thoroughly by defining technical and business requirements, evaluating existing tools and resources without defaulting to familiar solutions, choosing tools that specialize in addressing the specific challenge, conducting comprehensive research through features analysis and stakeholder consultations, and ensuring the selected tool's scalability to handle future demands. By following these practices, organizations can make informed decisions while avoiding the pitfalls of relying solely on familiar tools or over-promising solutions.
Dec 30, 2025
969 words in the original blog post.
Redpanda 22.2 is a major release focused on simplifying the management and operation of streaming data infrastructure by introducing new features that enhance day-2 operations, data pipeline reliability, and security. The release includes Remote Read Replicas, which function as a CDN for streaming data, allowing topics to be served from analytics clusters without duplicating data, thus reducing time and cost for DevOps teams. Continuous Data Balancing ensures optimal cluster performance by redistributing partitions to prevent data imbalance and automatically recovering from node failures. The new Redpanda Console, developed following the acquisition of CloudHut Kowl, provides a developer-friendly web UI for debugging and managing clusters, featuring tools like Programmable Push Filters for efficient message filtering and management of Kafka ACLs. Other enhancements include full-disk handling to prevent node crashes, integrated IAM roles for more secure cloud storage access, mTLS authentication with principal mapping, static consumer group membership to eliminate rebalancing, and a streamlined Prometheus metrics endpoint. These advancements are designed to enhance simplicity, reliability, and efficiency for developers, SREs, and sysadmins in managing large-scale, real-time data applications.
Dec 30, 2025
2,184 words in the original blog post.
OpenID Connect (OIDC) enhances data streaming platforms like Redpanda by providing a centralized, secure identity and access management layer built on OAuth 2.0, allowing for seamless integration with providers such as Okta and Keycloak. This setup supports single sign-on (SSO) and fine-grained access control, aligning with modern security practices without compromising performance. In Redpanda, OIDC authentication is available across various deployment methods but is limited to specific APIs in the Enterprise Self-Managed version. The blog post guides through configuring Redpanda's OIDC authentication using Docker Compose and Keycloak, detailing the steps to set up identity providers, configure Redpanda Console and APIs, manage user authorization through Access Control Lists (ACLs) and role-based access control, and test API access with OIDC tokens. It emphasizes the importance of audit logging in monitoring authentication and authorization events, noting the integration with the Open Cybersecurity Schema Framework (OCSF) for standardized audit logs. The post provides a comprehensive framework for securing Redpanda deployments while ensuring user-friendly operational flows.
Dec 30, 2025
2,015 words in the original blog post.
The blog post provides a step-by-step guide on setting up Redpanda and using it with Node.js and TypeScript to build a simple terminal chat room. It begins with instructions on installing Node.js and leveraging Kafka-compatible client libraries like kafkajs for Redpanda. The setup involves creating a project folder, initializing a package, and installing necessary dependencies such as TypeScript and uuid. After setting up Redpanda and creating a topic for the chat room, the guide outlines the creation of a producer and consumer using kafkajs to send and receive messages. The producer connects to the Kafka instance, allowing users to send messages, while the consumer subscribes to the chat-room topic to receive and print messages. The blog concludes by encouraging readers to explore various use cases for Redpanda and hints at future developments for a cloud service to simplify Redpanda's technical details.
Dec 30, 2025
868 words in the original blog post.
Integrating Redpanda and Benthos for streaming applications is straightforward, leveraging the strengths of both platforms to simplify data processing and persistence. Benthos, an open-source stream processor, offers features like data mapping, filtering, and enrichment through a minimal configuration setup, while Redpanda provides a Kafka-compatible streaming platform that enhances performance and reduces operational complexity. The guide demonstrates setting up both services using Docker, creating a network, and running instances of Redpanda and Benthos to implement a basic data pipeline. Data is sent from a standard input to a Kafka topic using Benthos, and then read back with transformations using a consumer configuration that employs the Bloblang processor to mutate data into structured documents or fallback string versions, adding metadata like message length and topic. The process is illustrated with Docker commands, highlighting the ease of managing unstructured and structured data within this environment.
Dec 30, 2025
681 words in the original blog post.
Modern businesses produce vast amounts of raw data from diverse sources like IoT devices and social media, requiring processing and analysis to extract valuable insights. End-to-end data pipelines provide a framework to connect and unify these disparate data sources, transforming raw data into formats that enhance business intelligence or offer customer value. These pipelines, which can be partially or fully automated, streamline data flow from ingestion to output, ensuring speed, accuracy, and scalability. They can be categorized into batch processing pipelines, ideal for large-scale periodic data handling, and streaming pipelines, which process data in real time for applications needing immediate insights. Unlike ETL pipelines that focus on data preparation and storage, end-to-end pipelines enable data consumption and insights across various stages. Streaming pipelines, in particular, support real-time analytics and operational efficiency, breaking down data silos and ensuring seamless integration across systems. Key components of these pipelines include data ingestion, real-time processing, storage, and consumption layers, all supported by continuous monitoring, orchestration, and robust data security measures. Platforms like Redpanda offer an end-to-end solution for real-time data processing, providing high performance, low latency, and compatibility with tools like Grafana and Prometheus for easy setup and monitoring.
Dec 30, 2025
1,903 words in the original blog post.
The text discusses the transition from centralized to decentralized and distributed data processing systems in the financial services industry, emphasizing the importance of building a robust data streaming infrastructure to support AI applications. It outlines the technical aspects, such as data governance and the need for visibility, standardization, and security, which are crucial for ensuring data quality and reducing errors. The narrative compares data systems to the human nervous system, highlighting how streaming data platforms can efficiently handle large data loads by quickly transmitting information, enabling real-time data processing and integration with microservices. It advocates for reducing reliance on batch pipelines in favor of agile, stateless pipelines to enhance scalability and cost-effectiveness. Additionally, the text suggests using proven data strategies to streamline data pipelines, preparing them for machine learning model training, and introduces Redpanda as an alternative to Kafka for building efficient streaming data platforms. The author promises a detailed use case to demonstrate these strategies in action, encouraging readers to subscribe for updates and reach out for support in implementing Redpanda.
Dec 30, 2025
1,192 words in the original blog post.
Redpanda is a real-time streaming engine designed for modern applications, offering capabilities that go beyond the Kafka protocol, including inline Wasm transforms and geo-replicated hierarchical storage. It is scalable from small projects to handling petabytes of data globally, with a focus on speed and ease of use, akin to running nginx without dependencies. The project, which began in January 2019, is released under the Source Available License (BSL) to prevent cloud providers from offering it as a service, while allowing open use for most users. The Redpanda community embraces contributions and offers enterprise features that are distinctly configurable. The platform is easy to start, with simple Docker deployment options and an active ecosystem, supported by insights from industry experts and community builders.
Dec 30, 2025
677 words in the original blog post.
Businesses that manage large volumes of data daily face increasing challenges in data analysis, necessitating the use of efficient databases like ClickHouse and streaming platforms such as Redpanda. Traditional relational databases are inadequate for the analytical demands of growing companies, leading to the adoption of OLAP databases like ClickHouse, known for its speed and scalability in handling vast data volumes. ClickHouse can integrate with various databases and streaming platforms, including Kafka and Redpanda, to facilitate real-time data processing. Redpanda, compatible with Kafka's API, offers a resource-efficient and high-performance streaming solution, which can be seamlessly integrated with ClickHouse for businesses that require reliable real-time analytics. The text provides a step-by-step tutorial for setting up a real-time streaming system using Redpanda and ClickHouse, using a real estate agency as a scenario to illustrate how these technologies can streamline data reporting and analysis processes. The tutorial guides users through setting up Docker containers for both platforms, creating databases and tables, and running SQL queries for data analysis, ultimately demonstrating how to achieve fast, reliable data streaming and querying.
Dec 30, 2025
2,174 words in the original blog post.
Modern SaaS companies are increasingly integrating AI-native interfaces and exploring the Model Context Protocol (MCP) to enhance API interactions, particularly with large language models (LLMs). This shift is facilitated by tools like Redpanda's open-source codegen plugin, which allows gRPC APIs to be exposed via MCP with minimal effort, leveraging the strong schema and code generation capabilities of Protocol Buffers. Introduced by Anthropic in November 2024, MCP is designed for LLM tools with structured inputs and consists of client and server components that enable diverse interactions, such as database lookups or cloud service configurations. Redpanda has developed protoc-gen-go-mcp, a plugin that seamlessly generates MCP servers for gRPC or ConnectRPC APIs, making Redpanda Cloud a prominent player in LLM integration. The tool supports both stdio and HTTP modes, with Claude Desktop being a primary client using stdio. This development allows for easy installation and interaction with Redpanda Cloud, enhancing AI-driven functionalities while maintaining security through permission checks and safeguards against destructive actions.
Dec 30, 2025
1,196 words in the original blog post.
In a comprehensive tutorial, users are guided through the creation of a real-time data pipeline that streams simulated e-commerce order data from Redpanda, a high-throughput data streaming platform, into Snowflake, a scalable data warehouse designed for analytics. The process involves setting up a Redpanda topic to receive data, creating necessary user permissions, and configuring both Redpanda Connect and Snowflake for seamless data integration. The pipeline is designed to provide real-time insights for applications needing immediate data processing, such as fraud detection and sales analytics. Users learn to simulate data with a Python script or Kafka producer and securely manage credentials using Redpanda's secret management. The tutorial concludes with verifying data flow into Snowflake, offering real-time analytics capability, and encourages experimentation with data transformations and additional data sources.
Dec 30, 2025
2,802 words in the original blog post.
This summer, Redpanda has introduced several significant updates to enhance its multi-cloud services, streamline the Apache Kafka developer experience, and strengthen enterprise compliance. The limited availability release of Redpanda Cloud on Azure now operates in six regions, offering enhanced security through Azure Private Link and supporting Azure's storage solutions. Redpanda Serverless has seen advancements with the addition of a fully managed Schema Registry and a 99.5% uptime SLA, alongside new features like dynamic cluster creation through the Cloud API and billing information integration. Version 24.2 brings developer enhancements, such as in-broker Data Transforms with JavaScript support, JSON Schema support in the Schema Registry, and compliance upgrades, including FIPS and HIPAA self-attestation. Additionally, Redpanda has implemented client throughput quotas per node and plans to expand its presence on Azure with new region support. Users can explore these updates through various Redpanda offerings, including the free Community Edition and Redpanda Cloud Dedicated trial.
Dec 30, 2025
1,159 words in the original blog post.
Redpanda Connect has introduced dynamic plugins, a new feature that enhances its data streaming platform by allowing for runtime plugin creation and loading, offering greater flexibility and language interoperability. Previously, plugins needed to be built in Go and compiled directly into the Redpanda Connect binary, but the new dynamic plugins framework enables the use of external executables that communicate with the main process via gRPC. This supports plugins in any language that supports gRPC, such as Python, thus lowering barriers for data scientists and ML engineers who prefer programming in languages other than Go. The new system uses gRPC for efficient data transfer and process isolation, enabling independent deployment and modular design while maintaining performance. The dynamic plugins are particularly beneficial for integrating non-Go libraries, performing complex data transformations, and facilitating real-time machine learning inference, with SDKs available for both Go and Python to streamline development. While dynamic plugins provide flexibility and ease of integration, compiled plugins are still recommended for performance-critical applications. The introduction of this feature marks a significant advancement for Redpanda Connect, expanding its capabilities and opening new possibilities for data streaming applications.
Dec 30, 2025
1,428 words in the original blog post.
The manufacturing industry has undergone significant transformation since the Industrial Revolution, shifting from a focus on manpower and machinery to a data-driven approach powered by the Industrial Internet of Things (IIoT). The integration of IoT technologies, Big Data analytics, and cloud computing has enabled factories to enhance efficiency and productivity through real-time data insights and automated processes. IIoT allows for predictive maintenance, real-time equipment monitoring, and energy optimization, resulting in reduced downtime and cost savings. A proposed reference architecture for IIoT highlights the use of components like Redpanda, a central data hub, to facilitate scalable real-time data ingestion and processing. Redpanda supports the seamless connectivity and communication between machines and IIoT devices, enabling effective data collection, analysis, and automation. With its lean and cost-efficient design, Redpanda is particularly suited for resource-constrained environments, offering manufacturers a robust foundation for implementing IIoT systems and driving continuous innovation in industrial operations.
Dec 30, 2025
1,755 words in the original blog post.
Enterprise AI adoption is challenged by a critical security gap as organizations stream sensitive data into vector databases, which traditionally store vector embeddings in plaintext, posing significant risks of data breaches. This is particularly concerning for regulated industries like finance and healthcare, where compliance is paramount. In response, Cyborg has partnered with Redpanda to create a secure streaming pipeline that encrypts vectors before storage, allowing for semantic search and RAG applications on encrypted data. This partnership introduces CyborgDB, a vector encryption proxy that transforms existing database infrastructures into encrypted vector stores, maintaining data security at rest, in transit, and during use. Redpanda Connect, a Kafka-compatible streaming platform, complements this by efficiently handling high-volume data ingestion and transformation. The integration of CyborgDB into Redpanda Connect pipelines enables organizations to deploy real-time AI capabilities without compromising on security, thus meeting compliance requirements and protecting sensitive data from potential breaches.
Dec 30, 2025
1,509 words in the original blog post.
At the Hydra Conference 2021, Alex and Noah discussed with Alexey Fyodorov the process and insights gained from building a new storage engine specifically designed for the Kafka API, incorporating a thread-per-core execution model alongside the Raft data replication model. This innovative storage solution eliminates the use of virtual memory and page cache, opting instead for custom read-ahead and write-behind strategies to achieve predictable latencies. The discussion emphasized the co-design approach that integrates these strategies with Raft, aiming to optimize data handling and replication efficiency.
Dec 30, 2025
86 words in the original blog post.
Oxla's query execution engine optimizes performance and memory utilization by forming a pipeline of processors, where each processor computes output batches from input ones based on a subset of columns. The article introduces a problem called "Minimum Expression and Column Subset" (MECS) to minimize the number of expressions and columns processed, enhancing efficiency. The solution involves using graph theory and Hall's theorem to compute an optimal valid solution by determining a max matching in a bipartite graph representing dependencies between expressions and columns. The authors present a polynomial-time algorithm that provides a solution that is at most twice the size of the optimal, while further refinements can yield the optimal solution. This approach, while effective for nested SELECT queries, shows limitations with joins, requiring different optimization techniques. The article invites readers to explore Oxla's features firsthand through its BETA deployment.
Dec 30, 2025
2,579 words in the original blog post.
The exponential growth of data has challenged traditional databases, particularly with unstructured and semi-structured data, leading to the rise of NoSQL databases like Amazon DynamoDB, which offers flexible storage without fixed schemas. Amazon DynamoDB is a fully managed NoSQL service by AWS, known for its low-latency and high-throughput performance, automatically distributing data across servers to meet scalability and throughput demands, supporting both key-value and document data models. The text outlines a tutorial on migrating data from PostgreSQL to DynamoDB using Debezium and Kafka Connect, highlighting the setup of various components like Redpanda and Kafka Connect instances, configuration of connectors, and the eventual data pipeline from PostgreSQL to DynamoDB. Companies like Lyft, Snapchat, and Capital One use DynamoDB for handling diverse data types, scalability, and rapid development needs, illustrating its practical applications in big data scenarios. The tutorial emphasizes the seamless scalability and flexibility of DynamoDB, especially for applications requiring rapid development and diverse data types, showcasing a practical migration example for a delivery company anticipating increased data loads.
Dec 30, 2025
2,578 words in the original blog post.
Redpanda has enhanced its cluster configuration system by using Raft and implementing a central API for configuration changes, which includes immediate validation to prevent operational issues in production systems. This improvement eliminates the need for traditional configuration files and reduces the risk of errors caused by invalid entries. The validation process now applies to all changes, whether made interactively or via scripted commands, ensuring that only verified configurations are stored. Redpanda Version 22.1 also introduces system sizing validation to manage resources efficiently during topic and partition creation, safeguarding against exceeding server capabilities. The use of C++20 and the Seastar framework enables robust live configuration changes without requiring full system restarts, allowing for atomic updates and active notifications of changes. Modern C++ features facilitate the translation of internal types to external metadata for the Admin API, and the Seastar model simplifies code while maintaining performance.
Dec 30, 2025
1,239 words in the original blog post.
Redpanda aims to simplify data streaming development, and this blog post introduces the Redpanda GitHub Action, which facilitates automated testing and continuous integration for projects using Redpanda. GitHub Actions allow for automation of software workflows, and the Redpanda GitHub Action specifically enhances the continuous integration process by enabling developers to run tests with a real Redpanda instance on GitHub CI. This setup is advantageous even for those not using Redpanda in production, as it serves as a drop-in replacement for Apache Kafka distributions, offering benefits like faster boot times and reduced RAM usage. The post provides a detailed guide on configuring GitHub CI to use the Redpanda GitHub Action, demonstrating its efficiency and cost-effectiveness in terms of reduced build minutes and resource usage. By using this tool, developers can ensure their code's reliability and foster collaboration through GitHub, with additional support and resources available via Redpanda's community and documentation.
Dec 30, 2025
866 words in the original blog post.
This blog offers a comprehensive tutorial on building a data pipeline on Kubernetes using Redpanda, Postgres, and the Go programming language. The tutorial guides users through setting up Redpanda and Postgres on Kubernetes, deploying producer and consumer applications, and exploring the infrastructure's ability to scale. The producer application collects data through an HTTP endpoint and sends it to a Redpanda topic, while the consumer application processes this data, inserting it into a Postgres database. The Redpanda cluster and Postgres setup utilize Helm for deployment, and the tutorial demonstrates horizontal scaling by increasing the number of consumer application instances. The producer and consumer applications are built using different Go client libraries—franz-go and sarama, respectively—to highlight their unique features. The guide emphasizes the potential of Redpanda and Kubernetes for developing scalable, cloud-native applications and suggests exploring tools like the Horizontal Pod Autoscaler and KEDA for automatic scaling. The tutorial concludes by inviting readers to engage with the Redpanda community for further learning and exploration.
Dec 30, 2025
2,101 words in the original blog post.
Real-time transaction processing has revolutionized how businesses handle data by allowing immediate analysis and response to incoming information, making it crucial for industries like finance where quick decision-making is essential. This tutorial explores a real-time transaction processing solution using Redpanda, Apache Flink, and MongoDB, which together create a robust stack capable of handling high-volume data streams with low latency. Redpanda serves as a streaming data platform with a Kafka API, while Flink offers real-time data processing capabilities, and MongoDB provides a flexible NoSQL database for high-speed data ingestion and real-time analytics. The integration of these technologies is demonstrated through a demo project that sets up a real-time transaction processing application, highlighting their ability to process and analyze transactions efficiently, thereby enabling businesses to make timely decisions and deliver enhanced customer experiences. Challenges such as ensuring data consistency and exactly-once processing are addressed, and the tutorial provides step-by-step guidance on setting up the system locally using Docker, developing a Flink application, and producing financial transaction events into Redpanda for processing and ingestion into MongoDB.
Dec 30, 2025
3,115 words in the original blog post.
Apache Kafka Streams (KStreams) is a Java client library designed for real-time stream processing, facilitating smoother application and microservices development. It processes incoming data streams, such as aggregating word occurrences or filtering fraudulent transactions, using a domain-specific language (DSL) that provides methods like Map, Filter, and Join for data transformation. Redpanda, an alternative to Apache Kafka, offers API compatibility, ease of management, and notable performance improvements, serving as an effective message broker for KStreams without requiring code changes. In a tutorial setting, users can set up a KStreams application with Redpanda, utilizing Docker for configuration and exploring functionalities such as stream aggregation and joining multiple streams. The process involves setting up producers and consumers, defining topics, and using KStream and KTable objects to manage and transform data flows, with Redpanda serving as a seamless drop-in replacement for Kafka to enhance message-processing capabilities.
Dec 30, 2025
1,943 words in the original blog post.
Apache NiFi is an open-source tool initially developed by the NSA as NiagaraFiles, designed to facilitate data movement and processing between systems using a flow-centric approach with a directed acyclic graph (DAG) model. It offers a no-code/low-code development environment via a drag-and-drop interface, allowing users to define dataflows visually and ensuring data provenance for auditability and governance. NiFi excels in lightweight data transformations and connectivity, boasting over 200 processors for integration with various systems. Despite functional overlaps with pub/sub messaging systems like Redpanda, Apache Kafka, and Pulsar, NiFi's unique focus on flow management makes it a suitable choice for specific use cases, particularly when paired with systems like Kafka for enhanced functionality. This combination provides a robust solution for data integration, leveraging Kafka's event bus capabilities alongside NiFi's graphical user interface.
Dec 30, 2025
1,010 words in the original blog post.
On June 12, 2025, a global outage on Google Cloud Platform (GCP) caused by an automated quota update disrupted many critical internet services, but Redpanda Cloud customers remained unaffected due to the company's robust cell-based architecture and service design. Despite the chaos affecting several prominent companies, Redpanda's internal systems, leveraging a self-managed observability stack and redundant service architecture, continued operation without major issues, highlighting the effectiveness of their safety and reliability practices. The incident underscored the challenges of managing complex systems, as the non-linear nature of these systems often leads to unpredictable outcomes, a concept akin to the butterfly effect in chaos theory. While Redpanda's cloud clusters were prepared for high availability with features like a replication factor of at least three and local NVMe disk storage, the company acknowledged the role of luck in emerging largely unscathed, with only one non-production cluster in the us-central-1 region affected. The incident emphasized the importance of systems thinking and control theory in managing socio-technical systems, particularly as technology continues to evolve with the rise of AI, suggesting a need for the industry to refine these skills in the absence of AI replacements.
Dec 30, 2025
1,618 words in the original blog post.
Setting up a streaming application with platforms like Redpanda requires integration testing to ensure correct configuration, and two popular Java-based libraries for this purpose are Testcontainers and Zerocode. Testcontainers allows testing within Docker containers, making it suitable for simulating data streams using Redpanda nodes, while Zerocode offers a declarative testing approach through JSON/YAML scenarios, suitable for end-to-end testing of data streams. Both tools enable testing the functionality of producers and consumers, with Testcontainers requiring a Java setup and Docker knowledge, while Zerocode relies on scenario files and configurations. Successful integration testing confirms that data streams are properly configured, and any errors in the setup will be visible through test logs.
Dec 30, 2025
1,731 words in the original blog post.
Companies are increasingly transitioning from Amazon Managed Streaming for Apache Kafka (MSK) to Redpanda to achieve better performance and cost-efficiency in handling streaming data. Despite MSK's hands-off appeal, some users find its performance and cost lacking, prompting a switch to Redpanda, which offers a seamless, API-compatible replacement with minimal disruption. The migration process from MSK to Redpanda involves setting up a Redpanda cluster, configuring MirrorMaker2 for data replication, and carefully managing consumer offsets to ensure continuity. The MirrorMaker2 client machine requires specific AWS Identity and Access Management (IAM) configurations to authenticate with MSK, and careful consideration must be given to resource allocation to handle the data flow efficiently. The migration aims to minimize application pipeline disruptions by requiring only minor configuration changes, allowing producers and consumers to continue operations with minimal downtime. Asynchronous replication with MirrorMaker2 is essential, and users must ensure adequate network and processing capacity to handle high-volume data transfer between platforms. Redpanda provides resources and community support to assist users in transitioning smoothly from MSK, ensuring they benefit from a more efficient and cost-effective Kafka platform.
Dec 30, 2025
2,191 words in the original blog post.
Redpanda, an Apache Kafka-compatible event streaming platform, has integrated with Bytewax to streamline real-time data processing, leveraging Bytewax's Python API and Redpanda's high-throughput capabilities. This integration is enhanced by the use of Redpanda's Schema Registry, which is pivotal for managing schema and ensuring data integrity in streaming environments. The Schema Registry centralizes schema management, facilitating seamless message serialization/deserialization and schema evolution without disrupting data pipelines. This collaboration allows developers to efficiently build and deploy scalable, high-performance data processing pipelines with reduced risk of data inconsistencies. The integration offers a simplified interface, making complex data processing and streaming tasks more accessible without requiring a steep learning curve. Through this partnership, Python developers can capitalize on the combined strengths of Bytewax and Redpanda to create robust applications, supported by active community engagement and resources for further exploration.
Dec 30, 2025
665 words in the original blog post.
Streaming data, characterized by its continuous flow from multiple sources, is effectively managed using time series databases, which are optimized for such data types. TimescaleDB, an extension of PostgreSQL, is celebrated for its efficient time-series query performance, supporting multi-node deployment and data chunking for better performance. The integration of Redpanda and TimescaleDB through Kafka Connect exemplifies how to build a system for analyzing real-time data streams, such as payment systems, to detect anomalies like fraud. The tutorial outlines a step-by-step process for setting up Redpanda, TimescaleDB, and Kafka Connect using Docker, showcasing a practical application for streaming time series data into a PostgreSQL-based environment for real-time analysis. This setup facilitates the creation of real-time applications across various domains, leveraging the compatibility of Redpanda with Apache Kafka API and the PostgreSQL nature of TimescaleDB to provide diverse connection options.
Dec 30, 2025
2,708 words in the original blog post.
Redpanda's recent developments position it as a comprehensive, end-to-end data streaming platform, offering rapid deployment capabilities and extensive connectivity options through Redpanda Connect. The platform supports a wide range of form factors and scales, from microsecond-level latency workloads to large-scale social networks and banking applications. Redpanda's acquisition of the Benthos stream processing framework, driven by Ash, its founder, enhances its offerings by integrating Benthos Studio into the Redpanda Console for better visualization of business connectivity. The platform supports an extensive list of open-source connectors under the Apache 2 license, ensuring reliability and scalability, with certified connectors receiving additional support. Two enterprise connectors, snowflake_put and splunk, require a subscription, reflecting Redpanda's monetization strategy. Redpanda Benthos' core engine remains MIT Licensed, supporting the community that relies on its robust features, while Redpanda emphasizes core values of customer obsession, teamwork, and human-centric growth.
Dec 30, 2025
2,112 words in the original blog post.
Setting up TLS on Redpanda and rpk involves acquiring a certificate and a key, with an optional truststore file for client authentication. Redpanda supports two levels of TLS: basic encryption of incoming requests and an enhanced version requiring client certificates for authentication. This configuration applies to the Apache Kafka®-compatible API, the Redpanda Admin API, and the internal RPC API. The process includes specifying listener addresses, configuring TLS settings for each API, and potentially using different certificates for added security. The rpk tool can be used to configure TLS settings, and changes to its configuration do not necessitate restarting Redpanda. Since Redpanda is API-compatible with Kafka, existing Kafka clients may not require significant changes unless file locations or formats are altered. TLS ensures encrypted communication and can enforce client authentication to restrict access to known clients, enhancing overall security.
Dec 30, 2025
1,186 words in the original blog post.
Redpanda, a high-performance streaming data platform compatible with Apache Kafka, offers a seamless integration with Google BigQuery through a new Redpanda Cloud connector, enabling automated and fully-managed data integration for advanced analytics. By channeling real-time streaming data into BigQuery, users can perform comprehensive analytics, including machine learning-based predictive analytics, on petabyte-scale data. The integration simplifies data streaming processes, allowing users to connect various data systems without needing to write code, while Redpanda's schema management ensures data consistency. An example scenario demonstrates how to identify user interactions with classified ads by streaming data through Redpanda into BigQuery, where SQL queries can be executed for data analysis. This integration empowers organizations to harness the full potential of their data, enhancing decision-making through real-time insights and robust analytics capabilities.
Dec 30, 2025
1,451 words in the original blog post.
Many engineering teams initially turned to NATS for real-time messaging due to its simplicity and speed, but as organizations scale, the limitations of NATS as a data backbone become apparent. With the introduction of JetStream, NATS enhanced its capabilities to include persistence and message replay, yet the demand for durable, scalable event streaming systems led to the emergence of platforms like Redpanda. Redpanda, compatible with Apache Kafka APIs, serves as a high-performance event log that supports event-driven applications, data pipelines, and business-critical workflows. It offers features like infinite retention, exactly-once delivery, and extensive data integration, positioning it as a strategic data platform. This shift from simple messaging to robust streaming supports real-time personalization, predictive analytics, and data-driven decisions. Migrating from NATS to Redpanda involves re-architecting around event streaming principles and leveraging Redpanda Connect for seamless integration, ultimately enhancing infrastructure alignment with business outcomes and future-proofing data capabilities.
Dec 30, 2025
1,116 words in the original blog post.
In the modern data landscape, the demand for real-time streaming data processing has surged due to the proliferation of IoT devices, mobile phones, and e-commerce, extending its necessity beyond traditional high-frequency domains like stock markets and airline systems. Stream processing patterns, essential for efficient real-time data handling, utilize resources effectively and are key to minimizing costly delays in data processing. These patterns are implemented across various technologies like Apache Kafka, Flink, and Spark, and include techniques such as filtering, stream-table joins, windowing joins, preprocessing, and aggregation. Each pattern offers distinct advantages, such as filtering irrelevant data, joining streaming data with reference tables, or preprocessing to transform data for further analysis. These methods empower industries ranging from finance to transportation to derive insights promptly and respond swiftly to dynamic data streams. Additionally, platforms like Redpanda provide streamlined, reliable solutions for managing streaming data by simplifying complex underlying technologies and offering community support for users.
Dec 30, 2025
1,657 words in the original blog post.
Redpanda 25.3 introduces several updates to address challenges in real-time systems related to risk, cost, and complexity. Key features include Shadowing, which enhances disaster recovery by enabling a near-instantaneous failover to a replica cluster, reducing downtime and ensuring business continuity. Cloud Topics optimize cost efficiency for latency-tolerant data by directing it to object storage while maintaining high performance for latency-critical data. The integration with Google BigLake enables seamless governance and analytics of streaming data in the GCP ecosystem. Additionally, the Microsoft SQL Server CDC input in Redpanda Connect facilitates real-time data integration from legacy SQL Server databases, allowing enterprises to modernize their data infrastructure without disruptive migrations. Overall, these updates enhance Redpanda's capabilities as a resilient, efficient, and interoperable data streaming platform suitable for modern enterprise needs.
Dec 30, 2025
1,556 words in the original blog post.
The text is a detailed technical guide on integrating TensorFlow, an open-source machine learning platform, with Apache Kafka using Redpanda, an alternative to Kafka, for seamless data streaming and processing. The author explains the process of installing necessary software and dependencies on Ubuntu, preparing and partitioning data using pandas, and utilizing TensorFlow IO to read and write data to Kafka topics. It includes instructions on setting up a machine learning model to process this data and evaluate its performance using TensorFlow's capabilities. The guide demonstrates the practical application of streaming data for machine learning tasks, highlighting the use of Python libraries such as sklearn and pandas for data handling, and emphasizes the successful initial trial of the integration with Redpanda. The author also shares personal experiences, including the installation of CUDA libraries for GPU support and the performance of a sample machine learning model on the data streamed from Kafka.
Dec 30, 2025
2,128 words in the original blog post.
This blog post delves into deploying Redpanda for high availability (HA) within a single availability zone (AZ), emphasizing the importance of eliminating single points of failure by spreading replicas across multiple failure domains or racks. It examines the pros and cons of single AZ deployments compared to multi-AZ deployments, noting that while single AZ setups avoid cross-AZ network charges, they rely on robust local infrastructure to prevent failures. The post also explains how Redpanda's rack awareness and the Raft consensus algorithm facilitate efficient HA deployments without significant cost or complexity increases. It outlines infrastructure considerations and describes how to automate deployments using Terraform and Ansible across AWS, Azure, and Google Cloud, with each cloud provider offering specific tools for spreading hosts across failure domains. Additionally, it provides a practical example of setting up a highly available Redpanda cluster, highlighting the configuration of rack awareness and the use of automation scripts to ensure seamless deployment. The series will continue by addressing HA clusters across multiple availability zones and regions.
Dec 30, 2025
1,402 words in the original blog post.
Redpanda's emphasis on observability enabled them to address a significant performance issue for Chess.com, a major online chess platform with over 200 million members and more than ten million games played daily. Chess.com faced excessive CPU usage due to inefficient compaction on one of their largest data streams, which threatened their system's stability. By leveraging observability tools, Redpanda identified the inefficiencies and implemented a seamless upgrade to Redpanda version 25.1, featuring smarter compaction controls. This upgrade resulted in a dramatic reduction in CPU usage, halved latency, and improved system stability, ultimately enhancing user experience. Redpanda's approach highlights their commitment to continuous product improvement and real-world customer validation through observability.
Dec 30, 2025
562 words in the original blog post.
Financial Services Industry (FSI) organizations encounter numerous obstacles in their pursuit of AI-driven transformation, primarily due to legacy systems, stringent regulations, and data silos, which create a chaotic data environment. To address these challenges, the text explores the integration of legacy systems, regulatory navigation, and data silo dismantling as crucial steps toward fostering data-driven decision-making. It emphasizes the adoption of data streaming technologies to revamp data pipelines, enabling real-time data ingestion and seamless system integration. The text also delves into various data pipeline types—batch, micro-batch, and real-time—highlighting their respective uses and challenges, such as latency issues and scalability. Additionally, it outlines the importance of data quality and infrastructure in supporting AI and machine learning applications, advocating for a phased approach to resolving data complexities. The upcoming second part promises to provide a detailed data strategy for streamlining pipelines and achieving efficient data integration and processing.
Dec 30, 2025
2,434 words in the original blog post.
Real-time data streaming has become essential for modern businesses to respond quickly to market changes and improve decision-making processes, which is vital in industries such as e-commerce and security monitoring. The tutorial illustrates how to build a real-time streaming data analytics system using Redpanda and Rockset. It guides users through setting up a streaming data pipeline for an on-demand delivery service, starting from data generation in a Golang application to streaming with Redpanda, transporting data via Kafka Connect, and conducting real-time analysis with Rockset. It details the configuration steps needed to integrate these technologies, create a Rockset collection using its RESTful API, and query the data using SQL to analyze delivery requests and revenue by state. The tutorial emphasizes the benefits of real-time data processing, enabling businesses to allocate resources efficiently and respond promptly to events, showcasing the integration's potential with further exploration encouraged through available resources and community support.
Dec 30, 2025
2,656 words in the original blog post.
Redpanda's latest update for its cloud platform emphasizes foundational improvements that enhance scalability, security, and operational efficiency, featuring the general availability of the Cloud API and Terraform provider, which streamline automation and infrastructure management. The Cloud API, now fine-tuned through customer feedback, offers programmatic control over streaming data, eliminating the need for manual configurations and supporting integration with tools like Terraform for repeatable workflows. The update also doubles partition density, optimizing resource usage and reducing costs, while enhanced role-based access control (RBAC) aligns access management with organizational needs. Additionally, Redpanda Connect's new certifications for BigQuery and BYOVPC environments on AWS and GCP facilitate seamless data streaming and integration. These enhancements aim to empower teams with robust, scalable, and secure cloud solutions, enabling them to focus on innovation without compromising on compliance or performance.
Dec 30, 2025
1,641 words in the original blog post.
Transparent HugePage (THP) is a Linux kernel feature designed to simplify the use of huge pages, providing a performance boost by automatically managing memory page sizes. Unlike HugeTLB, which offers greater control and performance but requires more manual configuration, THP aims for ease of adoption and transparency, making it more accessible to a wider range of applications. THP can automatically promote standard pages to huge pages and demote them when necessary, although it may require specific configurations such as using the madvise syscall to optimize its effectiveness. The system-wide settings for THP, such as enabling or defragmentation behavior, can be adjusted to suit different application needs, and while it does not reserve huge pages in advance, it can manage them dynamically through processes like memory compaction. Despite its advantages, THP has limitations, such as only supporting a fixed huge page size and potential performance drawbacks in certain scenarios. The ongoing evolution of THP reflects its importance in memory management within the Linux ecosystem, highlighting the balance between ease of use and performance optimization.
Dec 30, 2025
3,469 words in the original blog post.
Redpanda version 22.1 introduces several enhancements to its streaming data platform, focusing on operational simplicity, reliability, and Apache Kafka compatibility. The release marks the general availability of Transparent Tiered Storage, which leverages Redpanda's technology to provide seamless access to both real-time and historical data while optimizing costs and performance. Centralized Configuration simplifies cluster administration by allowing cluster-level properties to be managed centrally, reducing errors and enhancing automation capabilities. Maintenance Mode ensures uninterrupted operation during system updates by transferring partition leadership and preventing service disruption. Rack Awareness improves data protection by distributing data replicas across different failure zones, thereby minimizing the risk of data loss. Additionally, Consumer Offsets support enhances integration with the Kafka ecosystem, while Idempotent Producers increase communication reliability by eliminating the impact of duplicate messages. The update is aligned with Redpanda's new versioning system, which emphasizes a balance between feature development and stability.
Dec 30, 2025
1,271 words in the original blog post.
Leveraging Timely Dataflow and Redpanda, building real-time, scalable, fault-tolerant data processing systems has become more accessible. Timely Dataflow, a low-latency cyclic dataflow computational model, enables the construction of data-parallel systems that can scale from a single thread to a cluster, making it ideal for real-time anomaly detection. Redpanda, compatible with Apache Kafka APIs, facilitates real-time data monitoring across multiple sources. By integrating Bytewax, a Python binding for Rust-based Timely Dataflow, with Redpanda, developers can efficiently create applications in Python. The process involves Bytewax reading sensor data from a Redpanda topic, calculating anomalies using a five-second data aggregation window, and then outputting results to another topic. This setup involves generating mock air quality data, processing it with an online anomaly detection algorithm, and ensuring the system's real-time capabilities via Docker and Redpanda's API. The combination of Redpanda's Kafka-compatibility and Bytewax's performance provides a robust solution for both small-scale projects and extensive production environments.
Dec 30, 2025
2,576 words in the original blog post.
Redpanda's FY26 company kick-off (CKO) event in Arizona served as a vibrant gathering for team-building, strategizing, and celebrating accomplishments, with a mix of in-person and virtual participation. The event featured keynote addresses from company leaders, including Founding Panda Alex Gallego, who outlined Redpanda's strategic vision and highlighted recent achievements like Redpanda Connect and Redpanda Serverless. Customer success stories were shared, emphasizing the platform's efficiency and reliability, with companies like Johnson Controls and LiveRamp showcasing significant improvements in cost and performance. The event included activities like the Redpanda Rodeo, which emphasized collaboration and teamwork, and allowed attendees to engage in social activities, fostering personal connections. An independent board member, Simon Parmett, highlighted the company's unique culture of integrity and humility. The CKO encapsulated Redpanda's commitment to innovation and teamwork, leaving participants with renewed motivation and a clear path forward as the company continues to make strides in data streaming.
Dec 30, 2025
1,504 words in the original blog post.
Becoming a Product Manager (PM) offers a unique opportunity to work at the core of a company, exemplified by Towfiqa Yasmeen's experience at Redpanda, a startup recognized by Forbes as a potential billion-dollar company. The role of a PM at Redpanda involves diverse responsibilities, including product discovery, market analysis, and cross-functional collaboration to ensure successful product development and execution. Yasmeen highlights the importance of understanding customer needs and aligning organizational goals, while also emphasizing the significance of teamwork and effective communication during the Go-To-Market phase. At Redpanda, PMs are valued for their expertise and encouraged to innovate within a culture of respect and trust, where they can make meaningful contributions and see the tangible impact of their work. The company’s emphasis on empowering individuals to take ownership and make bold decisions fosters an environment ripe with new possibilities and innovation, making it an attractive choice for those seeking a dynamic and fulfilling career in product management.
Dec 30, 2025
870 words in the original blog post.
Redpanda is a streaming data platform compatible with Apache Kafka, designed for high-performance, data safety, and transactional workloads, making it easy to integrate with existing infrastructures like Apache Spark. Spark, an analytics engine developed to overcome the limitations of the MapReduce algorithm, supports iterative functions on Resilient Distributed Datasets (RDDs) and has evolved to include Spark SQL, MLLib, and Spark Streaming, which allows for integration with live data streams from sources such as Redpanda. Spark Streaming processes streams in micro-batches using the Spark engine, while Structured Streaming extends the Spark SQL API to handle streams similarly to static RDDs, making it straightforward to read messages from a Redpanda topic by creating a SparkSession and a streaming DataFrame. The integration allows for building complex data processing pipelines with high performance, scalability, and durability. Moreover, Redpanda's Wasm Data Transforms offer a way to perform simple transformations directly within Redpanda using WebAssembly functions, potentially replacing common Spark functions like map() or filter(), and are ideal for tasks like data redaction for GDPR compliance. For more complex processing involving aggregations or joining streams, tools like Spark and Apache Flink are recommended as supplements to Wasm Transforms.
Dec 30, 2025
1,214 words in the original blog post.
Pathway has developed a solution for processing streaming data in real time using small data sets, leveraging both Apache Kafka and Redpanda as data streaming platforms. The example provided involves creating a system to identify the top-rated movies for a video-on-demand service by processing ratings received through a Kafka stream and outputting the results to a CSV file. Pathway's approach allows for seamless integration and switching between Kafka and Redpanda, as Redpanda serves as a drop-in replacement for Kafka with the advantage of being simpler, faster, and more cost-effective. The practical application illustrates how Pathway can efficiently process data streams and update rankings dynamically. By using Pathway's connectors, which work identically for both Kafka and Redpanda, developers can maintain their existing codebase while taking advantage of Redpanda's efficiencies. Additionally, the text highlights the potential of combining machine learning with real-time data for immediate, actionable insights, such as dynamically mapping geofences or detecting sensor anomalies without predefined parameters.
Dec 30, 2025
2,701 words in the original blog post.
The tutorial provides a comprehensive guide to building a real-time inventory monitoring system using modern data technologies like MongoDB, Redpanda, and Apache Flink. It likens inventory management to running a corner store, emphasizing the importance of real-time updates to avoid stockouts and manage expiration dates. The system leverages change data capture (CDC) in MongoDB to track inventory changes, with Redpanda facilitating data streaming and Apache Flink processing the data streams. The setup involves configuring a development environment with tools like Docker and Maven, establishing connections between MongoDB and Redpanda using Kafka Connect, and executing an Apache Flink job to process and log data from inventory changes. The tutorial walks through each step, including setting up Docker networks, configuring Kafka topics, and writing Flink applications, providing practical commands and code snippets. This setup allows for automated inventory management, including reordering and managing discounts for near-expiry items, thereby enhancing inventory accuracy and operational efficiency.
Dec 30, 2025
2,224 words in the original blog post.
Redpanda is an advanced queue system designed to operate near the hardware's physical limits, emphasizing autotuning to optimize performance and reduce costs associated with storage systems, which traditionally require significant personnel efforts for tuning. The architecture employs a shared-nothing model with structured message passing, ensuring efficient partitioned data handling and eliminating locking on critical paths. To further enhance performance, Redpanda utilizes Direct Memory Access and sparse filesystem support for disk operations, minimizing latency and maximizing throughput. The Redpanda Keeper (RPK) tool automates system tuning, addressing network, disk, and CPU configurations, simplifying the complex task of performance optimization. By leveraging Symmetric Multiprocessing affinity, Redpanda effectively manages interrupts to improve I/O throughput and reduce latency, offering a more predictable and scalable performance model for large-scale systems.
Dec 30, 2025
1,254 words in the original blog post.
Observability is crucial for understanding and analyzing the internal state of systems, particularly in modern distributed environments, and is maintained through tools like Prometheus and Grafana. These tools help monitor Redpanda clusters, producers, and consumers by exporting and visualizing metrics, ensuring system health and performance optimization. Redpanda's observability setup involves predefined dashboards for monitoring key performance metrics, such as Kafka topic metrics and consumer offsets, and uses Docker and Docker Compose for seamless integration. By setting up a Redpanda cluster with Prometheus and Grafana, developers can gain real-time insights into system operations, facilitating efficient troubleshooting and performance management. The observability infrastructure supports both local and cloud deployments, offering flexibility for developers and site reliability engineers to maintain effective system operations.
Dec 30, 2025
1,015 words in the original blog post.
Advancements in storage systems have shifted the bottleneck from storage devices to CPU performance, as SSDs and modern networks have outpaced traditional spinning disks and previous network capabilities. Despite improvements, single-core CPU speeds remain limited due to power consumption constraints, leading to a focus on increasing core counts for better performance. To adapt to these changes, systems like Redpanda, a Kafka-API compatible platform, utilize a thread-per-core architecture to optimize resource use and minimize latency. This approach involves explicit core communication and asynchronous execution models, necessitating efficient memory management techniques like the iobuf framework. Iobuf, inspired by similar abstractions in Linux and FreeBSD kernels, enables zero-copy memory management by leveraging Seastar's threading model and SPSC queues, ensuring scalability and low latency in data-intensive environments. This architecture promotes synchronous task execution across multiple cores, offering a significant performance boost for mission-critical workloads that require low-latency operations.
Dec 30, 2025
1,612 words in the original blog post.
ShareChat, India's largest social media company, transitioned its streaming data infrastructure from Google Pub/Sub to Redpanda Cloud, leading to a 70% reduction in cloud infrastructure costs and operational complexity. The shift to Redpanda's managed cloud solution, with its flexible and efficient architecture, enabled ShareChat to deploy, operate, and maintain their system more effectively while ensuring data ownership and sovereignty. The incorporation of features like Tiered Storage, follower fetching, and optimized cloud instance profiles further enhanced cost savings and performance, allowing ShareChat to scale seamlessly during high-traffic events without impacting user experience. The transition also facilitated ShareChat's compliance with data protection regulations, thanks to Redpanda's BYOC offering, which maintained data hosting within ShareChat's cloud environment. This evolution not only benefited ShareChat but also led to broader enhancements in Redpanda Cloud's offerings, making it more robust for large-scale enterprises.
Dec 30, 2025
1,134 words in the original blog post.
Lacework, a leader in cloud security services, has transitioned to using Redpanda, an Apache Kafka-compatible streaming data platform, to enhance its application security services across the entire lifecycle. This transition was driven by the need for a scalable, reliable, and efficient solution to manage their rapidly growing 1GBps+ data streaming workloads, which their previous proprietary solution could not handle. Redpanda, with its high-performance architecture and ease of deployment, enables Lacework to manage data spikes and ensure high availability and disaster recovery across multiple clusters. This architecture, dubbed "pipe dream," allows for seamless topic migration between clusters, improving resource utilization and operational agility, while maintaining strict service level objectives. Lacework plans to leverage Redpanda's Tiered Storage to reduce storage costs and is preparing to migrate to a managed Kubernetes environment with Amazon EKS in 2023, underscoring Redpanda's flexibility and alignment with modern infrastructure needs.
Dec 30, 2025
1,412 words in the original blog post.
Redpanda has announced its Series A Financing aimed at expanding its cloud service, which focuses on transforming data streams into data products through the Intelligent Data API. This API is designed to address the challenges of operationalizing open-source components at scale by offering compatibility with the widely used Apache Kafka API, unifying historical and real-time data through tiered storage, and enabling stateful stored procedures for streaming data using WebAssembly. The platform aims to simplify enterprise demands for real-time data processing by providing a reliable and high-performance alternative to traditional systems, integrating with scalable storage solutions like Amazon S3, and allowing developers to perform in-situ transformations on data streams. While the Kafka API functionality is production-ready, features like Tiered Storage and WebAssembly are still experimental, and the company invites community involvement and collaboration through open development on GitHub and recruitment opportunities.
Dec 30, 2025
1,272 words in the original blog post.
A schema serves as a logical description of data organization, facilitating data understanding, API conformance verification, and API evolution with compatibility levels. A schema registry provides a central repository for schemas, simplifying data exchanges between producers and consumers by maintaining a history of schemas and supporting schema evolution strategies. In the context of Apache Kafka, schemas are crucial for message serialization and deserialization, allowing serializers and deserializers to interpret data formats like Apache Avro, JSON Schema, and Google's Protobuf. Redpanda offers an integrated schema registry within its Kafka-compatible streaming platform, eliminating the need for separate installations and simplifying operations by embedding the registry into its Raft-based storage engine. This integration allows users to leverage existing Kafka serializers and deserializers without code changes, enhancing performance and reducing cloud costs.
Dec 30, 2025
1,702 words in the original blog post.
Real-time analytics and streaming data infrastructure are becoming essential for modern enterprises, but have traditionally been complex and costly to implement. The blog highlights how Timeplus and Redpanda offer a more accessible streaming-first approach to real-time analytics, with Redpanda providing a fast, scalable, and reliable platform for managing streaming data, and Timeplus offering a powerful and intuitive analytics platform for transforming this data into actionable insights. Timeplus allows enterprises to implement real-time analytics with minimal latency, achieving 4ms end-to-end latency, and can handle over 10 million events per second on a single machine. The combination of Redpanda and Timeplus offers seamless integration, with Redpanda acting as both an upstream source and downstream sink for data, allowing Timeplus to directly query streaming data. Performance tests show that Redpanda with Timeplus outperforms Apache Kafka in terms of latency, making it particularly effective for large data batches. Their collaboration facilitates new use cases across industries like manufacturing, financial services, and e-commerce, providing faster and more cost-effective real-time capabilities.
Dec 30, 2025
1,444 words in the original blog post.
This introductory guide demonstrates how to create a simple integration using Apache Camel to stream real-time air quality data through a Redpanda streaming data platform, leveraging the JBang tool for ease of development. Redpanda, a Kafka-compatible platform, offers increased speed and reduced costs compared to Kafka, while Apache Camel simplifies connectivity and integration with its established enterprise patterns. The tutorial outlines setting up a Camel route to pull data from an API endpoint every 30 seconds, using Enterprise Integration Patterns like Splitter and WireTap to process and route data to separate topics for scalability. By utilizing JBang, developers can quickly run Camel routes as scripts without the need for extensive project setup, and Redpanda's Kafka API compatibility facilitates seamless integration with Camel. The guide also covers running Redpanda locally using Docker, demonstrating the streaming and monitoring capabilities through the Redpanda Console, and concludes with instructions to export the Camel route into a Quarkus-based Java project.
Dec 30, 2025
1,099 words in the original blog post.
Redpanda has introduced fully-managed connectors in Redpanda Cloud to simplify the integration of various data systems with real-time clusters, eliminating the need for manual setups using the Kafka API or Kafka Connect. These connectors, available in the Redpanda Console, include new offerings for MongoDB, MySQL, PostgreSQL, SQLite, and SQL Server, supporting both source and sink operations, along with an HTTP source connector for importing data from web services. The MongoDB connectors leverage MongoDB's oplog for real-time data changes, while the JDBC connectors facilitate interaction with relational databases, supporting tasks like database synchronization and audit logging. The HTTP source connector allows easy data import from HTTP endpoints, enhancing Redpanda's capability to handle change data capture from JSON/HTTP APIs. By using these connectors, developers can more efficiently build real-time and event-driven applications, benefiting from step-by-step guidance and configuration options within the Redpanda UI to optimize task allocation and performance.
Dec 30, 2025
850 words in the original blog post.
Redpanda is a modern streaming data platform built on the Apache Kafka protocol, designed to handle real-time data streams with exceptional performance and scalability. It provides a simpler, more efficient alternative to Kafka by utilizing a thread-per-core architecture and deploying as a single binary without the need for external dependencies like JVM or Apache ZooKeeper. This makes Redpanda particularly beneficial for latency-sensitive applications in areas such as financial trading and IoT devices. By offering drop-in compatibility with the Kafka ecosystem, Redpanda allows developers to produce and consume data using Kafka APIs seamlessly. It supports high-volume data ingestion, fault-tolerant storage, and asynchronous message exchange, serving diverse user personas including developers, data engineers, operators, and architects. Redpanda's developer-friendly tools, such as the Redpanda Console and Redpanda Keeper CLI, facilitate cluster management and monitoring, simplifying the deployment and maintenance of scalable streaming applications.
Dec 30, 2025
1,521 words in the original blog post.
Webhooks, or user-defined callbacks, offer an efficient way to enable real-time communication between systems over the Internet, as opposed to traditional API polling methods. This tutorial outlines the process of building a webhook delivery system using Redpanda to create fault-tolerant applications capable of seamlessly communicating with external services. It explains how to set up a webhook dispatcher that manages the delivery of webhooks to external clients, covering aspects such as event acknowledgment, error handling, and the creation of HTTP endpoints for queuing webhooks. The tutorial details the use of Redpanda for persisting and processing events, ensuring that failed event processing can be retried, and it provides guidance on setting up the necessary infrastructure using Docker and AdonisJS. Additionally, it discusses the creation of consumers to handle events from Redpanda and the use of Axios for sending webhooks to specified URLs, while suggesting further enhancements such as implementing retry policies, supporting various HTTP methods, and handling different payload content types.
Dec 30, 2025
3,129 words in the original blog post.
Kafka web interfaces provide a user-friendly alternative to command line tools for managing Kafka clusters, offering advantages in convenience, mobile access, and observability. These interfaces, including UI for Apache Kafka, Conduktor, and Redpanda Console, vary in their offerings, with each having unique strengths. UI for Apache Kafka is lightweight and straightforward, suitable for novice users, while Conduktor offers robust enterprise features and strong security options, making it ideal for large businesses. Redpanda Console stands out for its comprehensive feature set and excellent user experience, supporting extensive manageability and observability capabilities. However, each tool has drawbacks, such as limited maintenance features or observability components. Although Kafka Tools is not a web UI, it is included in comparisons to highlight the differences and possible integrations between graphical interfaces and traditional CLI approaches. Ultimately, choosing the right Kafka management tool depends on specific needs, including usability, security, and operational requirements.
Dec 30, 2025
3,055 words in the original blog post.
In the rapidly evolving landscape of big data, understanding the distinct roles of databases, data warehouses, and data lakes is crucial. Databases, which are structured collections optimized for quick read and write transactions, serve as a reliable foundation for storing information pertinent to individual applications or organizations. Data warehouses, on the other hand, aggregate structured data from multiple sources for large-scale analysis, making them ideal for business intelligence and reporting. Data lakes offer a more flexible approach by storing raw, unstructured data, thus allowing the accommodation of diverse data types and supporting complex analytics and machine learning applications. Each type of data store has its unique set of features, use cases, and popular tools, such as MySQL, Oracle, MongoDB, and PostgreSQL for databases; Amazon Redshift, Google BigQuery, and Snowflake for data warehouses; and Google Cloud Storage, Azure Data Lake Storage, and Amazon S3 for data lakes. The article suggests that these systems can be used in tandem to maximize data utility, with data lakes serving as a cost-effective repository for raw data, which can then be selectively moved to data warehouses for analysis. It also highlights Redpanda, a data streaming platform, which facilitates seamless data transfer between these stores and integrates with event-driven setups, enhancing the overall efficiency of data management.
Dec 30, 2025
2,891 words in the original blog post.
The text delves into the development of a streaming engine for modern applications, emphasizing the use of Raft-based protocols in Redpanda compared to Apache Kafka's synchronous replication method. The analysis highlights the protocol's strengths and weaknesses, particularly under different fault conditions, by simulating various disturbances such as leader and follower terminations and disk latency increases. It found that while both Redpanda and Kafka experience availability losses when a leader is affected, only Kafka is significantly impacted when a follower faces issues due to its reliance on synchronous replication. The discussion suggests that Raft-based systems like Redpanda can benefit from the wealth of research and optimizations developed for Paxos-based systems, particularly in areas like disk latency management and leader election. The text concludes by expressing enthusiasm for further exploration and optimization of these protocols to enhance performance and reliability.
Dec 30, 2025
1,003 words in the original blog post.
Real-time streaming data is transforming the way organizations utilize data warehouses by enabling immediate data processing and analysis as it is generated, contrasting with traditional batch processing methods that introduce delays. This approach allows businesses to respond swiftly to changing scenarios and is particularly beneficial in time-sensitive environments such as financial fraud detection and stock market trading. Implementing real-time streaming data within a data warehouse offers advantages like real-time processing, scalability, flexibility, and enhanced security, although it presents challenges such as high costs, technical knowledge gaps, and the need for organizational changes. Modernizing a data warehouse to incorporate real-time streaming involves upgrading storage systems, transitioning from traditional ETL processes to Streaming ETL, integrating real-time and historical data, and enhancing analytics capabilities. Redpanda is highlighted as a platform that simplifies the creation of real-time streaming infrastructure, offering a streamlined deployment process without the need for additional components, and providing resources for organizations to facilitate the adoption of real-time data streaming.
Dec 30, 2025
1,837 words in the original blog post.
Streaming data, stream processing, and real-time analytics are interconnected yet distinct concepts crucial to modern data-driven applications, such as online payments and social media updates. Streaming data refers to the continuous flow of information from various sources, which requires a robust system to manage its volume and speed, leading to the necessity of stream processing. Stream processing acts as the engine that organizes and interprets these data streams quickly and efficiently, ensuring that the data is ready for further analysis. Real-time analytics then extracts actionable insights from the processed data, enabling businesses to make rapid decisions that can enhance customer experiences, optimize operations, and prevent issues like fraud. Together, these components form a seamless pipeline where streaming data is the raw input, stream processing is the mechanism that handles it, and real-time analytics is the tool that provides value by allowing businesses to respond to events as they occur.
Dec 30, 2025
1,735 words in the original blog post.
In the digital era, real-time data processing has become crucial for delivering interactive user experiences and facilitating decision-making in distributed applications like microservices. This necessity has led to the widespread adoption of stream processing technologies such as Apache Spark, Apache Flink, and ksqlDB, each offering unique capabilities for handling large-scale data streams. Apache Spark, known for its fast in-memory processing, supports multiple languages and integrates well with technologies like Redpanda, though it demands high memory consumption and has a steep learning curve. Apache Flink, designed natively for stream processing, is praised for its low latency and easy-to-use features but lacks the community support that Spark enjoys. Meanwhile, ksqlDB leverages Apache Kafka's infrastructure to provide a simple SQL interface for stream processing, excelling in Kafka integration but falling short in analytics capabilities compared to Spark and Flink. All three tools can be used with Redpanda to perform tasks such as real-time analysis and fraud detection, allowing companies to choose based on specific needs and use cases.
Dec 30, 2025
2,176 words in the original blog post.
Securing millions of embedded IoT devices across diverse architectures presents significant challenges, which Exein addresses through its enterprise solution, Exein Runtime, and its open-source observability framework, Pulsar. This framework, powered by Rust and eBPF, enables real-time device monitoring and threat detection, which are sent to a central platform for action. The company overcame challenges related to real-time streaming in an IoT ecosystem by partnering with Redpanda, whose cloud-managed CQRS pattern supports independent scaling of services, simplifies deployment and maintenance, and enhances platform performance, handling over 39,000 events per second. Redpanda's infrastructure also allows seamless scalability and production operation ease, making it integral to Exein's system. As Exein expands, it plans to leverage Redpanda's advanced features and contribute to the Redpanda Community, continuing to enhance the platform's resilience and scalability.
Dec 30, 2025
941 words in the original blog post.
Change data capture (CDC) is a process that monitors and replicates changes such as inserts, updates, and deletes in real-time or near-real-time across databases and downstream systems. It is especially beneficial for synchronizing multiple databases, enhancing cache or search index performance, real-time logging, and implementing event-driven architectures like command query responsibility segregation (CQRS). The text details a tutorial on establishing a real-time CDC pipeline using Postgres, Google Cloud Storage, Redpanda, and Debezium. The process involves setting up a Postgres instance with logical replication, configuring a Debezium connector to monitor database changes, and creating a Kafka Connect container to link Postgres and Redpanda. Additionally, a Google Cloud Storage (GCS) connector is created to store the data from Redpanda into a GCS bucket, ensuring efficient data movement and storage. The tutorial provides step-by-step guidance on setting up the necessary software and configurations, showcasing how CDC can reliably maintain synchronized data across different systems and use cases.
Dec 30, 2025
2,117 words in the original blog post.
The second day of Redpanda Streamfest 2025 focused on enhancing data streaming skills for real-time applications, analytics, and AI, featuring a series of engaging sessions, product demos, and panels. Key highlights included Redpanda's introduction of Cloud Topics for cost-effective cloud streaming, integration with Apache Iceberg for efficient data management, and Shadowing for disaster recovery. The event also showcased Redpanda Connect's ability to capture real-time changes from Microsoft SQL Server, and the flexibility of Redpanda Serverless with AWS PrivateLink. Customer sessions highlighted serverless streaming benefits for companies like Crane Worldwide Logistics and Rivian's transition from Kafka to Redpanda for improved cost-efficiency. The panels explored the future of Kafka in the AI era and the impact of Apache Iceberg on data connectivity and productivity. The event concluded with interactive AMA sessions and prize giveaways, emphasizing Redpanda's commitment to community engagement and innovation in the data streaming landscape.
Dec 30, 2025
2,875 words in the original blog post.
Jean-Baptiste Dalido, Head of Infrastructure at Zenly, discussed the complexities and challenges associated with operating large-scale streaming systems during a conversation on March 17th. The discussion, presented as a fireside chat, invites further engagement through Red Panda Data's Community Slack and Twitter, encouraging listeners to connect and explore the topic more deeply.
Dec 30, 2025
45 words in the original blog post.
Amazon Redshift is a cloud-based data warehouse service designed to handle large volumes of data and complex queries, making it highly effective in various fields such as supply chain management, real-time analytics, and business intelligence. It allows organizations to integrate data from multiple sources into a centralized repository for comprehensive analysis. Redpanda, an Apache Kafka® alternative, enhances Redshift's capabilities by enabling real-time data streaming, which is crucial in sectors like finance and gaming. This integration provides immediate insights and facilitates dynamic responses to challenges such as inventory optimization and supply chain disruptions. A tutorial outlines how to set up an efficient data pipeline using Redpanda with Amazon Redshift, leveraging IoT sensors and GPS data to provide real-time analytics and insights. The process involves configuring Kafka Connect for integration, establishing a connection using a JDBC sink connector, and setting up a data pipeline for real-time data ingestion and analysis. This combination helps organizations optimize operations, reduce costs, and enhance service delivery by enabling analysis of live data and preemptive identification of trends and issues.
Dec 29, 2025
3,362 words in the original blog post.
Oxla is announcing the upcoming launch of its public tech preview, marking a significant milestone in its mission to revolutionize data analysis with what it describes as the world's fastest distributed analytical database. Designed to address common challenges in analytical databases such as poor performance, high costs, and scalability issues, Oxla supports the PostgreSQL wire protocol and syntax, making it accessible to those familiar with PostgreSQL. It boasts high efficiency, capable of executing both simple and complex queries swiftly, reducing the need for extensive server resources and integrating seamlessly with tools and libraries already compatible with PostgreSQL. The architecture is modern, focusing on cloud deployment with a decoupling of storage and compute, allowing for easy scalability and cost-effective storage solutions. As Oxla prepares for its tech-preview release, it invites users to test its capabilities, provide feedback, and engage with the community via email or Discord as it continues to refine its offerings and roadmap for the official commercial release.
Dec 29, 2025
642 words in the original blog post.
Redpanda introduces Pandaproxy, a new subsystem that enables users to interact with their data through a REST API, simplifying the use of the Apache Kafka-compatible event streaming platform. By eliminating the need for ZooKeeper and the JVM, Redpanda enhances performance and offers a streamlined experience with a single binary. Pandaproxy allows users to produce and consume event streams using HTTP CLI or libraries, offering JSON and base64 encoded payload support. The system supports consumer groups, enabling message distribution across multiple consumers, while ensuring consumer offsets can be managed efficiently. Redpanda provides detailed documentation for setting up and using Pandaproxy, including examples for creating topics, producing and consuming messages, and managing consumer groups. As Pandaproxy evolves, Redpanda plans to enhance its performance and feature set, encouraging users to engage with their community for updates and support.
Dec 29, 2025
1,467 words in the original blog post.
In the blog post, the author narrates their journey into the field of site reliability engineering and DevOps, beginning with an early interest in computing during high school in the mid-1990s. They describe how initial experiences with Linux and tech startups, including hardware performance testing, led to a career path that eventually brought them to Redpanda, where they now serve as the Site Reliability Engineer Manager. At Redpanda, the author is involved in coordinating engineering efforts and participating in architectural discussions while prioritizing cultural fit over technical skills in their hiring process. The author emphasizes the importance of humility and empathy in candidates and shares insights on the future of event streaming, which they envision as evolving toward more autonomous, reactive ecosystems. Additionally, the author offers advice on succeeding in a remote work environment, highlighting the benefits of goal-oriented work, enhanced quality of life, and the importance of having a dedicated workspace.
Dec 29, 2025
1,479 words in the original blog post.
Redpanda Cloud has achieved SOC 2 certification with no exceptions found by auditors Barr Advisory, covering all its products across AWS, GCP, and Azure. This certification underscores Redpanda's commitment to providing a secure platform with a robust security posture, allowing customers peace of mind regarding data safety. The company attributes its success in compliance to its "4 C's of compliance": Controls, Consistency, Culture, and Commitment, emphasizing that a culture valuing compliance and ethical behavior is crucial. Redpanda has spent the past year enhancing its controls to meet and exceed industry standards, reflecting its dedication to customer trust and security. The SOC 2 certification is a testament to the organization's continuous commitment to top-tier security and compliance efforts, and customers are encouraged to review the latest SOC 2 audit report to appreciate these efforts.
Dec 29, 2025
633 words in the original blog post.
Kafka Connect, a widely used tool since its release in 2016 for integrating streaming data with Apache Kafka and various data sources, faces challenges like complex deployment and limited debugging tools. Redpanda Connect emerges as a modern alternative, addressing these concerns with a cloud-native, single-binary setup that simplifies deployment and configuration, offering superior performance and easier scaling. While Kafka Connect relies on Java for data transformations, often requiring complex custom development, Redpanda Connect utilizes Bloblang for straightforward transformations directly within YAML configurations. For observability and debugging, Redpanda Connect provides an integrated experience with built-in tracing, logging, and metrics, whereas Kafka Connect demands additional infrastructure for log aggregation. Both systems ensure at-least-once delivery guarantees, but Redpanda Connect uses an in-memory transaction model that simplifies recovery and scaling. Additionally, Redpanda Connect supports AI-enhanced data pipelines with connectors for major LLMs, facilitating data extraction and streaming for AI-driven insights, positioning it as a future-focused tool for intelligent data pipelines.
Dec 29, 2025
1,311 words in the original blog post.
Redpanda is a modern, distributed platform designed for streaming data with a focus on maintaining data integrity and availability even under high-throughput conditions. The deployment choices for a Redpanda cluster significantly impact both performance and availability, particularly in business-critical environments where failures must be anticipated and mitigated. The text explores several deployment patterns for high availability (HA) and disaster recovery, such as clustered, multi-availability-zone, multi-region, and multi-cluster deployments, each offering different trade-offs between latency, resilience, and complexity. Key considerations include understanding failure scenarios, replica synchronization, rack awareness, partition leadership, producer acknowledgment, and partition rebalancing, all of which contribute to maintaining system availability. Additionally, the document emphasizes the importance of using remote storage over local storage in containerized environments and having a robust disaster recovery plan, supported by Redpanda's tiered storage architecture for data backup and recovery. The article is the first in a series on HA deployments, promising further exploration of these topics in future posts, while encouraging engagement with the Redpanda Community for further discussion and insights.
Dec 29, 2025
2,165 words in the original blog post.
Redpanda is revolutionizing the development of data-intensive applications with its upcoming engine, Redpanda One (R1), which emphasizes simplicity, flexibility, and efficiency. Originally conceptualized as a user-friendly and cost-effective alternative to Apache Kafka, Redpanda has evolved to support large-scale, mission-critical systems with ultra-low latency and high reliability. R1 introduces a multimodal streaming data engine allowing developers to dynamically select engineering pillars like availability, consistency, and latency for each topic, and features seamless integration with cloud object storage to reduce costs and enhance performance. With over 260 connectors through Redpanda Connect, it facilitates robust data and AI pipelines by ensuring comprehensive connectivity across major databases and systems. The introduction of "Bring-Your-Own-Cloud" (BYOC) and Sovereign AI further extends Redpanda's capabilities, offering secure, private, and efficient AI deployment within user-owned networks. This innovative approach marks a pivotal shift towards more accessible and integrated data management solutions, setting a new standard for the future of data engineering.
Dec 29, 2025
1,302 words in the original blog post.
Nick Chiu's blog post provides a step-by-step guide on setting up Redpanda, a Kafka API-compatible streaming platform, with PHP using a simple producer and consumer example in a Docker environment. The tutorial is designed for beginners and involves creating a Dockerfile for a PHP container, configuring a Docker Compose file, and writing PHP code for the producer and consumer using the simple_kafka_client extension. The producer script sends test messages to a Redpanda topic, while the consumer script reads and displays these messages, demonstrating the ease of integrating PHP with Redpanda. The post encourages readers to explore further use cases and engage with the Redpanda Slack community for support and advanced topics.
Dec 29, 2025
660 words in the original blog post.
Internet of Things (IoT) devices generate big data characterized by volume, velocity, and veracity, necessitating advanced data storage and processing systems. A powerful combination for real-time IoT data processing is Redpanda and Materialize, which together enable developers to create applications that handle large-scale data streams efficiently. Redpanda, compatible with the Apache Kafka ecosystem, provides high-throughput data ingestion and scalable storage, while Materialize allows real-time querying of data streams from Redpanda. This synergy is exemplified in an application that processes data from a Fitbit tracker to update a real-time leaderboard, demonstrating capabilities such as real-time dashboards and anomaly detection. The tutorial outlines setting up Redpanda and Materialize, developing producer code to simulate Fitbit data, and creating materialized views for live analytics, showcasing the potential of these technologies for real-time analytics in diverse industries like health tracking, industrial automation, and financial services.
Dec 29, 2025
1,657 words in the original blog post.
Cloud data transfer costs can be significant, especially when running distributed systems like Redpanda in a multi-AZ deployment. The main factor affecting these costs is whether the streaming platform should stay operational during an Availability Zone (AZ) outage. Using multi-AZ deployments increases resilience but incurs higher inter-AZ data transfer fees, whereas single-AZ deployments reduce costs but risk downtime during outages. Strategies to mitigate these expenses include optimizing data flow through efficient serialization formats like Apache Avro or Protocol Buffers, and enabling compression for data batches, which reduces the size of transmitted data. Additionally, Redpanda's follower fetching feature allows consumers to access data from the nearest replicas, lowering cross-AZ traffic. Leadership pinning, a more advanced technique, can be used to manage producer traffic by manually adjusting partition leaders within the same AZ. Together, these strategies can significantly reduce cloud costs, as demonstrated by a case where applying follower fetching and leadership pinning reduced monthly data transfer charges from $200 to $120.
Dec 29, 2025
1,674 words in the original blog post.
Redpanda incorporates access control lists (ACLs) as a core security feature to manage client access to data within its clusters, emphasizing authorization to ensure data protection. While authentication verifies trusted clients, ACLs determine the specific data access permissions for each client, employing a Kafka-compatible implementation to specify actions that authenticated users can or cannot perform. ACLs allow for detailed control by setting permissions based on user identification, host location, resource type, and specific operations, using the Redpanda rpk utility for creation, listing, and deletion of ACLs. This approach ensures that data access is restricted to the least privilege necessary, aligning with Redpanda's broader security strategies and ongoing development efforts to enhance its data API capabilities.
Dec 29, 2025
1,062 words in the original blog post.
Serverless architecture represents a significant evolution in cloud computing by abstracting infrastructure management, allowing businesses to concentrate on application logic and offering cost savings, scalability, and improved agility. It operates on a Function-as-a-Service (FaaS) model where functions are executed in response to specific events, and resources are provisioned on demand, leading to a "pay-as-you-go" billing system. This model is particularly beneficial for startups and enterprises, providing automatic scaling and cost-efficiency during traffic spikes and downtimes. Serverless architecture supports various use cases, including building APIs, trigger-based actions, event processing, media processing, data pipelines, webhooks, microservices, chatbots, and running scheduled tasks. While it differs from container architecture, the two can be used in a hybrid approach to maximize flexibility and performance. Redpanda Serverless further simplifies the adoption of this architecture, enabling organizations to develop impactful, real-time applications without infrastructure concerns.
Dec 29, 2025
1,386 words in the original blog post.
Redpanda has introduced a beta version of its schema registry, which is integrated into its Apache Kafka-compatible event streaming platform and eliminates the need for additional services or binaries. The schema registry supports storing schemas in a raft-based storage engine, providing high availability and allowing users to manage schemas directly through a RESTful interface available on every broker. This system is designed to simplify operational complexity by using optimistic concurrency control and symmetric brokers, eliminating the need for leadership configuration or failover strategies. The schema registry currently supports AVRO schemas, with plans to include JSON and PROTOBUF, and it allows for the evolution of APIs by providing different compatibility levels, such as BACKWARDS and FORWARDS, to ensure smooth version transitions. Users can easily register, retrieve, and manage schemas and their versions through documented endpoints, facilitating the decoupling of producer and consumer rollouts in an asynchronous event-driven architecture.
Dec 29, 2025
1,648 words in the original blog post.
MQTT, a lightweight messaging protocol, is widely used in IoT applications for its fault tolerance and adaptability to slow networks, while Redpanda is a modern streaming data platform compatible with the Kafka API, designed for high performance and simplicity in data streaming. The blog explores how combining MQTT with Redpanda using Waterstream, a novel approach that turns Redpanda into a full-fledged MQTT broker, can enhance real-time data streaming for IoT use cases such as fleet management, smart agriculture, and health monitoring. Waterstream integrates MQTT with Redpanda by using the latter as its only persistence layer, ensuring scalability, low latency, and operational simplicity while supporting MQTT protocol features like WebSockets and X.509 authentication. A live demo illustrates Waterstream's capabilities by simulating taxi movements in New York, showcasing how MQTT messages are processed and visualized with real-time analytics using Materialize. The blog further discusses deploying Redpanda and Waterstream at the edge, highlighting benefits like reduced latency and local data processing, although potential drawbacks include lack of high availability in minimal setups. Performance tests demonstrate Waterstream's ability to handle high message rates and client loads, affirming its suitability for modern IoT data streaming solutions.
Dec 29, 2025
2,965 words in the original blog post.
Batch data processing, characterized by its slow and costly nature, involves loading data into a warehouse before analysis, often leading to delays and outdated data. In contrast, stream processing offers real-time data analysis, allowing for more timely and informed business decisions by processing data as it arrives. The transition from batch to stream processing, while seemingly complex, is simplified by understanding that batch processing can be viewed as an extension of micro-batch processing. Stream processing brings significant advantages, such as reduced storage needs, quicker detection of data issues, and a resource-efficient system that's easier to scale. A simple streaming architecture involves data sources producing streaming data, which is written into a message bus for resilience, processed on the fly, and stored in a data lake for long-term use. This architecture can be implemented using tools like Redpanda, a high-performance data streaming platform, and Upsolver SQLake, which enables seamless integration of batch and stream processing. Together, they facilitate the creation of a data pipeline that combines historical with real-time data, delivering fresh insights and reducing manual, error-prone tasks, thereby enhancing data-driven decision-making processes.
Dec 29, 2025
1,688 words in the original blog post.
Redpanda 23.2 introduces a new feature called follower fetching, which allows consumers to fetch records from the closest physical replica of a topic partition, thereby optimizing performance and reducing cloud infrastructure costs. This feature is particularly beneficial for large streaming data clusters, as it minimizes cross-availability zone (AZ) network traffic, which can be costly on cloud platforms like AWS. By enabling follower fetching, companies can significantly cut down on network expenses, as illustrated by a hypothetical example of Acme Corporation, which experiences a 60% reduction in monthly networking costs, translating to a 53% annual saving in AWS infrastructure costs. Follower fetching allows consumers to read data from any replica in the cluster, optimizing network traffic and minimizing read latency. The feature is currently in beta and is expected to become generally available soon, offering companies substantial savings by configuring their Redpanda clusters and nodes to support rack awareness and client rack properties.
Dec 29, 2025
882 words in the original blog post.
Redpanda's integration with Kubernetes offers a robust data streaming solution that leverages the orchestration capabilities of Kubernetes for efficient deployment and management. Users have two main options for deploying Redpanda on Kubernetes: the Helm chart for quick setups and the Redpanda Operator for more complex, production-grade deployments. The Redpanda Operator, which has evolved to unify internal and customer-facing versions, provides advanced management features such as automated scaling, updates, and lifecycle automation. The unification process aims to simplify deployment across both cloud and self-managed environments by removing dependencies like FluxCD, leading to a streamlined codebase. This evolution supports a flexible and scalable infrastructure for diverse streaming use cases, with comprehensive documentation and resources available to assist users in navigating the deployment process.
Dec 29, 2025
1,551 words in the original blog post.
Redpanda Cloud has reinforced its status as a leading multi-cloud streaming platform with new updates and enhancements. Over June and July 2025, Redpanda Connect was made generally available across all deployment types, including Serverless, Dedicated, and Bring Your Own Cloud (BYOC) options. The platform introduced a new Spanner CDC connector for Google Cloud Spanner, enabling seamless change data capture and integration into Redpanda topics, alongside a new cache component optimized for write-heavy workloads. Additionally, a Slack reaction output feature was added for dynamic messaging based on streaming events. The introduction of an HTTP gateway input allows direct pipeline triggering via HTTP requests, enhancing flexibility for developers. Iceberg Topics have reached general availability for AWS and GCP BYOC clusters, enabling Kafka-compatible streaming data integration with Apache Iceberg tables for enhanced data lakehouse capabilities. Redpanda Serverless has been expanded with a new Schema Registry UI, client connection controls, and deployment options in three new AWS regions. Furthermore, Redpanda BYOC is now available on Microsoft Azure, offering users full control over their deployments while benefiting from Redpanda's managed services. These updates collectively enhance Redpanda Cloud's multi-cloud streaming capabilities, data connector ecosystem, and real-time lakehouse integration.
Dec 29, 2025
874 words in the original blog post.
The text discusses the importance and implementation of Change Data Capture (CDC) in traditional data platforms, highlighting its ability to provide real-time data capture, which is crucial for software companies. It explains how CDC helps in scenarios like auditing databases, invalidating cache, and monitoring real-time user behavior, using tools like Redpanda and visualization platforms. The document delves into the integration of CDC with SQL Server, Debezium, and Redpanda, detailing how this setup captures data changes in real-time by logging and forwarding them to a streaming platform, thus supporting event-driven architectures. The tutorial outlines the technical steps to configure SQL Server for CDC, integrate with Debezium and Redpanda, and visualize data changes, emphasizing the benefits of real-time data visibility for business insights and efficient backend operations. The guide also notes the versatility of Redpanda in integrating with various storage solutions and encourages further exploration of its capabilities through documentation and community resources.
Dec 29, 2025
4,227 words in the original blog post.
A developer advocate's primary mission is to simplify complex concepts for the developer community, allowing them to focus more on coding rather than searching for solutions. This role, which varies across companies, acts as a bridge between internal teams and the developer community to better understand and address developers' needs. This narrative is shared by Christina Lin, a developer advocate at the remote-first startup Redpanda, who emphasizes the importance of storytelling in effectively communicating ideas. Despite being an introvert, Lin enjoys creating easily digestible content through various mediums such as webinars, blog posts, and explainer videos, aiming to demystify real-time data streaming and other technical topics. Her journey from a lead developer in a traditional insurance company to her current role highlights her passion for helping others learn and adopt new technologies. Engaging with the community, sharing valuable insights, and collaborating across teams are crucial aspects of her daily responsibilities. Lin encourages those interested in becoming developer advocates to actively participate in the community, start experimenting with Redpanda's resources, and contribute to discussions and content creation.
Dec 29, 2025
1,233 words in the original blog post.
In the cloud computing era, Snowflake offers a SaaS data warehouse solution that simplifies data processing and analytics, integrating well with Apache Kafka and allowing flexible data ingestion methods. Redpanda, a Kafka-compatible streaming data platform, provides a simpler, faster, and safer alternative for building real-time data pipelines, making it an ideal partner for Snowflake in mission-critical workloads. This tutorial demonstrates setting up a data archiving system using a Redpanda cluster, Kafka Connect, and Snowflake for a fictional bookstore, PandaBooks LLC, which operates globally. It guides the user through creating a Snowflake database and table for data archiving, setting up a single-node Redpanda cluster, and configuring Kafka Connect to stream data to Snowflake, illustrating how Redpanda's reduced operational burden and Snowflake's cloud capabilities can enhance data management and analytics. The tutorial emphasizes seamless integration between Redpanda and Snowflake, leveraging Kafka Connect's source and sink connectors to efficiently handle data streams, and highlights the advantages of using cloud solutions to improve data operations.
Dec 29, 2025
3,535 words in the original blog post.
Software architecture patterns are crucial for building complex systems, offering tried-and-tested methods to achieve specific system traits such as availability, modularity, or scalability. There are no universally best patterns, as the choice depends on the specific requirements and use case. The text explores three common architectures: Layered (n-tier), event-driven, and microservices. Layered architecture enhances modularity and maintainability, but can be monolithic and challenging to scale individual features. Event-driven architecture promotes loose coupling and scalability but introduces complexity due to its asynchronous nature, which may affect data consistency and error handling. Meanwhile, microservices architecture allows independent scalability and fault tolerance, beneficial for large teams and complex applications, though it adds overhead and complexity in service communication and data management. Choosing the right architecture involves understanding the specific benefits and limitations of each pattern in relation to the application's needs.
Dec 29, 2025
2,332 words in the original blog post.
Organizations face challenges in managing and gaining insights from vast and constantly changing data streams across multiple systems, especially in scenarios requiring real-time decisions like financial trading. Change Data Capture (CDC) is a popular technique that addresses this by tracking database changes as they occur, streaming only those changes to reduce latency. Redpanda Connect offers a fresh alternative to Kafka Connect, providing flexibility, scalability, and simplicity in deploying data pipelines with its CDC connectors, which support popular databases like PostgreSQL, MySQL, MongoDB, and Google Cloud Spanner. Redpanda's approach includes parallel snapshotting, which allows for concurrent data reading, reducing the time required for snapshots and enhancing the performance of large-scale data migrations. The platform ensures reliable data delivery with features like logical replication, parallelized snapshots, and flexible checkpointing, making it a robust solution for real-time data streaming. Redpanda Connect supports various CDC use cases, including dashboard updates, database replication, real-time business workflow triggers, and AI/ML model training, offering both cloud-based and self-managed solutions with an Enterprise license.
Dec 29, 2025
1,034 words in the original blog post.
Apache Beam is a powerful open-source framework designed for creating and executing data processing pipelines, capable of handling both batch and streaming data. It allows developers to write pipeline code in their preferred programming language through its language-specific SDKs, including Python, Java, and Go, and supports execution on various engines such as Apache Flink, Apache Spark, and Google Cloud Dataflow, offering high portability. In a practical demonstration, a streaming ETL pipeline is constructed using Apache Beam and Redpanda to process real-time data from an e-commerce application. This pipeline involves reading data from a Redpanda input topic, filtering and enriching data based on regional information, and writing the processed data to an output topic, showcasing Apache Beam's flexibility and ease of use in building data processing workflows. The tutorial also includes steps for setting up necessary software, creating Java classes for data processing, and executing the pipeline using Maven, illustrating how Beam simplifies the development of scalable data processing systems.
Dec 29, 2025
3,300 words in the original blog post.
Agentic AI is poised to revolutionize modern enterprises by creating a digital workforce with superhuman capabilities, transforming how tasks are executed, but it also introduces new challenges related to unpredictability and governance. Redpanda proposes the Agentic Data Plane (ADP) as a solution, providing a managed, governed data control plane that ensures secure connectivity between AI agents and enterprise data while maintaining governance and scalability. The ADP incorporates three foundational components: Redpanda Streaming, Redpanda Connect, and Oxla, a high-performance SQL query engine, which collectively support seamless data integration and rigorous governance. The platform treats every agent interaction as a durable event, allowing for analysis, compliance, and reproducibility, thus enabling enterprises to manage AI agents effectively by granting them fine-grained, temporary access to sensitive data. By offering flexible deployment options and integrating advanced access control, the ADP aims to balance the efficiency of AI agents with operational and compliance security, empowering organizations to harness AI's potential responsibly and sustainably.
Dec 29, 2025
991 words in the original blog post.
Apache Kafka, once a leader in event streaming, faces challenges keeping up with modern demands for high-performance, low-latency data processing. Organizations dealing with mobile, AI, and edge applications that process trillions of events daily are finding Kafka's cost and complexity prohibitive and are exploring alternatives like Redpanda. Redpanda is designed to address the limitations of Kafka, offering a high-throughput, low-latency streaming platform that reduces infrastructure costs and complexity. It features a single-binary deployment, operates without Java Virtual Machine (JVM) or ZooKeeper, and employs an architecture that ensures consistent, low latencies and high throughput, making it significantly faster than Kafka. Redpanda's tiered storage, intelligent rebalancing, and compatibility with Kafka's API ecosystem allow seamless integration with existing Kafka applications while significantly reducing costs.
Dec 29, 2025
1,434 words in the original blog post.
In response to JPMorgan Chase's Global CISO Patrick Opet's open letter emphasizing the need for secure-by-default enterprise software, Redpanda Cloud's Bring Your Own Cloud (BYOC) model emerges as a solution that balances security and convenience for highly regulated industries like financial services. Opet criticized the vulnerability of the current SaaS model, which often sacrifices control for convenience, and called for solutions that offer secure, continuous control validation and flexible deployment options. Redpanda's BYOC allows organizations to maintain full control over their data and compliance measures while benefiting from the efficiencies of managed services. The BYOC approach enables enterprises to stream sensitive processes such as KYC checks and fraud detection within their own cloud environments without compromising governance or performance. This model exemplifies a shift towards SaaS 2.0, which focuses on returning control to organizations without reintroducing the complexities of traditional data centers, supporting innovation and compliance in high-stakes environments.
Dec 29, 2025
845 words in the original blog post.
Originally from Poland and now residing in the Czech Republic, the author transitioned from a background in cognitive science to a career in technology, ultimately becoming a Site Reliability Engineer (SRE) at Redpanda. Despite initial intentions to work on the business side of tech, exposure to command line interfaces during a developer bootcamp sparked a passion for automation and coding, leading to a pivot towards software engineering. The role of an SRE, which involves ensuring product reliability and responding to urgent engineering tasks, requires a unique blend of technical, communication, and business skills. The author emphasizes the importance of focus and routine to manage the fast-paced demands of SRE work, and expresses a keen interest in the evolving field of monitoring and observability within engineering. Redpanda is seeking new team members with expertise in areas like Kubernetes, monitoring, and public cloud vendors, while valuing candidates who are intelligent, kind, and open to learning.
Dec 29, 2025
1,016 words in the original blog post.
Redpanda Cloud, an efficient streaming data service launched last year, offers a 99.9% uptime SLA with features like zero downtime upgrades, partition balancing, and 24/7 support. The service has enhanced its security and performance with significant updates, such as the introduction of Bring Your Own Cloud (BYOC) clusters, which combine the benefits of managed services with the cost-effectiveness and data sovereignty of self-managed deployments. Recent improvements include customer-managed VPC deployments, allowing users to deploy Redpanda clusters within their existing VPCs, enhancing control over security and governance. This deployment model, available on GCP and soon on AWS, supports organizations with strict network and security requirements by avoiding the need for new resource creation. Redpanda has also bolstered its compliance posture with SOC2 Type 2 Attestation, showcasing its commitment to data security and privacy. These updates reflect Redpanda's mission to give developers control over their data while meeting compliance and privacy obligations in the cloud environment.
Dec 29, 2025
784 words in the original blog post.
The growing demand for personalized, real-time customer experiences has led businesses to depend heavily on continuous data streams, with Apache Kafka® Streams and Spark Streaming emerging as two popular technologies for data stream processing. Kafka Streams, a component of Apache Kafka, offers a Kafka-native library that processes real-time data without requiring an external stream processing cluster, supporting JVM languages like Java and Scala. It excels in ease of use, scalability, and fault tolerance, although it lacks native SQL support and machine learning capabilities. On the other hand, Spark Streaming, part of the Apache Spark analytics engine, handles real-time data with high throughput and provides built-in support for Java, Scala, and Python, as well as extensive integration with other technologies. It offers a rich set of analytical operations and machine learning libraries, making it a versatile choice for complex data processing tasks, though it demands a steeper learning curve. Both technologies are open-source under the Apache License 2.0, with Spark Streaming gradually being overshadowed by Spark Structured Streaming. Redpanda, an alternative streaming data platform, offers enhanced performance and cost efficiency by being API-compatible with Kafka, facilitating seamless data integration and processing with Apache Spark.
Dec 29, 2025
2,310 words in the original blog post.
Redpanda provides a method for exporting streaming data into Amazon S3, initially using JSON, but for more complex analytical workloads, Apache Parquet is recommended due to its binary, columnar format that is efficient for analytics, compressing well and loading quickly. Redpanda Connect can encode streaming data directly into Parquet files, which can serve various purposes such as JSON for web applications and Parquet for data analytics, facilitating seamless data integration. In this process, Redpanda reads messages from a topic, encodes them into Parquet format, and writes the compressed files to S3, requiring a schema definition for data structuring. The use of the zstd compression algorithm optimizes storage costs, and the resulting Parquet files can be queried using tools like Pandas, Spark, or Athena. The series discusses setting up a Redpanda Connect pipeline, which continuously processes data until manually stopped, and emphasizes the importance of cleanup and adjusting configurations for production environments. The next installment will explore integrating SQS and S3 notifications for building event-driven workflows that respond to new data in real time.
Dec 29, 2025
716 words in the original blog post.
Operational IT systems are critical in various industries, including healthcare, transportation, retail, and finance, as they facilitate essential functions such as accessing medical records and managing financial transactions. In the realm of financial technologies (FinTech), these systems must be fast, scalable, accurate, and reliable to handle high-stakes scenarios like real-time stock trading and peer-to-peer payments. Event-driven microservices offer an effective architecture for FinTech systems, providing improved reliability, loose coupling, and high availability through asynchronous interactions and independent service management. Redpanda emerges as a modern event streaming platform that addresses the limitations of traditional platforms like Apache Kafka, offering low-latency, high-throughput data processing, and cost-efficient scalability. Built in C++ and compatible with the Kafka protocol, Redpanda enhances real-time applications by ensuring reliable and scalable communication between microservices, making it a suitable solution for the demanding needs of FinTech operations.
Dec 29, 2025
1,108 words in the original blog post.
Redpanda has launched its native Apache Iceberg integration called Iceberg Topics, aimed at enhancing stream data processing by allowing users to access the latest data in their data lakes for real-time analytics. Alongside this, PuppyGraph, a graph computing engine that requires no ETL, is the first to integrate with Redpanda, facilitating high-speed graph queries directly on data lakes. This integration allows seamless, no-ETL real-time graph analytics by using Redpanda's Iceberg Topics, which stores data in both streaming log and table formats, making it accessible for SQL queries through various analytics tools. The blog post provides a step-by-step guide for integrating these technologies to analyze financial transaction data, showcasing the potential of this streamlined approach for real-time graph analysis.
Dec 29, 2025
1,185 words in the original blog post.
Redpanda's inaugural Streamfest was a half-day virtual event designed to offer developers a concentrated dose of data streaming insights, focusing on real-time applications, analytics, and AI. Featuring over 2,000 registrants, 14 diverse topics, and a hands-on workshop, Streamfest emphasized practical knowledge sharing from experts who have firsthand experience. Key sessions included technical AMAs, visionary talks on data streaming, and discussions on the integration of Redpanda with tech like Apache Iceberg and Snowpipe Streaming. Highlights also covered Redpanda's innovations such as the new Postgres Change Data Capture connector and the Sovereign AI approach, which keeps data local to enhance privacy. The event showcased the potential of Redpanda's BYOC model for cost efficiency and scalability, as well as its applicability in industry-leading use cases, like NYSE's cloud streaming. The engaging format, including panel discussions, lightning talks, and a lively chat room, underscored the communal spirit of the developer community, culminating in an interactive workshop that allowed attendees to experiment with Redpanda Connect in AI scenarios.
Dec 29, 2025
3,581 words in the original blog post.
Redpanda Cloud has introduced several updates to enhance both the security and usability of its Dedicated and Bring Your Own Cloud (BYOC) products, including the integration of Google Cloud's Private Service Connect for improved network connectivity and security, support for mutual TLS (mTLS) authentication for Kafka API, and single sign-on (SSO) via OpenID Connect. A private beta for Data Transforms is available, allowing in-broker data processing with languages like Go and Rust, and Cloud API capabilities have been expanded to facilitate cluster operations. A new Terraform provider is in private beta to streamline infrastructure deployment, and additional cloud regions have been added across AWS and GCP to meet data sovereignty requirements. The high-throughput tier now supports up to 67,500 logical partitions, offering enhanced performance and value for customers.
Dec 29, 2025
790 words in the original blog post.
Redpanda claims to significantly outperform Apache Kafka in terms of speed, particularly in tail latency, which is crucial for applications requiring swift and reliable message delivery. Unlike Kafka, which is written in Scala and operates within the Java Virtual Machine, Redpanda is built in C++ and leverages specific architectural enhancements such as thread-local data structures, memory pinning, and the use of advanced Linux libraries like DPDK and io_uring. These innovations enable Redpanda to handle memory more efficiently, bypassing the limitations of the page cache and optimizing the use of RAM for caching. Additionally, Redpanda benefits from automatic kernel tuning and a thread-per-core architecture, which minimizes latency by eliminating the overhead of shared memory and ensuring that no instruction blocks for more than 500 microseconds. This makes Redpanda particularly suited for applications with strict latency requirements, such as algorithmic trading, real-time gaming, and IoT use cases, which might otherwise necessitate custom solutions. By providing superior performance while maintaining compatibility with Kafka APIs, Redpanda offers substantial value and flexibility to developers and businesses.
Dec 29, 2025
1,369 words in the original blog post.
Redpanda has implemented a process to enhance developer productivity by optimizing their continuous integration (CI) pipeline for parallel execution using Buildkite. By leveraging Buildkite's parallelism attribute and pre-command hook, alongside GitHub labels, developers can initiate parallel builds with ease, simply by adding a label such as "ci-repeat-X" to their pull requests. This workflow allows multiple instances of CI steps to run concurrently, reducing build times and enhancing stability. The process avoids clutter and manual steps by automatically managing the labels, ensuring a fast, simple, and productive experience. This approach reflects Redpanda's commitment to improving the efficiency of both their internal engineering team and external developers using their platform.
Dec 29, 2025
964 words in the original blog post.
Modern software often fails to fully exploit the capabilities of advanced hardware, such as superscalar CPUs and high-speed memory units, resulting in significant performance inefficiencies. Despite hardware advancements like NVMe SSDs and 3dXpoint technology offering faster access times and higher throughput, software architecture frequently lags behind, causing performance losses up to 100 times in complex operations compared to simple data read/write tasks. This inefficiency is attributed to factors such as locks, cache contention, and garbage collection, which hinder optimal performance. The development of Redpanda illustrates an effort to bridge this gap by understanding and leveraging hardware constraints and capabilities, thus optimizing software performance and reducing costs by aligning software operations with the physical realities of hardware.
Dec 29, 2025
397 words in the original blog post.
The final installment of a five-part series explores how to automate data workflows using S3 Event Notifications and Amazon SQS, enabling real-time, event-driven data pipelines with Redpanda. By configuring an S3 bucket to send notifications to an SQS queue upon new object creation, Redpanda Connect can listen to this queue to automatically process incoming data. This setup leverages AWS-native services to build robust, automated data ingestion pipelines directly from S3 to Redpanda streams without manual intervention. The series provides a detailed walkthrough of setting up S3 Event Notifications, configuring Redpanda Connect to monitor the SQS queue, and deploying an event-driven pipeline that reacts to new data uploads. It also highlights best practices such as cleaning up resources to avoid unnecessary costs, encouraging users to explore Redpanda Serverless for efficient, real-time data streaming without the complexity of managing infrastructure.
Dec 29, 2025
796 words in the original blog post.
OpenAI's revolutionary ChatGPT is being leveraged by developers, including those at Redpanda, to make technical tools more accessible to non-technical users. By using OpenAI APIs, a prototype was developed that allows users to describe message filters in plain English, with AI generating the necessary code for Redpanda Console, a web UI for managing Redpanda or Apache Kafka clusters. This innovation aims to simplify complex tasks like message filtering, which typically require coding skills. The experiment highlighted the challenges of cost and reliability, as OpenAI's models can be expensive and sometimes produce errors or nonsensical outputs. Despite these issues, AI-driven code generation is seen as a promising tool for democratizing software development, akin to the impact of AI on digital illustration, though it requires carefully crafted prompts and constraints to be effective. The experiment serves as a proof of concept for potential future applications and invites feedback from the Redpanda community.
Dec 29, 2025
1,344 words in the original blog post.
Redpanda utilizes the C++ Seastar framework, which employs a thread-per-core architecture that enhances memory allocation efficiency by overriding default methods, although it limits the use of virtual memory and existing analysis tools. To address memory usage analysis, Seastar introduced a sampled heap memory profiler that operates with minimal overhead, allowing continuous use in production environments to debug real-life workload issues. This profiler records only a fraction of allocations, using a dynamic, randomized sampling interval to prevent biases, and tracks live allocations to identify the most significant memory consumers. Small and large allocations are managed differently, with small ones grouped into pools and large ones tracked via page metadata, allowing the profiler to attribute memory usage to specific code sites through backtraces. The profiler's integration into Seastar enables real-time memory usage monitoring through an HTTP API, although it's particularly useful during out-of-memory scenarios by identifying top memory allocation sites. Despite added profiling overhead, optimizations to the Seastar allocator improved overall performance, with the profiler proving instrumental in diagnosing and resolving memory issues efficiently, and the enhancements have been merged into upstream Seastar for broader community benefit.
Dec 29, 2025
1,708 words in the original blog post.
Redpanda is presented as a Kafka replacement for mission-critical systems, emphasizing the importance of rigorous testing to ensure system reliability. The text highlights the role of chaos testing, a method used to validate systems under fault conditions, which was popularized by Netflix and involves intentionally injecting faults like network partitioning and process termination to test system resilience. Redpanda incorporates the Raft protocol for linearizability tests, ensuring correct concurrent operation execution, and has developed an in-house consistency checking tool, Gobekli, to address the challenges of validating long operation histories. This approach allows for real-time validation of linearizability with low computational complexity, essential for maintaining the integrity and reliability of distributed systems. The article also details the fault injection process and the observed effects on latency and availability, concluding with Redpanda's commitment to providing a trustworthy and robust system, inviting users to try the product and engage with their community.
Dec 29, 2025
2,182 words in the original blog post.
In the evolving landscape of cloud-native applications, distributed architectures have become prevalent, moving from monolithic to microservices and now to even smaller services or functions. This shift, while beneficial, introduces challenges in data sharing and consistency across systems, commonly tackled with dual writes, which often lead to inconsistencies. Change Data Capture (CDC) offers a solution by capturing row-level changes and processing them downstream, ensuring data durability and consistency without the issues associated with dual writes. CockroachDB, a distributed database, facilitates CDC through its Changefeeds, allowing integration with systems like Redpanda, a Kafka API-compatible streaming data platform. This setup enables efficient implementation of CDC use cases such as reporting and data consistency across CockroachDB shards. The tutorial guides users through setting up CockroachDB and Redpanda, creating data tables, and configuring CDC to capture, stream, and consume change events, ultimately addressing data inconsistency challenges in distributed systems.
Dec 29, 2025
2,118 words in the original blog post.
Exploring the challenges of testing complex codebases, the text delves into refining test suites using custom allocators and partial fixtures within the Google Test framework. The author describes the initial problem of managing independent resources within test suites and proposes a solution that involves combining partial fixtures using C++'s multiple inheritance and the CRTP pattern. Though multiple inheritance presents issues like virtual function ambiguity and inheritance diamonds, the post suggests leveraging mix-ins as an alternative, drawing inspiration from Java's Mixin library. By implementing recursive templates and fold expressions, the solution simplifies the integration of partial fixtures, enabling a more organized and flexible testing environment. The author shares insights from developing these tools at Oxla, inviting readers to explore further resources and a code repository for practical examples.
Dec 29, 2025
1,280 words in the original blog post.
Organizations seeking to build real-time data-processing applications can benefit from using ksqlDB, an event-streaming database that simplifies application development by combining Apache Kafka API components—Kafka Connect and Kafka Streams—into one system. Unlike other popular tools like Flink or Apache Spark, ksqlDB allows for the creation of complete streaming applications using only SQL statements, eliminating the need for additional programming languages. This tutorial demonstrates how to integrate ksqlDB with Redpanda, a Kafka-compatible storage solution, to build a stream-processing application, specifically focusing on creating a materialized cache for fast access to precomputed query results. The tutorial guides users through the installation and configuration of Redpanda and ksqlDB using Docker, the creation of streams and materialized views for data processing, and the execution of SQL queries to test the application. By leveraging ksqlDB and Redpanda, developers can efficiently manage and query streaming data for various use cases, such as anomaly detection and log monitoring.
Dec 29, 2025
1,743 words in the original blog post.
Redpanda offers a streamlined alternative to Apache Kafka by providing built-in automation that simplifies workflow management and eliminates the operational complexities associated with Kafka's architecture. While Kafka's flexibility is valuable for creating real-time data pipelines, it often leads to significant manual upkeep, creating bottlenecks and inefficiencies, especially for teams with limited DevOps resources. Redpanda addresses these challenges with a self-contained, zero-dependency system that integrates automation directly into its core, reducing the need for external coordination services and making deployment, management, and upgrades more straightforward. The platform automates tasks such as partition distribution, data replication, and fault recovery, ensuring high durability and availability without manual intervention. Additionally, Redpanda's native DevOps tools and observability features, including the rpk command-line tool and Redpanda Console GUI, enhance operational efficiency. This approach accelerates development and testing, simplifies schema management within CI/CD pipelines, and facilitates the orchestration of data streams across diverse environments, enabling teams to focus on business logic rather than infrastructure management.
Dec 29, 2025
914 words in the original blog post.
Alexander Gallego joins Eric Anderson on the Contributor Podcast to discuss Redpanda, an event streaming platform designed to serve as a drop-in replacement for Kafka. The podcast highlights Redpanda's appeal to developers due to its high performance and simplicity, particularly in its ability to eliminate the complexity associated with scaling Kafka by offering a single binary deployment. The conversation delves into how Gallego and the Redpanda team aim to push the boundaries of streaming technology into the future, with further details available in the full podcast notes.
Dec 29, 2025
105 words in the original blog post.
Redpanda's Tiered Storage, introduced in 2021, is a crucial component of its data streaming platform, facilitating cost-efficient data management by offloading log segments to cloud storage. By integrating with cloud storage solutions like AWS S3, GCP Cloud Storage, and Azure Blob Storage, and supporting development with MinIO, Redpanda ensures data portability and self-sufficiency through archival storage that includes topic and partition manifests. The platform's v22.2 release enhances functionality with Remote Read Replicas (RRR), enabling read-only clusters that reduce production load by utilizing archived data in the cloud. This setup allows users to create separate clusters for consumers, offering flexibility in cluster size and accommodating additional consumers without impacting performance. The architecture relies on the ntp_archiver within the scheduler_service to manage data uploads, downloads, and synchronization between local and remote clusters. Redpanda's approach provides a framework for efficient data handling, which can be leveraged for various applications, such as offline machine learning training and edge streaming CDNs, while ensuring compatibility across updates through a feature manager.
Dec 29, 2025
1,577 words in the original blog post.
The text discusses the differences and use cases of Apache Flink and Redpanda Data Transforms for data stream processing. Apache Flink excels in handling complex, stateful transformations involving aggregations, joins, and event time processing, making it suitable for operations that require maintaining state across events and interacting with external systems. In contrast, Redpanda Data Transforms, leveraging WebAssembly, offers cost-efficient, in-broker stateless transformations such as filtering and transcoding, ideal for scenarios where low latency and real-time processing are critical. Redpanda allows developers to use multiple programming languages, providing more flexibility and avoiding the need to manage additional distributed systems, making it a preferable choice for simpler, latency-sensitive transformations.
Dec 29, 2025
827 words in the original blog post.
Data streaming is crucial for modern businesses to ensure real-time data reflection across systems, particularly in retail where inventory accuracy and operational efficiency are paramount. Platforms like Redpanda facilitate this by serving as a unified engine for streaming data from various sources through Kafka-compatible connectors, such as those for MongoDB. The text details a tutorial on setting up a data streaming system using Redpanda and MongoDB, leveraging Docker for containerization and Kafka Connect for data integration. The guide explains the installation and configuration of Redpanda, MongoDB, and Kafka Connect, and demonstrates how to consume and test real-time data streams from MongoDB collections. By subscribing to the change feed from MongoDB, companies can enhance efficiency and customer satisfaction through timely data updates across departments. This tutorial opens up possibilities for using data streaming with technologies like MongoDB and Redpanda for various applications, encouraging exploration of additional integrations with platforms like Snowflake and Google Cloud Storage.
Dec 29, 2025
2,502 words in the original blog post.
Redpanda Connect provides a streamlined solution for transforming and streaming CSV data from Amazon S3 into Redpanda Serverless topics, enabling real-time data integration without the need for custom infrastructure. By using Redpanda Connect's configuration, users can efficiently read CSV files from an S3 bucket, normalize data by converting fields and stripping sensitive information, and publish the cleaned data to a specified Redpanda topic for downstream processing. This approach leverages Bloblang for in-stream transformation logic, ensuring flexibility and performance while maintaining data security through Redpanda's built-in secrets store. The article outlines the setup and implementation of this data pipeline, emphasizing the ease of deployment and continuous operation, and hints at future posts that will reverse the data flow from Redpanda back to S3, offering additional filtering and formatting options.
Dec 29, 2025
877 words in the original blog post.
Modern cloud-based applications often utilize microservices for their scalability and flexibility, but this approach can introduce architectural challenges such as service discovery, communication protocols, logging, configuration management, and security. To address these issues, event-driven architecture (EDA) is proposed as a solution, leveraging asynchronous data flow to reduce service coupling and enhance resilience. The tutorial illustrates how to implement EDA using Redpanda and Spring Boot to transform a coffee shop's ordering system from a request-driven to an event-driven model, thereby improving scalability and fault tolerance. By utilizing Kafka for inter-service communication, the system can handle increased workloads and prevent order loss without altering existing operational rules. This transition demonstrates EDA's efficacy in managing microservices, ensuring that orders are processed efficiently even if individual service instances are temporarily unavailable.
Dec 29, 2025
3,670 words in the original blog post.
Real-time data streaming has become essential for businesses, allowing them to gain immediate insights and make rapid decisions by continuously processing and analyzing data as it is generated. This capability is central to numerous business functions such as fraud detection, user experience optimization, and maintaining high availability in critical applications. Unlike batch processing, which handles data in large chunks at scheduled intervals, real-time streaming processes data as it arrives, offering instant insights but requiring low-latency infrastructure. The architecture involves producers, brokers, and consumers, with Redpanda emerging as a notable platform that simplifies deployment and management while delivering significant performance improvements over alternatives like Apache Kafka. Although real-time data streaming offers substantial benefits, including enhanced agility and improved customer experiences, it also presents challenges such as managing traffic spikes, minimizing costs, maintaining high availability, and ensuring data safety. Redpanda addresses these challenges with features like tiered cloud storage and the Raft consensus algorithm, making it a robust solution for developers needing efficient, scalable, and secure data streaming capabilities.
Dec 29, 2025
1,432 words in the original blog post.
Equal Experts, a global IT services company, recently assisted a FinTech client in developing a low-latency market data alerting system using Redpanda, a Kafka-compatible streaming data platform. The firm initially considered a technology stack including Apache Kafka, but faced challenges with latency and costs, leading to the adoption of Redpanda for its simplicity, performance, and lack of dependencies. After benchmarking, Redpanda met the stringent latency requirements with its lean architecture, enabling the client to achieve a 20% performance boost by using ARM-based Graviton instances in AWS. Despite minor challenges with SASL authentication and configuration, the transition was successful, aided by modular system design and active support from Redpanda engineers. The experience highlighted the importance of evaluating streaming data platforms to ensure they meet specific project requirements, and the ease of migrating to Redpanda due to its Kafka API compliance.
Dec 29, 2025
893 words in the original blog post.
Redpanda 25.2 introduces significant advancements in Iceberg capabilities, enhancing enterprise capabilities to unify streaming and analytics workloads efficiently within a real-time open lakehouse architecture. The update includes integrations with Databricks Unity Catalog and AWS Glue Catalog, elevating Iceberg to a first-class capability within the Redpanda ecosystem by expanding schema support and introducing features like Schema Registry Authorization key updates for improved security and compatibility with Apache Kafka® clients. Redpanda 25.2 supports AWS Glue as the Iceberg REST Catalog for seamless integration with AWS tools, allowing real-time streaming data to integrate into existing AWS ecosystems. It also enhances collaboration with Databricks, enabling Redpanda's real-time event streams to be queryable in Databricks using Spark or DB SQL, supporting AI/ML models and analytics without batch ETL delays. The release brings Iceberg support for JSON schemas, catering to customers who prefer JSON for Kafka workloads, and introduces Schema Registry Authorization for granular access control, enhancing data governance and security. Additionally, Redpanda officially certifies kafka-javascript as a mainstream Node.js client for Kafka, providing developers with a reliable tool for building modern backend applications.
Dec 29, 2025
1,236 words in the original blog post.
Redpanda developed an HTTP client to integrate with the Seastar framework, addressing the lack of a compatible client library for Seastar's threading model. The new client facilitates operations like accessing the Amazon AWS API by incorporating features such as TLS 1.2, chunked encodings, and custom headers while ensuring efficient performance through zero-copy networking and asynchronous operations using Seastar futures. To overcome the complexity of protocol parsing and compatibility issues with existing libraries, Redpanda utilized Boost.Beast as a customizable request serializer and response parser. This allowed for efficient data handling with minimal memory usage and copying, by leveraging Redpanda's iobuf zero-copy buffer implementation. The client seamlessly interfaces with Seastar's input and output streams, enabling data streaming directly from disk to network without additional memory overhead. This approach not only optimizes performance but also provides flexibility for integrating with existing transport layers, maintaining standards compliance, and minimizing cache pollution.
Dec 29, 2025
1,120 words in the original blog post.
Global organizations increasingly rely on event-streaming tools like Redpanda and stream-processing platforms such as Databricks to handle large volumes of data for real-time decision-making. For instance, these tools can be used to recommend content to users based on their interactions with mobile or web applications, by streaming clickstreams through Redpanda to Databricks, where a recommendation engine processes the data. Redpanda serves as a fast, scalable alternative to Apache Kafka, featuring compatibility with Kafka's API, and it operates on various platforms, including virtual machines and Kubernetes. The tutorial demonstrates how to set up a data pipeline using Redpanda and Databricks, from producing data to a Redpanda topic to storing it in Databricks as CSV files, and subsequently analyzing the data in real-time. The process includes setting up Redpanda using Docker, configuring Databricks to process streaming data, and running Apache Spark queries to analyze and visualize the data. This setup enables organizations to efficiently analyze data in real-time for various projects, leveraging the speed and scalability of Redpanda, which integrates well with existing Kafka tooling and operates without dependencies on JVM or ZooKeeper.
Dec 29, 2025
1,686 words in the original blog post.
Redpanda, a new streaming storage engine, underwent rigorous testing with Jepsen to ensure its reliability and consistency in distributed systems. Redpanda employs verified protocols like Raft and two-phase commit to safeguard data despite failures, but theoretical soundness doesn't always guarantee flawless implementation. Partnering with Jepsen, a company known for auditing distributed systems, Redpanda aimed to validate its consistency and availability. The Jepsen tests, which lasted several months, revealed that while Redpanda's consensus layer is robust, there were issues in the idempotency and transactional layers that have since been addressed. The remaining consistency challenges relate to inherent properties of the Apache Kafka protocol rather than implementation flaws. Additionally, Jepsen identified some availability issues that Redpanda is investigating, though they don't result in data loss or event reordering. This partnership allowed Redpanda to enhance its documentation, improve in-house chaos testing, and gain insights into the differences between Kafka and database transactional models, ultimately contributing to a safer and more efficient platform.
Dec 29, 2025
3,353 words in the original blog post.
Data flows through businesses in various forms, and real-time data integration allows organizations to process this data instantaneously, enhancing decision-making, operational efficiency, and customer experiences. This process involves capturing, processing, and combining data from multiple sources as it is generated, providing instant access to the latest information. While it offers numerous benefits such as faster response times, improved operational efficiency, and enhanced customer experiences, real-time data integration also presents challenges, including handling high data velocity, ensuring data quality, managing diverse data formats, and maintaining data security. Best practices like simulating and testing data pipelines, prioritizing data quality, using scalable and flexible platforms, leveraging cloud-native services, and developing a clear data governance framework can help overcome these challenges. Redpanda's streaming data platform simplifies real-time data integration with its ultra-low latency, cost efficiency, and intuitive design, enabling businesses to act quickly on actionable insights through scalable compute and storage options.
Dec 29, 2025
1,122 words in the original blog post.
Data streaming has emerged as a crucial solution for modern organizations seeking to quickly transform vast amounts of data into actionable insights, especially in scenarios requiring real-time responses like fraud detection and inventory management. Unlike batch processing, which analyzes fixed-size data chunks at specified intervals, stream processing deals with continuously generated data, allowing for real-time or near real-time analysis and decision-making. This approach is essential for applications such as financial trading, emergency response, and IoT systems, where instant data processing can significantly impact outcomes. However, implementing streaming data solutions can be complex, requiring expertise in distributed systems and considerations for scalability, reliability, and security. While traditional tools like Apache Kafka are widely used, newer platforms like Redpanda offer simplified, cost-effective alternatives with high performance and lower resource consumption. As organizations increasingly rely on real-time data processing, understanding the nuances of streaming technologies and choosing the right tools becomes vital for maintaining a competitive edge.
Dec 29, 2025
2,498 words in the original blog post.
In Pub/Sub messaging systems, last value caching (LVC) is a technique used to ensure new subscribers quickly access the most recent data, suitable for applications like stock tickers that require real-time updates without historical data. Apache Kafka supports this through compacted topics, which manage data size by removing older messages once newer versions are available, though this results in data loss unsuitable for use cases needing full historical data, such as back-testing. Traditional methods for managing LVC include dual writes, which risk data inconsistency, and streaming pipelines, which require managing external processors. Redpanda offers an innovative solution using WebAssembly-powered Data Transforms, allowing efficient, no-code, in-broker transformation that creates a consistent copy from a full to a compacted topic, addressing scalability, fault tolerance, and simplicity without external dependencies. This feature, available in Redpanda 24.1, enhances last value caching and opens possibilities for diverse applications through user-defined message transformations.
Dec 29, 2025
2,157 words in the original blog post.
Transactional databases, commonly known as OLTP systems like PostgreSQL or MySQL, are versatile and capable of running various types of queries, making them suitable for structured, semi-structured, and unstructured data. However, for performance and scalability in handling large datasets, OLAP databases such as Oxla excel, especially when running queries over extensive data with mechanisms like partitions to enhance efficiency. In a benchmark comparison using the Star Schema Benchmark dataset, Oxla demonstrated faster query performance compared to PostgreSQL, highlighting OLAP's strengths in scenarios involving large-scale data analysis. OLAP databases use columnar storage, which enables quick column scanning and reduced storage size, making them easier to scale than OLTP systems, which focus on transactions and have limitations in scalability and indexing. Despite OLAP's advantages in cost-effective storage and scaling, they are not ideal for frequent data modifications due to constraints in updating single rows and the use of distributed object storage, which complicates row-level changes. Consequently, OLAP systems are best suited for large, relatively static datasets where modification needs are minimal, whereas OLTP databases are preferable for applications requiring frequent updates and transactions.
Dec 29, 2025
713 words in the original blog post.
Fortis Games is revolutionizing the gaming experience by creating a scalable and flexible real-time data infrastructure using Redpanda, a Kafka-compatible streaming data platform, to support game development across mobile, PC, and console platforms. The company aims to provide an inclusive and engaging player experience by utilizing real-time game data for personalized gameplay, while ensuring that developers can easily interact with data systems without needing to manage backend complexities. Redpanda was chosen for its ability to deliver high performance, reduced administrative overhead, and cost efficiency compared to traditional Kafka solutions, allowing Fortis Games to retain data sovereignty and leverage existing cloud service discounts. By integrating Redpanda with other advanced tools like ClickHouse and Apache Flink, Fortis Games has established a modern analytics platform that empowers developers to enhance gameplay and maintain high player satisfaction.
Dec 29, 2025
1,515 words in the original blog post.
Out of memory errors are common challenges in database systems with vectorized query engines, particularly when handling large data fields such as JSON files in industries like cybersecurity. These engines process data in batches, often requiring substantial RAM, which can lead to errors when the memory capacity is exceeded. Traditional solutions involve adjusting block sizes, running queries on smaller data subsets, or using servers with higher RAM-to-CPU ratios. Oxla offers an innovative approach by automating batch size adjustments, ensuring that batch sizes remain within resource limits during query execution. This dynamic adjustment allows for efficient processing of large JSONs and strings, providing a more user-friendly experience without requiring developers to delve into database internals.
Dec 29, 2025
639 words in the original blog post.
In the digital era, analyzing clickstream data in real-time is crucial for businesses to optimize user experiences and gain insights into user behavior. The blog post discusses the complexities of building systems capable of processing large volumes of clickstream data in real-time and introduces a streamlined solution using DataSQRL, Redpanda, and AWS. DataSQRL serves as a compiler for data pipelines, allowing developers to use familiar SQL for defining data processing logic, which is then integrated with Apache Flink and PostgreSQL. Redpanda offers a high-performance streaming data platform compatible with Apache Kafka, ensuring reliable and efficient data processing, while AWS provides the necessary cloud infrastructure. The solution leverages Generative AI (GenAI) and large-language models to enhance personalization in recommendations. The tutorial demonstrates building a real-time clickstream recommendation system with only 45 lines of code, emphasizing how DataSQRL simplifies the development process by compiling SQL and GraphQL into a cohesive data pipeline. The post concludes by encouraging developers to explore the open-source code and join the communities of DataSQRL and Redpanda to further their understanding and application of these technologies.
Dec 29, 2025
3,767 words in the original blog post.
The tutorial provides a comprehensive guide on setting up a Change Data Capture (CDC) stream using Redpanda and Debezium, highlighting Redpanda's compatibility with the Kafka Connect ecosystem due to its API compatibility with Apache Kafka. It explains the process of recognizing data changes in a source system, enabling downstream systems to act on these changes, and outlines various use cases such as real-time analytics and safe migration from legacy systems. The tutorial walks through the setup of several services, including Redpanda, Kafka Connect, Debezium, and MySQL, using Docker and Docker Compose, and demonstrates how to configure Kafka Connect to monitor changes in a MySQL database and stream these changes to Redpanda. It concludes by encouraging users to experiment with CDC streams from other database systems and join the Redpanda Community on Slack for further engagement.
Dec 29, 2025
1,345 words in the original blog post.
Redpanda has introduced Redpanda Migrator, a tool designed to facilitate the migration of workloads from Apache Kafka® systems to Redpanda, emphasizing simplicity and efficiency in the process. The Migrator streamlines the transfer by reducing the number of components to manage and avoiding complex configurations, offering a single Go binary to migrate topics, schemas, consumer groups, and ACLs, with the added benefit of a single metrics endpoint for monitoring progress and health. Unlike the traditional MirrorMaker2 (MM2), which can be cumbersome due to its complex deployment and setup, Redpanda Migrator addresses common issues, including misconfiguration challenges and an incomplete metrics experience. The tool is integrated within Redpanda Connect and available to both existing and new users, with the latter able to try it for free through Redpanda Serverless. By using a timestamp-based approach for offset translation, the Migrator minimizes the risk of message duplication without complex logic, making it easier for developers to move Kafka messages and configurations seamlessly to Redpanda.
Dec 29, 2025
1,905 words in the original blog post.
This text discusses the challenges of managing and understanding data in modern organizations and introduces a solution involving the integration of Redpanda and OpenMetadata. It highlights the inefficiencies caused by inadequate metadata management, such as using incorrect data tables and unexpected schema changes, which can disrupt various teams like backend, data engineering, and data analysis. The article emphasizes the importance of breaking down knowledge silos and fostering collaboration through transparent metadata management, which involves ingesting metadata into OpenMetadata to centralize data discovery and collaboration. By setting up Redpanda, a fast Kafka API-compatible streaming data platform, with OpenMetadata, organizations can streamline metadata ingestion and management, allowing for better asset curation and context provision. The approach also includes automated versioning and change detection to minimize disruptions and enhance data lifecycle management, ultimately enabling data teams to generate more value and respond efficiently to data evolution.
Dec 29, 2025
1,375 words in the original blog post.
Redpanda Cloud has announced significant upgrades aimed at enhancing hardware performance and cost-efficiency, including a 50% reduction in VM costs and a doubling of supported partition counts. The update introduces ARM processor-based instances for better cost-effectiveness and support for AWS Graviton instances, which promise substantial savings in total cost of ownership. Additionally, a managed connector for Apache Iceberg simplifies integration with data lakes, allowing seamless analytics on streaming data, while a user-friendly schema registry management tool accommodates non-experts with ease. Customizable maintenance windows offer greater flexibility for IT teams to schedule updates during less disruptive times, ensuring the service remains available without interruption. These improvements are set to expand the scope of streaming data use cases and enhance the overall user experience in Redpanda Cloud's fully managed service.
Dec 29, 2025
1,038 words in the original blog post.
Redpanda has demonstrated a significant performance advantage over Apache Kafka across various workloads, being up to 10 times faster while requiring fewer nodes. The introduction of KRaft in the latest version of Kafka, which replaces Apache ZooKeeper with the Raft protocol for cluster metadata management, was assessed to see if it affects Redpanda's performance lead. Testing methodologies used include the OpenMessaging Benchmark project, with configurations ensuring data safety and security through fsync, TLS encryption, and SASL authentication. Redpanda's use of Raft for replication offers superior data safety compared to Kafka's ISR mechanism, which can incur data loss. Redpanda consistently outperformed Kafka in tests, including scenarios with KRaft, showing lower latencies and handling GBps+ workloads effectively on smaller hardware setups. Even when Kafka was configured without fsync to potentially improve speed, Redpanda maintained its performance edge. Despite Kafka's updates, Redpanda's architecture and efficient design continue to provide robust advantages in real-time streaming data applications.
Dec 29, 2025
1,777 words in the original blog post.
AdTech, a fusion of technology and advertising, involves tools and strategies to enhance digital ad transactions, with programmatic advertising as a crucial aspect that automates and optimizes ad placements via data-driven mechanisms. Central to this is real-time bidding (RTB), where advertisers and Demand-Side Platforms (DSPs) engage in rapid auctions for ad impressions, necessitating advanced architecture for performance and scalability. The programmatic ecosystem consists of advertisers, publishers, Supply-Side Platforms (SSPs), ad exchanges, and Data Management Platforms (DMPs), all working together to ensure precise ad targeting and efficient transactions. The proposed reference architecture for a DSP emphasizes low latency, scalability, and efficient data integration, leveraging technologies like Aerospike for user data management and Redpanda for real-time data enrichment and analytics. This design enables DSPs to make instantaneous, informed bidding decisions, enhancing ad targeting and maximizing return on investment, while maintaining compatibility with existing RTB protocols.
Dec 29, 2025
2,045 words in the original blog post.
Redpanda has introduced a new central configuration system to simplify the management of cluster configurations and minimize operational issues caused by local configuration file discrepancies. Instead of storing configuration in local text files, Redpanda's central system stores uniform cluster configuration properties internally, updating them via an API and replicating them across nodes using Raft, which eliminates the risk of inconsistent configurations. The rpk CLI tool facilitates editing these configurations in real-time, ensuring settings are applied consistently across the cluster without needing node restarts, unless specified. This centralized approach offers a single authoritative configuration source, enhancing reliability and preventing non-deterministic behaviors from node-specific configurations. The system is designed for seamless upgrades, automatically activating for existing users after all nodes are updated to Redpanda 22.1.x, and supporting live configuration changes to most properties. Additionally, it maintains compatibility with Apache Kafka by allowing certain properties to be modified using Kafka's API, although using rpk is recommended for a more robust DevOps experience.
Dec 29, 2025
1,196 words in the original blog post.
Redpanda is presented as a more cost-effective and high-performing alternative to Apache Kafka for data streaming, claiming to be up to six times cheaper and 10 times faster. The analysis outlines the total cost of ownership (TCO) by evaluating infrastructure, administration, and operational costs, revealing that Redpanda requires significantly fewer resources to maintain low latency across varying workloads. Key advantages of Redpanda include its single binary deployment, automatic partitioning, and leader balancing, which reduce administrative overhead compared to Kafka's more complex setup involving ZooKeeper and ISR mechanisms. Performance tests show that Redpanda maintains consistent latency even at high throughputs, whereas Kafka struggles to manage similar workloads without substantial scaling. The introduction of Redpanda Enterprise features such as tiered storage further enhances cost savings by optimizing data retention and offering additional functionalities for analytics and disaster recovery. As Kafka looks towards serverless configurations and automated tools to manage costs, Redpanda is already capitalizing on these trends by simplifying operations and reducing dependencies, thus providing a competitive edge in cost management and operational efficiency.
Dec 29, 2025
2,656 words in the original blog post.
Redpanda University offers a comprehensive range of interactive courses designed to help individuals master real-time data applications and streaming data technologies. The curriculum includes beginner to advanced courses covering fundamentals of event streaming, installation and usage of Redpanda, Kafka building blocks, and stream processing with Redpanda. Intermediate learners can explore the Redpanda ecosystem, cluster operations, and batch-to-streaming integration workshops, while advanced users can delve into algorithmic trading systems using Apache Flink. Each course is designed for developers, architects, and administrators, providing hands-on tutorials and practical applications in real-time data processing. Participants can earn completion certificates by registering for courses, which require the correct entry of personal details to ensure accurate documentation.
Dec 28, 2025
357 words in the original blog post.
Redpanda Keeper (rpk) is a versatile command-line interface tool designed for efficiently managing Redpanda clusters, offering a range of capabilities beneficial to both local and production environments. Written in Go, this single binary application simplifies installation and quickstart processes on various operating systems, including Linux and macOS, while utilizing WSL2 and Docker for Windows setups. It enhances cluster management by facilitating the administration of topics, consumer groups, and ACLs, allowing for operations such as creation, deletion, and configuration alterations with incremental updates. Additionally, rpk supports rich formatting for producing and consuming data, tuning host systems for optimal production workloads, and streamlining configuration management through a centralized system, thereby reducing the need for individual file management. These features collectively make rpk a valuable tool for developers working with Redpanda, supporting both evaluation and maintenance of streaming systems.
Dec 28, 2025
2,051 words in the original blog post.
Machine learning has revolutionized software development by enabling algorithms to learn from data, and frameworks like PyTorch and TensorFlow have simplified the development process with their extensive libraries and support systems. PyTorch is favored in research settings due to its straightforward, Pythonic API and dynamic computation graphs, which facilitate quick prototyping and debugging. Conversely, TensorFlow, with its support for static computation graphs and a broader range of programming languages, is better suited for large-scale production applications where resource optimization is crucial. TensorFlow's extensive community and resources make it a dominant player in the production environment, while PyTorch's dynamic capabilities have made it popular in academic circles. Both frameworks support streaming data applications and can integrate with Apache Kafka, though Redpanda offers a more efficient alternative for real-time data processing in machine learning workflows. Ultimately, choosing between PyTorch and TensorFlow depends on the specific needs of the project, whether it prioritizes research flexibility or production robustness.
Dec 28, 2025
2,270 words in the original blog post.
Google Cloud's introduction of BigQuery continuous queries addresses the challenge of moving real-time data from data warehouses to downstream platforms, eliminating the latency and operational overhead of traditional batch exports or change data capture methods. These continuous queries enable real-time data extraction from BigQuery, which can be routed through Google Pub/Sub to Apache Kafka® or Redpanda, facilitating the Reverse ETL pattern and providing immediate insights. By integrating with Redpanda, a Kafka-compatible streaming platform, businesses can efficiently manage large data volumes with familiar SQL syntax, enabling real-time applications such as personalization, anomaly detection, and analytics. This integration ensures that the latest data is always available for business actions by streaming continuous query results into Redpanda in real time, enhancing the ability to respond to new data as it arrives. Although the feature is currently in preview and subject to "Pre-GA Offerings Terms," it promises to maximize the utility of BigQuery for insights, especially for users already engaged in ETL workflows with Redpanda, and aims to close the data loop with Reverse ETL capabilities.
Dec 28, 2025
854 words in the original blog post.
BigQuery, introduced by Google in 2010, serves as a serverless, cost-effective, multi-cloud data warehouse that allows scalable analysis of large datasets using SQL-based queries. It provides an interface for third-party developers to access features of Google's internal Dremel technology via a REST API, command-line interface, and web UI, maintaining high query performance. The tutorial detailed in the text guides users through the process of integrating Redpanda, a Kafka-compatible data streaming platform, with BigQuery via Kafka Connect, to enable real-time data streaming and analysis. It outlines steps to set up and configure Docker containers for Redpanda, create a dataset and table in BigQuery, and configure a Kafka Connect cluster to facilitate data flow from Redpanda to BigQuery. Users are also guided to run SQL queries on BigQuery to analyze data, including finding specific user interactions with classified ads, demonstrating the practical application of these tools in a data analytics scenario. The tutorial emphasizes the ease of streaming and analyzing large volumes of structured data in real-time, highlighting the usefulness of integrating Redpanda and BigQuery for interactive analytics and business intelligence purposes.
Dec 28, 2025
3,412 words in the original blog post.
Redpanda's latest 24.3 release introduces a host of new features and updates aimed at enhancing data streaming, storage, and management capabilities. The beta of Iceberg Topics allows for seamless integration of streaming data with data lakes or warehouses without ETL, providing a unified approach for streaming and analytics workloads. Mountable Topics facilitate resource conservation by allowing unused topics in Tiered Storage to be safely hibernated and reactivated. Leader pinning optimizes latency and cost efficiency in geographically distributed clusters. New connectors for Redpanda Connect, including the Postgres CDC beta, improve data integration and quality control. Secrets management and offset mapping for Redpanda Migrator enhance compliance and migration efficiency. Additionally, the introduction of AI connectors and deep partner integrations, along with expanded availability through the Azure Marketplace and a new Terraform provider, demonstrate Redpanda's commitment to flexibility and scalability in data management. The release also highlights enhanced uptime guarantees and customizable maintenance windows for Redpanda Cloud, emphasizing its focus on reliability and user control.
Dec 28, 2025
1,671 words in the original blog post.
The Redpanda AI Integration and Innovation Hackathon, held from September 30th to October 28th, encouraged developers globally to devise creative solutions using Redpanda Data Transforms, Redpanda Connect, AWS, and other advanced technologies, emphasizing simplicity and real-world applicability. The winning project, Sovereign Structure, developed by Dan Goodman, offers enterprises a straightforward way to leverage AI without compromising data privacy by transforming unstructured data into structured JSON-schema format using large language models directly within Redpanda, thus keeping data within the enterprise's infrastructure. Goodman's background as co-founder of Tangia and his extensive experience with distributed systems and data streaming technologies informed his innovative approach, which highlighted the importance of effective communication in conveying the value of technical solutions. His project underscores the growing enterprise demand for AI solutions that ensure data sovereignty, providing a competitive edge in the market by allowing seamless integration with existing data streams.
Dec 28, 2025
697 words in the original blog post.
Elasticsearch is an open-source distributed search and analytics engine built on Apache Lucene, designed to handle various data types, including textual, numerical, and geospatial data. It is widely used by companies like Wikipedia, GitHub, and Facebook for tasks such as full-text search, analytics, and document storage. The tutorial explains how to integrate Elasticsearch with Redpanda, a high-performance streaming platform compatible with the Apache Kafka API, using Kafka Connect and compatible connectors. By setting up a real-time data streaming pipeline, users can efficiently index and search data, as demonstrated with a fictional news company, PandaPost, which requires fast and efficient text search capabilities for incoming news reports. The process involves running Elasticsearch and Redpanda using Docker, configuring Kafka Connect for data integration, and employing Elasticsearch for real-time data indexing and retrieval.
Dec 28, 2025
3,409 words in the original blog post.
In the second part of the chat app tutorial using Redpanda and Quix, the focus is on updating the frontend to integrate with the backend changes from part one, which included implementing a profanity filter. The tutorial guides users through adapting the frontend service to connect with new data from the "messages_sanitized" topic, stored either in Redpanda Cloud or Quix Cloud. It introduces SignalR, a .NET library facilitating real-time communication, which is integrated into the Quix workspace, eliminating the need to install a separate SignalR Hub. The steps involve configuring a bearer token for authentication, setting up environment variables, and updating several Angular source files to render censored messages in the chat interface. The tutorial concludes with instructions for testing the updated chat application and invites users to explore further resources or seek help through Quix and Redpanda communities.
Dec 28, 2025
1,583 words in the original blog post.
Redpanda Connect is now available for Bring Your Own Cloud (BYOC) deployments and offers a beta version for serverless environments, providing a robust framework for seamless data integration and processing across various platforms. With over 100 out-of-the-box connectors and a new Kafka Migration tool, it simplifies real-time data workflows for developers and enterprises, enabling them to focus on innovative solutions rather than the complexities of distributed streaming platforms. Redpanda Connect is built using Golang and supports a declarative approach with YAML configurations, making it user-friendly and efficient for developers. Its capabilities include handling multi-cloud data integration, facilitating low-latency data processing to cloud storage like Amazon S3, and enhancing AI workflows by transforming unstructured data for AI models, making it a versatile tool for businesses of all sizes. The platform's scalability and developer-first design ensure that it can manage everything from small integrations to complex data pipelines, thus addressing common challenges in modern data environments.
Dec 28, 2025
825 words in the original blog post.
Data processing transforms raw digital data, generated through human-computer interactions, into actionable insights, significantly impacting industries like IoT, stock trading, and music streaming. The process involves using tools like Apache Spark, Amazon S3, and Redpanda to build efficient data processing pipelines. Apache Spark is an open-source analytics engine that supports SQL analytics, data science, and machine learning operations, offering high computational speed and compatibility with various programming languages. Amazon S3 provides secure and scalable object storage, allowing data management and access optimization, while Redpanda, a streaming data platform compatible with Kafka, simplifies data processing tasks. A practical example is demonstrated through a tutorial on building a data processing pipeline for a hypothetical music streaming service, PandaMusic, which ingests audio files and extracts features for analysis. The tutorial guides setting up a pipeline with Redpanda, Apache Spark, and Amazon S3, involving creating a Streamlit interface for uploading files, processing them into mel spectrograms, and storing them in an S3 bucket. The tutorial also covers setting up necessary software, creating helper functions for data processing, and running the pipeline, with additional resources available through the Redpanda GitHub repository and community.
Dec 28, 2025
3,263 words in the original blog post.
Change data capture (CDC) is a design pattern that tracks and audits changes in databases, making it possible to process data in real-time and improve operational insights. The text discusses the implementation of CDC in a MySQL database using the "binlog" change log functionality, facilitated by tools like Debezium, Kafka Connect, and Redpanda. This approach allows changes to be captured and processed instantly, supporting applications such as e-commerce platforms that require real-time updates on payment methods. The article highlights CDC's advantages over traditional extract-transform-load (ETL) methods, emphasizing its ability to reduce system load and handle hard-deletion scenarios efficiently. It also provides a step-by-step guide for setting up a CDC pipeline with MySQL, Debezium, and Redpanda, including instructions on configuring Docker containers and leveraging schema registries to ensure data consistency. By adopting CDC, organizations can design real-time dashboards and reports for better data visibility, with the flexibility to integrate with various tools and platforms.
Dec 28, 2025
2,620 words in the original blog post.
The comparison between RabbitMQ and Redpanda highlights their distinct functionalities and use cases in distributed systems architecture. RabbitMQ is an open-source messaging platform based on the AMQP protocol, known for its lightweight and fast nature, emphasizing smart brokers and passive clients. It excels in scenarios that require rule-based routing, scheduling, and sophisticated error handling, but its performance can degrade with increased durability and message size. Conversely, Redpanda is a streaming data platform similar to Apache Kafka, offering high scalability and resilience, with the ability to persist and replay data, supporting sophisticated event-driven architectures through smart clients and simple broker designs. This makes Redpanda suitable for environments requiring a central data bus for replaying business events, with a focus on horizontal scalability and client-side intelligence. Ultimately, the choice between RabbitMQ and Redpanda depends on specific architectural requirements such as message size, throughput, scalability, and the desired level of client versus broker intelligence, with some applications potentially benefiting from a combination of both platforms.
Dec 28, 2025
3,228 words in the original blog post.
In the realm of digital advertising, real-time ad performance analytics is essential for optimizing campaigns, maximizing return on investment, and delivering personalized content. The proposed reference architecture leverages Redpanda for streaming data ingestion, Apache Flink for streaming ETL, and Apache Pinot for OLAP queries, ensuring speed, reliability, and accuracy in processing ad events. Redpanda offers scalable and high-throughput data ingestion, Flink provides low-latency processing and exactly-once event handling, and Pinot enables quick analytical queries on massive data sets. This architecture supports real-time dashboards and data products, allowing advertisers to swiftly adjust to market changes and accurately measure ad effectiveness. Additionally, the integration with data warehouses and data lakehouses facilitates reporting, ad-hoc queries, and machine learning applications, providing a comprehensive solution for the advertising industry.
Dec 28, 2025
1,373 words in the original blog post.
Enterprise application integration focuses on message patterns to enable communication between diverse software systems across various environments through standard messaging formats and protocols. Messaging patterns describe how components in a message-passing system connect and communicate, utilizing messaging channels that allow applications to send and receive information. A key tool for implementing these patterns is Apache Camel, an open-source integration framework that supports the creation of routes and mediation rules using various domain-specific languages. Apache Camel integrates well with messaging systems like Redpanda, which serves as a messaging channel similar to Kafka but with enhanced performance and safety. The article details a practical tutorial on implementing enterprise integration patterns (EIPs) using Apache Camel and Redpanda, specifically focusing on the dead letter channel pattern to handle messages that cannot be processed. Through a step-by-step guide, it demonstrates setting up a project structure, creating necessary scripts, and configuring Java-based applications to process messages, illustrating the application of these patterns in real-world scenarios.
Dec 28, 2025
3,202 words in the original blog post.
Real-time data processing serves as a vital tool for organizations to manage and analyze the continuous influx of data generated from various sources like website clicks, fintech transactions, sensor readings, and social media interactions. Unlike batch processing that handles data in periodic chunks, real-time processing enables immediate analysis and insight generation, helping businesses stay agile, enhance customer experiences, and maintain a competitive edge. Key components of this process include event streaming platforms, stream processing engines, and real-time databases, each playing a crucial role in managing data streams and producing actionable insights. Practical applications span across industries, from personalized recommendations in retail and dynamic pricing in eCommerce to real-time fraud detection in banking and operational logistics in supply chains. In healthcare, real-time data from patient monitors and diagnostic tools can lead to faster and more accurate medical interventions. Redpanda's streaming data platform exemplifies a user-friendly solution for implementing real-time data processing, offering compatibility with Kafka APIs and simplifying the management of data streams without the complexity of traditional systems.
Dec 28, 2025
775 words in the original blog post.
A recent technical discussion with LinkedIn's storage experts focused on the intricacies of Apache Kafka® and Apache Samza®, delving into memory pressure issues related to per-core data structure materialization. Despite minimal preparation for the presentation, the conversation naturally transitioned into technical specifics, underscoring a shared enthusiasm for technology. The session concluded with a 30-minute Q&A, encouraging further engagement through Community Slack and Twitter. Special appreciation was extended to Noah, Sarah, and Denis for their last-minute assistance in preparing the presentation.
Dec 28, 2025
132 words in the original blog post.
An engine of record is a scalable data storage solution that can adapt to varying data volumes without requiring an overhaul of the existing architecture. The text explains how to construct an engine of record using Redpanda, BigQuery, and Apache Spark, emphasizing the importance of integrating these tools for low latency, fast computation, and data integrity. It provides a detailed tutorial on setting up Redpanda with Docker, connecting Apache Spark to BigQuery, and streaming data from Redpanda to BigQuery for real-time processing. The process includes creating a Redpanda cluster, producing data streams, setting up a BigQuery project and dataset, and configuring Apache Spark with appropriate dependencies and authentication. The tutorial illustrates the practical application of the engine of record in scenarios like food ordering systems and IoT sensor data management, demonstrating how real-time data streaming can efficiently handle large volumes and dynamic data needs. The tutorial also provides guidance on running the setup and validating outputs in BigQuery, with resources and code available for reference.
Dec 28, 2025
2,482 words in the original blog post.
The tutorial outlines a process for automating and managing bookstore inventory data using cloud storage and streaming technologies, specifically Amazon Web Services (AWS) and Google Cloud Platform (GCP). It describes setting up Amazon S3 and Google Cloud Storage (GCS) for storing data from two bookstore branches in London and New York, respectively. The tutorial involves configuring Redpanda, an Apache Kafka-compatible streaming platform, and Kafka Connect to integrate with these cloud storage services. The scenario demonstrates creating Kafka topics for each branch's inventory, using Redpanda to manage data streams, and configuring sink connectors to transfer data to the appropriate cloud storage. The process ensures that data is automatically updated and versioned, reducing errors and improving efficiency in handling the bookstore's inventory. The tutorial is designed to be a practical guide for integration engineers looking to streamline data management processes using modern cloud and data streaming tools.
Dec 28, 2025
3,723 words in the original blog post.
Redpanda 23.3 introduces the capability to perform stateless data transformations within its broker using WebAssembly (Wasm), enabling tasks such as format transcoding, data validation, filtering, and redaction of sensitive information directly within Redpanda. A new feature, Data Transforms, allows users to embed data processing without relying on external platforms, and it includes a pre-built library for config-driven data redaction, which is essential for handling sensitive data in compliance with regulations like GDPR and HIPAA. The process of redaction is simplified with a framework that offers a variety of built-in actions for modifying sensitive data fields, and the introduction of a no-code solution makes it accessible to non-developers. Users can deploy redaction transforms by configuring redaction rules, creating output topics, and running deployment scripts. The redaction framework is extensible for custom use cases, and a demonstration using Docker Compose shows the practical application of these transformations, highlighting how Redpanda facilitates secure and efficient data management.
Dec 28, 2025
1,213 words in the original blog post.
Redpanda is a drop-in replacement for Kafka that focuses on compatibility, performance, and correctness for mission-critical workloads, and it leverages code generation and type safety to improve software reliability. The engineering team ensures compatibility with existing Kafka clients by treating any communication failure as a critical bug, while simultaneously optimizing performance through low-level design patterns and system features. As the Kafka protocol consists of numerous endpoints with multiple versions, Redpanda employs code generation to automate the creation of detailed network protocol parsers, thereby reducing human error and allowing engineers to focus on performance improvements. Additionally, Redpanda utilizes strong typing in C++ to prevent common errors associated with parameter passing, enhancing reliability and allowing for more confident low-level optimizations. These practices provide the foundation for Redpanda’s ongoing development and innovation in building a robust Kafka alternative.
Dec 28, 2025
1,964 words in the original blog post.
Change data capture (CDC) is an essential component of modern data architectures, enabling organizations to monitor and react to data changes in real time. This functionality supports real-time analytics, event-driven applications, and seamless data replication across different environments. The tutorial provides a comprehensive guide on implementing a CDC system using PostgreSQL and Redpanda Connect, specifically for a retail company aiming to establish a real-time inventory tracking system. The process involves setting up a PostgreSQL database to capture inventory changes, streaming these changes to Redpanda topics, and configuring security and access controls for data handling. The tutorial also includes steps for creating a Python application to simulate inventory data updates and verify the CDC setup's efficacy by consuming data from the Redpanda topic. The integration of CDC with Redpanda Connect demonstrates the ease and effectiveness of bridging traditional databases with modern streaming platforms for efficient data processing and real-time insights.
Dec 28, 2025
2,361 words in the original blog post.
Emerging data sovereignty requirements are reshaping cloud deployment strategies, leading to the development of the Bring Your Own Cloud (BYOC) model, which combines the control and compliance of self-hosting with the operational benefits of fully managed SaaS offerings. BYOC allows organizations to maintain their data within their own virtual private cloud (VPC) while a vendor manages the infrastructure remotely, offering a solution to the complex issue of ensuring data sovereignty in cloud environments. This approach provides greater visibility and control over data than traditional SaaS models, while still enabling access to expert management of infrastructure operations. As data regulations become more stringent and organizations increasingly rely on real-time data streaming, BYOC offers a balanced solution that addresses both compliance and operational efficiency, making it a compelling option for platform engineering teams grappling with the challenges of data sovereignty and cloud management.
Dec 28, 2025
1,006 words in the original blog post.
Jump Trading, a proprietary trading firm specializing in algorithmic and high-frequency trading, has adopted Redpanda as its streaming data platform to support its mission-critical telemetry pipeline. After evaluating various messaging systems, Jump Trading chose Redpanda for its simplicity, performance, and open-source compatibility. Unlike Apache Kafka, Redpanda's implementation in C++ aligns with the firm's software architecture preferences and eliminates Java dependencies, enhancing performance and reducing complexity. Redpanda offers a Kafka-compatible API, enabling seamless integration with existing open-source ecosystems and providing flexibility without vendor lock-in. The platform's efficient architecture supports low-latency data processing, crucial for Jump Trading's latency-sensitive operations, and is deployed across bare metal, containerized environments, and fully-managed cloud services. Redpanda's source-available code aligns with Jump Trading's open-source values, facilitating in-depth understanding and troubleshooting. Over two years, Redpanda has reliably handled billions of messages daily, playing a pivotal role in the firm's global financial market leadership.
Dec 28, 2025
1,115 words in the original blog post.
Change Data Capture (CDC) is a technique for capturing change events from a source database, which can then be used for various downstream applications like triggering events and analytics. The tutorial demonstrates how to set up a real-time CDC pipeline between ScyllaDB, a high-performance NoSQL database, and Redpanda, using a Debezium CDC connector compatible with Kafka Connect. This setup involves configuring Docker to run instances of ScyllaDB and Redpanda, setting up a standalone Kafka Connect cluster, and enabling CDC on ScyllaDB to capture changes in an e-commerce orders table. The tutorial guides users through inserting, updating, and tracking data changes in the orders table and accessing these changes through the Redpanda Console. The integration allows for seamless data streaming, enabling various use cases like asynchronous messaging between systems and data integration for creating data lakes.
Dec 28, 2025
2,218 words in the original blog post.
Redpanda is a high-performance streaming data platform designed for low-latency data ingestion, which is compatible with Kafka APIs, allowing seamless integration with existing Kafka-based stream processing technologies. It supports a variety of stream processors, such as Kafka Streams, ksqlDB, Apache Flink, Spark Structured Streaming, Materialize, Bytewax, Timeplus, Faust, and Benthos, each offering unique features for processing and analyzing data streams. These processors enable real-time analytics, anomaly detection, fraud detection, content recommendations, and machine learning model inferencing by consuming data streams, applying transformations, and outputting results to Redpanda or other systems. Redpanda's operational simplicity and compatibility with Kafka make it an attractive choice for scalable stream processing workloads, with ongoing support and community engagement available through documentation and a Slack community.
Dec 28, 2025
1,786 words in the original blog post.
Agentic AI pipelines, essential for creating next-generation intelligent systems, present unique challenges in governance and observability compared to traditional machine learning pipelines. These pipelines involve autonomous agents capable of perceiving their environment and taking actions dynamically, requiring a robust infrastructure to safely transition from prototypes to production-grade systems. Redpanda's agentic AI platform, the Agentic Data Plane, addresses these challenges by providing governed data orchestration without complex external tools, ensuring connectivity, security, and comprehensive audit trails through its real-time event bus and Redpanda Connect framework. This architecture allows seamless automation and secure agent actions across diverse environments, making it suitable for applications like AI-driven customer support systems. Redpanda's approach eliminates the need for additional workflow engines, leveraging its own systems to manage data flows and agent interactions effectively, paving the way for smarter, safer AI pipelines.
Dec 16, 2025
1,283 words in the original blog post.
The rise of the Internet of Things (IoT) has expanded the scope of networks from merely connecting computers and servers to integrating a wide array of devices such as sensors, appliances, and vehicles, necessitating real-time data communication. This evolution has fostered the adoption of complex event processing (CEP) frameworks across various sectors like healthcare, manufacturing, and agriculture. Modern data pipelines for IoT and event data typically involve components such as data sources, ingestion layers, processing layers, storage layers, and output/integration platforms. Tools like ClickHouse and Snowflake are utilized for their complementary strengths in real-time analysis and scalable storage, respectively. Redpanda and Redpanda Connect facilitate the seamless integration and streaming of data between these systems, offering robust solutions for building data pipelines that cater to both short-term analysis and long-term retention. The guide highlights the importance of schema design and optimization, especially in time-series data contexts, and demonstrates how real-time processing and historical analysis can be achieved using these technologies, with applications spanning from healthcare monitoring to logistics and smart city infrastructure.
Dec 09, 2025
3,384 words in the original blog post.
Redpanda Connect enables the construction of streaming data pipelines through a straightforward YAML configuration, offering over 300 pre-built connectors for data movement and transformation. As organizations scale and pipeline usage increases, transitioning from experimental to operationalized platforms becomes necessary, where GitOps plays a crucial role. By managing Redpanda Connect on Kubernetes declaratively via Git and using tools like Argo CD, organizations can achieve enhanced security, auditability, and efficient deployments. The blog explains deploying Redpanda Connect using GitOps in two modes—Standalone, for isolated and predictable workloads, and Streams, for flexible and multi-tenant environments—by utilizing Helm and Kustomize for dynamic configurations. With GitOps, changes are tracked and synchronized automatically, ensuring a consistent and reproducible setup, while observability is maintained via Prometheus and Grafana for real-time monitoring. This approach reduces manual operations, fosters predictability, and aligns with modern software development practices, paving the way for Redpanda Connect to become a central data platform capability.
Dec 02, 2025
2,366 words in the original blog post.