January 2025 Summaries

At Redpanda Streamfest, a session showcased the concept of Sovereign AI, emphasizing data security by keeping AI processing within local networks instead of sending data to external models. The demonstration involved building an e-commerce application using AI and Redpanda Connect, illustrating how to trigger personalized follow-up emails based on customer purchases without relying on public APIs. The process involves using Redpanda Connect to manage data ingestion into ClickHouse, compute vector embeddings, and perform vector searches to identify similar products for personalized marketing emails. The pipeline ensures transaction processing in real-time, utilizes local GPUs, and includes safety checks with LlamaGuard to ensure email content quality before sending, all while maintaining data sovereignty. The session highlighted Redpanda Connect's capabilities as a full-stream processor, supporting transactional guarantees, automatic retries, and extensive plugin options, with the entire setup available for experimentation through GitHub resources and further community engagement encouraged via Redpanda's Slack.

As businesses increasingly depend on real-time data for decision-making, integrating MongoDB with Redpanda using Redpanda Connect offers a seamless and efficient solution for real-time data streaming and processing. This integration is particularly beneficial for industries such as AdTech, SaaS, AI, and gaming, where real-time data ingestion can enhance user personalization, improve experiences, and optimize performance. The tutorial provides a step-by-step guide to setting up a real-time analytics pipeline by linking MongoDB with Redpanda Serverless, illustrating how to configure topics, collections, and security settings to facilitate data flow from MongoDB to Redpanda. By employing Redpanda Connect, users can simplify the complexities typically associated with traditional streaming platforms like Kafka, thus enabling streamlined, high-performance data streaming tailored to diverse real-world applications.