July 2025 Summaries

Streamkap offers a streamlined approach to real-time data processing by enabling swift integration between AWS DynamoDB and MotherDuck, facilitating immediate data insights crucial for business decision-making. This guide provides detailed instructions on setting up and configuring AWS DynamoDB for compatibility with Streamkap, including creating IAM roles, setting up S3 buckets, and configuring DynamoDB Streams and Point-in-Time Recovery. It also covers the creation of MotherDuck warehouses and connecting these systems via Streamkap, emphasizing the security and efficiency of data management through encryption and access control measures. By simplifying the data pipeline setup, Streamkap allows businesses to efficiently manage data flow, ensuring scalability and high performance in data-intensive applications.

The text provides a comprehensive guide on using Streamkap, a high-performance Change Data Capture (CDC) solution, to stream data in real-time from AWS-hosted SQLServer to Databricks. It details the steps required to configure and set up a robust data pipeline that ensures low-latency and reliable data transfer for analytics platforms, eliminating the need for complex Extract, Transform, Load (ETL) workflows and batch delays. The guide covers prerequisites such as setting up accounts for AWS, Databricks, and Streamkap, and offers detailed instructions for configuring new or existing AWS RDS SQLServer instances for Streamkap compatibility. Additionally, it explains the process of setting up a Databricks account, creating a SQL data warehouse, and fetching necessary credentials. The guide emphasizes the importance of secure connections, proper configuration of SQL commands, and the use of Streamkap's features to establish a seamless connection between AWS RDS SQL Server and Databricks, enabling efficient real-time data analysis and operational decision-making.

Modern data teams are increasingly tasked with ensuring that operational data, such as that stored in AWS-hosted MySQL databases, is readily available in analytics platforms like Databricks with minimal delay. This blog post provides a detailed guide on using Streamkap, a high-performance Change Data Capture (CDC) solution, to facilitate real-time data streaming from AWS MySQL to Databricks, thereby supporting live dashboards, real-time machine learning, and key business metric monitoring. The guide covers the necessary configuration and setup steps, such as setting up AWS RDS MySQL and Databricks workspaces, ensuring Streamkap compatibility through specific MySQL parameter adjustments, and establishing data pipelines. Emphasizing the ease of integrating Streamkap for efficient and reliable data flow, the guide walks users through the process of creating source and destination connectors, ultimately enabling sub-second latency streaming for real-time insights.

Streamkap is a data streaming platform that facilitates real-time data pipelines, offering an intuitive way to sync production applications with data platforms. The guide details how to use Streamkap to connect AWS RDS SQL Server with ClickHouse, eliminating complex ETL workflows and batch processing delays. It includes step-by-step instructions for setting up AWS RDS SQL Server and ClickHouse for Streamkap compatibility, creating a data pipeline, and ensuring Change Data Capture (CDC) is enabled for tracking data changes. After setting up the necessary accounts and configurations for both AWS RDS and ClickHouse, users are guided through creating and managing schemas, tables, and users to facilitate seamless data streaming. Additionally, the guide covers setting up destination connectors in Streamkap to relay real-time data to ClickHouse and provides insights into handling schema evolution in SQL Server to ensure changes are reflected in ClickHouse, maintaining data consistency and integrity.

Data teams often require seamless and low-latency integration between operational databases and analytics platforms, which this blog addresses by demonstrating how to use Streamkap, a Change Data Capture (CDC) solution, to transfer data from AWS-hosted MySQL to Google BigQuery in real-time. The guide provides step-by-step instructions for setting up a new or existing AWS MySQL instance, configuring it to work with Streamkap, and establishing a connection with Google BigQuery for efficient data storage and analysis. It includes detailed processes for creating necessary accounts, configuring parameter groups for compatibility, and setting up streaming pipelines to ensure data is continuously synchronized without the need for complex ETL workflows. This setup allows organizations to access real-time insights and make informed decisions swiftly, enhancing their business intelligence capabilities.

SpotOn transitioned from a batch ingestion process to using Streamkap for synchronizing data from MongoDB to Snowflake to improve data latency and reduce maintenance costs. The migration involved refactoring over 2,000 dbt models to accommodate the new data source, ensuring data validation, and efficiently handling Change Data Capture (CDC). By moving to Streamkap, SpotOn achieved ultra-low latency, allowing data to be available almost in real-time, and reduced both ingestion and compute costs. Refactoring dbt models included updating source references, validating data types, and efficiently managing CDC data. The process also involved integrating historical data from existing dbt snapshots with new data from Streamkap using techniques like Common Table Expressions (CTEs) and UNION ALL operations to maintain historical integrity. The migration simplified data pipelines, reduced infrastructure costs by threefold, and improved the ability to track record changes, ultimately enhancing both internal analytics and customer-facing reporting capabilities.

Two new features have been introduced to enhance connector reliability and accommodate more database environments: Read Only Snapshot Support and Read Only Heartbeat Events. The Read Only Snapshot Support enables connectors to operate in a read-only mode, offering zero-impact backups, cost efficiency, and faster test environments, making it ideal for read replicas or write-restricted databases. This feature allows snapshots to be taken without modifying the source database and does not require signal tables or locks, ensuring safe usage in read-only or compliance-sensitive settings. The Read Only Heartbeat Events keep connectors responsive and active even when source tables are inactive, providing better visibility and faster issue detection, which is particularly beneficial for low-activity sources or intermittent traffic. Both features are now available across all connectors, and feedback is encouraged.