April 2023 Summaries

Onehouse is a cloud-native, managed data lakehouse platform available on the AWS marketplace that simplifies the process of building end-to-end analytics pipelines by automating data management tasks and offering enterprise-level security features. As demonstrated in a real-life use case involving Github analytics, Onehouse enables users to quickly set up data lakes, ingest and transform data, and conduct real-time analysis using tools like Amazon Athena and Preset. The platform supports incremental ETL transformations and provides built-in optimizations through Apache Hudi, including clustering, compaction, and cleaning, to enhance performance and reduce costs in large-scale data environments. By syncing metadata with AWS Glue, Onehouse facilitates seamless querying and visualization of data, making it a valuable tool for organizations looking to derive insights without the overhead of managing complex data infrastructures.

Incremental processing is a crucial technique in data systems for efficiently managing large data volumes by processing data subsets separately, leading to improved data freshness and reduced resource usage. Designing custom incremental pipelines requires careful consideration of factors such as data size and latency, along with robust monitoring and data validation to ensure consistency. Apache Hudi offers a streamlined alternative, equipped with built-in incremental processing capabilities, including a Change-Data-Capture (CDC) feature from version 0.13.0, which supports detailed data changes for enhanced downstream processing. This feature logs before-and-after images of data changes and supports complex operations like incremental joins and CDC processing in Spark streaming applications. The blog provides detailed code examples to illustrate these concepts and highlights the operational efficiencies gained by using Hudi for incremental data processing, citing Uber's implementation as a case study.

Onehouse is now available on the AWS Marketplace, simplifying access for customers to build a modern data lakehouse using AWS services such as S3, EMR, Athena, Glue, and Redshift. The managed data lakehouse product offered by Onehouse combines the benefits of data lakes and warehouses, providing a turnkey solution that is no-code, easy to manage, and cost-effective, capable of scaling with business needs while handling any workload. It supports continuous data ingestion from various sources, automates performance analytics tasks, and allows for the development of low-code incremental pipelines that synchronize with data catalogs like AWS Glue. Onehouse's innovative multiplexed streaming architecture reduces cloud costs compared to DIY Spark pipelines, and the platform is built on Apache Hudi technology, allowing integration with data warehouses like Redshift or Snowflake, data lake query engines like EMR or Databricks, and real-time analytics solutions like StarRocks or ClickHouse. This accessibility is aimed at businesses of all sizes, from startups to large enterprises, to facilitate their transition to a modern data lakehouse architecture.