July 2023 Summaries

The "Meet the Residents of Onehouse" series introduces Andy Walner, a Product Manager at Onehouse, who shares insights into his career and experiences that led him to join the company. Andy initially learned about Onehouse through a conversation with the Head of Product about the company's innovative work, which aligned with his passion for building products in emerging categories. With a background in launching a data labeling network at Google and leading product development at Dashworks, Andy brings valuable entrepreneurial experience to Onehouse as it pioneers the data lakehouse category. He was drawn to Onehouse by its team-oriented culture, the opportunity to work with cutting-edge open-source technology, and the company's commitment to solving important data challenges. Andy appreciates the learning culture at Onehouse, where collaboration with industry veterans enhances their ability to provide value to customers. Motivated by the dynamic startup environment, he values the team mentality and the "Tough & Persevering" culture, which emphasizes resilience and positivity. As Onehouse continues to grow in 2023, Andy is excited about the expansion of the team and product capabilities, aiming to make a significant impact across various industries.

Ensuring data quality is crucial for organizations to maintain accurate, consistent, complete, and reliable data, which is essential for trustworthy analysis and decision-making. Apache Hudi, a data lakehouse technology initially developed at Uber, offers a feature called pre-commit validation to enhance data quality by validating data before it is committed to storage. This feature allows users to enforce data quality checks such as uniqueness constraints, schema compliance, and adherence to business rules during the data writing process, using tools like DeltaStreamer or Spark Datasource writers. Users can configure these validators through the `hoodie.precommit.validators` setting, choosing from built-in validators or creating custom ones to meet specific needs. Apache Hudi also provides a SQL-based pre-commit validator to check specific data conditions, and users can extend the SparkPreCommitValidator class to define custom validation logic. By leveraging these pre-commit validators, organizations can ensure data integrity and accuracy, thereby enhancing the reliability of their data-driven applications and analytics processes.

Onehouse has partnered with Confluent to integrate Confluent's data streaming platform with Onehouse's managed data lakehouse, enhancing data ingestion and transformation capabilities for users. This collaboration introduces a new feature that automates real-time database replication with Change Data Capture (CDC), allowing for seamless analytics on live data. Through this integration, companies like the fictitious FinCo can quickly set up a fully-functional analytics streaming system, replicating data from databases like Postgres into a data lakehouse for comprehensive analysis. The process is streamlined by Onehouse's automation of resource management within Confluent accounts, facilitating a maintenance-free database replication experience and enabling users to leverage Confluent's additional streaming services. This partnership aims to expand and deepen the integration for a wide range of data streaming use cases, further benefiting users with enhanced real-time analytics capabilities.

Data partitioning is a crucial strategy for enhancing query performance, scalability, and manageability, yet it requires careful design to avoid adverse effects. The blog explores how different databases and data warehousing systems, like Vertica, Teradata, Redshift, BigQuery, and Snowflake, implement partitioning to distribute data, improve performance, and manage availability. It highlights the evolution of data partitioning on data lakes, emphasizing the shift from row-based to column-based file formats like Apache Parquet and Apache Orc, which support better query performance through horizontal and vertical partitioning. The text discusses common pitfalls in data partitioning, such as mismatched partitioning schemes with query shapes, overly granular partitioning leading to small file problems, and inconsistent partitioning practices that can degrade query performance. The blog stresses the importance of choosing appropriate partition granularity and maintaining consistency across datasets, suggesting that modern data storage technologies offer opportunities to refine traditional partitioning approaches to ensure predictable performance and cost efficiency.

The recent Snowflake Summit and Databricks Data + AI Summit highlighted significant advancements in data lakehouse technology, with Snowflake improving the performance of unmanaged Iceberg tables and Databricks introducing Delta UniForm to enhance compatibility between Delta Lake, Apache Hudi, and Apache Iceberg. Despite the progress, the proprietary nature of Delta UniForm's deployment within Databricks raises concerns about limited interoperability and vendor lock-in. In response, Onehouse advocates for Onetable, a project aiming for omni-directional interoperability among the three major data lakehouse formats, promoting a collaborative, vendor-neutral approach to foster broader compatibility and innovation in the data community. This approach seeks to alleviate the fragmented nature of the data lakehouse ecosystem and the associated analysis paralysis, ultimately supporting a more cohesive and versatile data framework that can complement data warehouses effectively.

Onehouse provides a lakehouse-as-a-service solution that enhances machine learning (ML) capabilities by seamlessly integrating with major ecosystems like Amazon Sagemaker. This open and resource-efficient platform allows companies to transform their data lakes into fully managed lakehouses, optimizing data management services such as performance, ingestion, and quality. By using Apache Hudi technology, Onehouse facilitates interoperability and supports powerful querying through engines like Amazon Athena. In the context of Amazon Sagemaker, data scientists can leverage Onehouse to train and test ML models efficiently, using it as a centralized data store that supports continuous improvement and real-time data inflow. The integration between Onehouse and Sagemaker showcases how businesses can enhance their analytics workflows by creating a unified data lakehouse, thus optimizing costs and improving time-to-value for their ML projects.