July 2021 Summaries

Dremio is a lakehouse platform built on open-source Apache Arrow, designed to enhance the performance and scalability of data lakes by providing efficient SQL query management and direct access, distinguishing itself with high-speed query capabilities and resource optimization. This platform integrates seamlessly with cloud ecosystems and is compatible with various data sources, offering features like intelligent resource scaling, columnar data caching, and query acceleration through reflections. Reflections, which create optimized representations of base data, significantly reduce query times, and when used with Apache Superset, they contribute to a highly responsive analytics experience. Superset's customizable caching and dashboard-native filters further enhance the user experience by optimizing data retrieval and enabling efficient data management. Overall, Dremio's capabilities position it as a competitive alternative to first-party cloud data lake services, demonstrating the potential of third-party open-source solutions in the data lakehouse landscape.

Atlassian's acquisition and subsequent shutdown of the proprietary BI platform Chart.io highlights a trend of consolidation in the business intelligence market, with several major acquisitions in recent years. In contrast, the open-source BI platform Apache Superset, which has become a popular tool among modern data organizations like Airbnb and Lyft, offers flexibility and resilience against such consolidations. Superset provides a range of features, including a no-code query builder, an extensive set of visualizations, and a powerful SQL editor, with the added advantage of a semantic layer for data modeling. Preset, built on Apache Superset, enhances the open-source platform with enterprise features such as role-based access control, single sign-on, managed private cloud deployment, and compliance with SOC2 standards. It has grown significantly, becoming a major contributor to the Superset project and offering a scalable, secure, and user-friendly experience. Preset's continuous development and support make it an attractive option for organizations seeking a robust open-source BI solution.

Apache Superset 1.2 introduces several new features aimed at enhancing data visualization and exploration capabilities for end-users. This release includes the addition of new chart types such as mixed time-series and radar charts, which allow for more complex data visualizations by combining various metrics and categories. The pivot table has been redesigned to incorporate heatmap and bar chart visualizations, addressing frequent user requests for more dynamic data presentations. Additionally, the platform now supports CrateDB and Databricks Cloud as new data sources, expanding its ability to query diverse datasets. Notably, this version introduces these enhancements without any backward-incompatible changes, consistent with the community's semantic versioning approach.

Data technology is rapidly evolving, leading to challenges in keeping up with the multitude of new innovations such as databases, ETL tools, and analytics platforms. This discussion focuses on clarifying the distinctions between real-time databases, time series databases, and real-time analytics, all of which operate under specific performance and data handling paradigms. Real-time databases are designed to provide swift read/write operations within milliseconds to seconds, making them ideal for applications like fraud detection and gaming. Time series databases are optimized for managing sequential data over time, with applications in monitoring weather patterns and stock market fluctuations, and can sometimes offer real-time capabilities. Real-time analytics involves an entire data stack that ensures low-latency performance from data ingestion to analysis, suited for operational needs like ride-sharing services but often comes with high costs and complexity. The interest in these distinctions is driven by the goal of enhancing Apache Superset, an open-source BI platform, to support real-time data querying and analytics effectively.