September 2021 Summaries

The blog post provides a step-by-step guide on setting up Airbyte, an open-source data ingestion framework, specifically for ingesting data from open-source communities like GitHub and Slack. It begins by detailing the installation using a Docker-Compose setup on an EC2 instance, highlighting Airbyte's simple setup process. The post then explains how to install and configure source connectors for data sources, with a focus on the native connectors for GitHub and Slack, and offers guidance on selecting repositories and managing API rate limits. The configuration of data connections includes setting schedules, prefixes, and streams, with the author noting the importance of external orchestration tools like Airflow for managing sync jobs, especially for large repositories. The author concludes with best practices for overcoming challenges with large data sets, such as syncing one stream at a time and understanding the limitations of certain GitHub API tokens, while hinting at future discussions on data transformation and validation for business intelligence purposes.

Rockset is a real-time indexing database designed for cloud environments, utilizing RocksDB for rapid data storage and retrieval, and is particularly suited for powering data-intensive, customer-facing applications. It achieves high-speed querying through its Converged Index, which automatically indexes semi-structured data in three ways, enabling millisecond query latency on large data sets. Apache Superset, an open-source BI platform built with modern technologies, offers a powerful no-code chart builder that simplifies the creation of real-time dashboards when paired with Rockset's data processing capabilities. To integrate Rockset with Superset, a SQLAlchemy dialect and a Python DB-API 2.0 library are required, along with a specific database engine spec to support features like time grains. Recent developments have included the addition of a Rockset database engine spec to Superset's codebase, facilitating its use in Superset deployments, whether through Docker or a native setup, and allowing connections via API keys for various regions. However, certain features, such as query lambdas and CSV upload, are still under development, with ongoing efforts to enhance integration and functionality.

The blog explores the concept of open-core companies, specifically focusing on Preset, which builds its business around the open-source project Apache Superset. The author discusses the importance of community involvement and the necessity of understanding the community's needs to guide the development of Superset. The piece also highlights the creation of a reference data stack using open-source tools to gain insights from open-source communities. It delves into the process of data ingestion, emphasizing the use of modern open-source batch ingestion frameworks like Airbyte to streamline extract, load, and transform (ELT) tasks. Airbyte is praised for its ease of setup and flexibility, offering connectors for various data sources like GitHub and Slack, and supporting custom transformations using dbt models. Deployment considerations, such as using Kubernetes for production environments and integrating with Apache Airflow, are also discussed, with a focus on ensuring seamless data synchronization and transformation processes. The blog promises further insights into data transformation using dbt and hints at future presentations at industry conferences.