October 2021 Summaries

Airbyte began with the decision to tackle data integration challenges through open-source solutions, specifically focusing on ELT (Extract, Load, Transform) processes, to build a strong community of contributors. This decision was influenced by the limitations and costs of existing closed-source tools like Fivetran and StitchData, which prompted companies to create their own in-house connectors. Airbyte's open-source approach, supported by their Connector Development Kit (CDK), allows users to efficiently build and maintain connectors, addressing a wide range of integration needs. Their infrastructure pricing model, based on compute time, supports scalable and cost-effective database replication, which is crucial as database volumes can far exceed those of API sources. Airbyte aims to expand beyond ELT to include reverse-ETL and event-driven integrations, leveraging its growing community of data engineers to cover a broad spectrum of connectors. Ultimately, Airbyte seeks to become the standard platform for all data movement, offering comprehensive solutions with transparent pricing and extensibility, and paving the way for advanced data products such as data quality, observability, and compliance.

Apache Airflow is a popular open-source workflow management platform often used for ETL pipelines. However, it may not be the best choice for all businesses due to its tightly coupled sources and destinations in transfer operators, making it difficult to cover long-tail integrations. An alternative approach is to use Airflow as a scheduler while integrating with other open-source projects like Airbyte for extracting and loading data, and dbt for transforming data within the data warehouse. This combination allows businesses to remove boilerplate code needed with Airflow and handle dependencies between tables in SQL files instead of Airflow DAGs.