July 2021 Summaries

Airbyte is celebrating its one-year anniversary. The company started on July 27th with the aim to address the long tail of integration, customization, data security, and cost control. Initially, they experimented with an open-source approach for a month but decided to extend their project evaluation period indefinitely. They faced challenges while leveraging Singer and eventually built an extensible and modern data protocol. Airbyte's community has grown significantly over the past year, leading to increased contributions from members. The company now supports more than 100 high-quality open-source connectors and aims to support 200 by the end of the year.

Over the past year, interviews with over 200 companies revealed that data integration remains a challenge in 2021. Many users of ETL technology still build and maintain their own connectors despite using an ETL solution. The primary reasons for this are incomplete connector coverage, significant friction around database replication, and the inability to cover all connector needs. Open-source solutions have emerged as a potential solution to these problems by providing customization capabilities, addressing the long tail of connectors, enabling broader integrations with data tools and workflows, offering debugging autonomy, and ensuring out-of-the-box security and privacy compliance. However, the barrier to entry for creating robust and full-featured connectors remains high. To address this issue, Airbyte has developed an open-source Connector Development Kit (CDK) that automates much of the connector development process, reducing implementation time from 2 days to 2 hours. The CDK is designed with onion layers of abstraction to maximize productivity and enable building robust connectors within a short period.