October 2020 Summaries

The company launched its MVP a month ago and received significant feedback and support from the community, including a large pull request. Despite this momentum, they didn't release new connectors immediately. They focused on defining the best data protocol for all companies and building a manufacturing plant for data integration connectors. This article outlines how to build a large number of connectors, emphasizing initial build, liveliness & change detection, and maintenance. The company aims to reach a rate of 5 connectors per day and provide tools for the community to contribute their own connectors with minimal or no code.

Version 0.2.0 introduces significant enhancements, including the ability for users to build custom connectors, update existing ones, and manage them through a new Admin section in the UI. This section allows users to update source connectors by changing tag versions and provides direct links to Docker images, with a future focus on expanding available connectors. Additionally, it offers a feature to add new connectors by providing necessary details like display name and repository name. The documentation, which is open for feedback, guides users on building connectors in Python and Java, with plans to support more languages. The release also emphasizes the development of high-quality destination connectors, particularly for Postgres and BigQuery, with more to come. Users are encouraged to follow updates via the changelog and engage with the community on Slack for upcoming milestones and new connector releases.

The data integration industry has evolved significantly over the past 16 years, with new open-source projects like Singer.io emerging alongside commercial software solutions such as Snowflake, Bigquery, and Redshift. While companies like Fivetran have made it easier for non-engineering teams to set up and manage data integration connectors, many teams still build their own connectors in-house. This article explores the landscape of open-source data integration technologies, focusing on projects such as Singer, Airbyte, PipelineWise, and Meltano. It also highlights related noteworthy open-source projects like Apache Airflow, DBT, and Apache Camel. The analysis aims to provide a comprehensive overview of the current state of open-source data integration alternatives and their potential impact on the industry.

Airbyte's data protocol will be compatible with Singer's, allowing users to integrate and use Singer's taps. However, the two protocols differ in many ways. Unlike Singer, which lacks standardization and real ownership, Airbyte has one single repo for the whole project and all connectors, promoting consolidated developments and a unified community vision. Additionally, Airbyte runs connectors as Docker containers, allowing contributors to build connectors in their preferred language. The platform also separates extract-load from transformation, enabling users to choose how they want to use data. Furthermore, Airbyte offers a UI and API for addressing the needs of various teams and is fully committed to maintaining open-source connectors without premium options.

The debate on DBT's Slack focused on the relevance of open-source (OSS) approach versus commercial software in addressing data integration problems. Key challenges discussed include having a large number of high-quality well-maintained pre-built connectors, addressing unique corner cases and unique needs, pricing model indexed on usage, debugging autonomy, getting internal resources, ease of use by any teams, security and privacy compliance, and moving data between internal databases. The conclusion drawn is that if the open-source project is well structured and thought through, there will be no points where a commercial approach can be better.

Airbyte, an open-source data integration platform, launched its MVP on September 24th with limited connectors. The company opted for a soft launch, reaching out to a finite number of people through social networks and YC's internal forum. They learned that a soft launch can help startups learn faster and adjust their products accordingly to achieve product-market fit. Airbyte gained community growth, identified product improvements, and increased its visibility among domain experts and active contributors during the soft launch. The company plans to focus on improving developer experience, supporting multiple destinations, adding more pre-built connectors, and enabling multi-node support in the coming weeks.