December 2021 Summaries

Airbyte has raised $150 million in a Series B funding round led by Altimeter Capital and Coatue Management, alongside Thrive Capital, Salesforce Ventures, Benchmark, Accel, and SV Angel. The company aims to address the growing problem of data integration with its community-powered solution. Airbyte focuses on the first pillar of the new world of data: data movement (ETL/ELT). Its North Star is to provide the most reliable and ubiquitous platform for moving data, focusing on connectivity, location, and new types of data movement. The company plans to double its team size by the end of 2022 while maintaining a diverse and motivating work environment.

The article discusses how a simple ETL (Extract, Transform, Load) script can evolve into a more complex project as it becomes production-ready. It highlights the need for scheduling, monitoring, debugging capabilities and handling API changes in an ETL script. The author shares their experience of watching this narrative unfold in multiple high-performance engineering organizations. They emphasize that using an ETL framework can provide built-in connectors to extract and load data, common transformation logic, scheduling, monitoring and a better developer experience. Airbyte is mentioned as an open-source ETL framework that helps users avoid the "little ETL script" antipattern by making it easy to write source and destination connectors using their Connector Development Kit (CDK) or ETL framework in Java, Python, C# or JavaScript. The CDK provides improved developer experience and abstracts away low-level glue boilerplate. The author believes that an OSS approach is the only way to solve the problems of data integration as it allows for continuous improvement and support from a community rather than relying on a single company.

Airbyte, a data replication platform, has partnered with Prefect, a leading dataflow automation platform. The integration allows users to leverage Prefect's orchestration capabilities for triggering Airbyte sync jobs, providing better control over ELT job scheduling and improved visibility into sync failures. With this partnership, both platforms aim to help data engineers, analysts, and scientists save time and have more control over their data flows.

Airbyte is an open-source project that orchestrates Docker containers to sync data from various sources to data warehouses and lakes. The process involves two types of containers - source containers for reading data, and destination containers for writing data. These containers implement the Airbyte Protocol which specifies a command line interface and the structure of stdout messages passed between them. When initially orchestrating third-party containers in Kubernetes, Airbyte encountered challenges with extending container entrypoints to perform syncs. Since Kubernetes does not allow inspecting Docker entrypoints, they needed a strategy to identify this entrypoint when launching the container. They considered three options: retrieving the entrypoint from the Docker registry APIs, requiring a fixed entrypoint, and setting the entrypoint in an environment variable. Airbyte ultimately chose Option 3 - requiring developers implementing their connectors to add an environment variable containing their entrypoint as a string that could be executed with eval. This allows them to wrap Docker entrypoints on the fly to inject complex logic for reading in stdin from a network connection, relaying stdout/stderr over the network, and performing other operations.

In November, Airbyte focused on improving connector reliability and user experience. They implemented data type fixes across database connectors, refactored destination databases for increased resilience, and enhanced source acceptance tests. Their Hacktoberfest event resulted in 42 contributed connectors from participants worldwide. Tuan Nguyen was recognized as the October Airbyte Community MVP with six contributions. The company is hiring for various positions and has added new community members on Slack. They have also gained over 40 new PR contributors, increasing their unique contributor count to 201.