January 2022 Summaries

In 2021, Airbyte made significant progress in the data integration field by building a community from scratch and developing over 160 out-of-the-box connectors. The company's focus was on solving three main problems faced by data teams: complex data systems, creation and maintenance of data connectors, and lack of autonomy among data consumers. Airbyte also launched its hosted version, Airbyte Cloud, in beta. The year 2021 saw rapid growth for the company, with a team size increase from 6 to 35 members across various departments. The community grew as well, reaching almost 5,000 members on their public Slack channel. Over 10k companies used Airbyte for data replication in just one year. In 2022, Airbyte aims to become the new standard for modern data stacks by focusing on reliability and ubiquity of its platform and connectors. The company plans to expand coverage of the long tail of connectors with community support and improve database replication capabilities. Additionally, Airbyte will focus on Reverse ETL, streaming, and accessing data within private VPCs. The company also intends to grow its content hub for data engineers and analysts, providing new high-quality content every day. It plans to attend most data-related events and foster local meetups worldwide. Furthermore, Airbyte will focus on enhancing developer experience and documentation with a dedicated team. As the product and community continue to grow in 2022, Airbyte aims to increase its team size to around 200 members by the end of the year while maintaining strong values and building proper recruiting and people infrastructure. The company will also prioritize Diversity & Inclusion in its hiring strategy.

This article discusses how to clean and transform raw data using SQL functions, focusing on data extracted from Google Sheets. It highlights common issues encountered when working with messy data and provides solutions using built-in SQL string functions such as split_part(), lower(), try_to_number(), iff(), round(), len(), charindex(), and to_date(). The article also emphasizes the importance of maintaining separate databases for raw and transformed data, following ELT principles.

Behavioral data collection is essential for understanding user interactions with products and creating personalized customer experiences. This guide explores the technologies and tools available for collecting such data, focusing on CDI (Customer Data Infrastructure) and CDP (Customer Data Platform) systems, as well as ELT (Extract, Load, Transform) tools. CDI is designed for gathering data from primary sources like web and mobile apps, while CDP adds a layer for identity resolution and audience syncing. ELT tools, like Airbyte, Fivetran, and Stitch, are better suited for extracting data from secondary sources such as third-party tools. Product analytics tools, including Amplitude and Mixpanel, also play a role in data analysis and collection. The guide emphasizes the importance of setting up a data warehouse to store all collected data and advises against building custom tracking solutions due to maintenance challenges. Collaboration with stakeholders is crucial when deciding on event tracking and data destinations.

In December 2021, Airbyte raised a $150 million Series-B funding round led by Altimeter and Coatue, valuing the company at $1.5 billion. The company decided to raise funds earlier than planned due to its ambitious growth plans for 2022, which included increasing team size from 30 to 200 members, expanding from 16k deployments to 100k, and growing from 150 connectors to 500 high-quality ones. The funding will help boost recruitment efforts and brand perception, enabling Airbyte to reach its next milestones faster.

Airbyte, an open-source data replication tool, initially used Docker for its containerization needs. However, as the user base grew and data volumes increased, scaling became a challenge. Kubernetes emerged as a potential solution due to its capability of horizontally scaling workloads. The main challenge in adapting Airbyte from Docker to Kubernetes was passing data between different containers, especially since there are no guarantees that pods will be scheduled on the same Kubernetes nodes. To address this issue, the team used Linux's minimalist and modular approach to software development by leveraging various networking tools like socat and named pipes. They also employed the sidecar pattern to create a sidecar socat container alongside the main container within the same Kube pod. This allowed them to switch between different networking tools without affecting users, ensuring isolation and encapsulation. The team used the Kubernetes API to dynamically create job containers as Kubernetes pods whenever Airbyte receives a job. They also utilized named pipes for piping data between the sidecar and main containers within the same Kube pod. The worker pod in an Airbyte Kubernetes deployment orchestrates all necessary pods to complete a job, ensuring smooth communication between source and destination pods. Despite some challenges like managing multiple STDIO streams and inefficient networking, this architecture has enabled Airbyte to scale its operations effectively. The team is currently working on V2 of the architecture to further improve efficiency and performance.