June 2022 Summaries

Airbyte, an open-source data integration engine, has shared its Q3 roadmap which includes expanding the number of Generally Available Connectors across both their offerings. They will be focusing on database connectors such as Postgres, msSQL, and mySQL. Additionally, they plan to auto-detect schema changes and release a closed Alpha of their Public API for enterprise users. Airbyte is also working towards achieving ISO 27001 certification and GDPR best practices to ensure data protection compliance.

The author, a former Data Scientist and current software engineer at Airbyte, shares their experience in building data teams and how modern data tools are redefining roles within these teams. They discuss the importance of hiring technical Data Engineers before onboarding more Data Analysts or Scientists, as well as the changing role of Data Engineers due to automation and new data tools. The author also shares their thoughts on the structure of a growing data team at Airbyte, emphasizing the need for diverse skill sets and balanced teams.

This article discusses the shift from a data pipeline-centric view to a data product-focused approach, driven by the need for better understanding and management of complex data environments. It explores the concept of data orchestration tools that introduce data assets as first-class objects and how they help reconcile the worlds of data analysts and business users with those of data engineers. The article also covers various abstractions (jobs, tasks, resources, triggers, and data products) and their role in building a Data Mesh. It highlights the importance of focusing on data products instead of data pipelines and why it is better to write declarative code so that data pipelines can update themselves when upstream data assets change.