June 2023 Summaries

Airbyte, an open-source project, is updating its open-source license to expand the Elastic License v2 (ELv2) coverage. The change will cover strategic API, database, and data warehouse source connectors that Airbyte actively maintains, as well as some destination warehouse connectors. This decision aims to protect users from hosted ETL services and maintain community focus by preventing derivative services. However, the majority of Airbyte's user community (99.9%) will not be impacted by this change. The company remains committed to transparency and open communication with its users.

Michel Tricot, co-founder and CEO of Airbyte, discusses the company's mission to address the long tail of data connectors with an open source solution. He explains how they have managed to overcome the plateau of supported connectors by working with their community and customers to share the maintenance load. The conversation also touches on Airbyte's partnership with Snowflake, its growth during the pandemic, and how it can help companies become AI factories by providing more data for powering AI algorithms.

Everything as Code (EaC) is an approach that extends to managing not only software but also infrastructure and configuration in code. It involves using a Git workflow and code review process for changes to resources, rather than manual deployment. This post explores how EaC can be applied to data infrastructure, offering benefits such as easier environment reproduction, auditability, collaboration via pull requests, and automation of manual processes. The use of Airbyte Terraform provider is highlighted, along with its integration with Kestra orchestration for managing data ingestion, transformation, and scheduling. By adopting the EaC philosophy, data operations can become more resilient through the adoption of software engineering and DevOps best practices.

Airbyte, a modern data integration platform, has released its Terraform Provider for managing infrastructure as code. This new feature allows users to programmatically manage their Airbyte resources through configuration files and integrate them with existing infrastructure. The provider enables automation at scale, reducing errors and improving efficiency. It is currently available for Airbyte Cloud customers and will be made available for OSS and self-hosted Airbyte Enterprise deployments in Q3 of this year.

In this episode, we have Michel Tricot, the CEO of Airbyte, an open-source data integration platform. We discuss how Airbyte has grown since its inception and what's next for the company. Michel shares insights on data security and privacy measures taken by Airbyte to ensure user trust. He also talks about the emerging trends and challenges in the industry and how AI will play a significant role in shaping the future of data integration. We explore the modern data stack and how it might evolve into an AI stack, with companies needing to adapt quickly to keep up with the rapidly changing landscape. Michel shares resources for learning more about Airbyte connectors and ELT, as well as where people can reach out to him for further discussion on these topics. Overall, this episode provides valuable insights into the world of data integration and how it is evolving in response to emerging trends and technologies.

This article provides an introduction to preliminary data analysis using PostgreSQL, focusing on basic features of the dataset such as maximum, minimum, sum, average values, and percentile cutoff ranges. It also demonstrates how to divide data into different subgroups and obtain these values for each subgroup. The text includes prerequisites, systems, example datasets, and three options for obtaining the data: downloading a database dump file, using a Docker image with PostgreSQL preloaded with the preprocessed data, or starting from the raw CSV file and processing it yourself. Additionally, the article covers importing the dataset into PostgreSQL, preprocessing the data, and performing exploratory analysis of the data itself. The final section discusses aggregate functions, grouping operations, partitioning, and window functions in PostgreSQL.

This article demonstrates how to perform basic statistical analysis using PostgreSQL built-in functions. It covers the mean, variance, standard deviation, coefficient of variation, outliers, covariance, correlation, and regression. The advantages of running these analyses within the database include fewer IT systems to manage and maintain, avoiding passing data back and forth between different systems, and leveraging a mature RDBMS for enforcing data integrity and consistency. The example dataset used in this article is based on cancer statistics.

Airbyte, a data integration platform launched in July 2020, has released three significant features to enhance its functionality. These features are checkpointing for uninterrupted data syncs, column selection for controlled data integration, and schema propagation for automatic changes from source to destination schemas. The adoption of these features demonstrates Airbyte's commitment to secure, reliable, and efficient data integration.

Schema propagation has been introduced in both Airbyte Open Source and Cloud, allowing users more control over the schema of replicated data. This feature enables automatic handling of changes in source schemas, with options for adding new fields or streams, ignoring changes, pausing connections on changes, or prompting manual review before resuming syncing. The implementation relies on existing Airbyte protocol primitives and is available to all connectors. The goal is to reduce operational load in maintaining data pipelines by making them more resilient to infrastructure issues and source changes.

dbt-expectations is an open-source testing library for dbt that allows users to assert expectations about their data. It enables analytics engineers and data analysts who use SQL more than Python to take advantage of the same data quality testing as data engineers and software engineers. The package uses SQL code and integrates directly into your dbt project, allowing you to test sources, models, and seeds at every layer of your data model. Key aspects of data that can be tested include row count, column count, string patterns (regex), anomalies, and datatypes. By implementing these tests, users can maintain high data quality throughout various fields and detect abnormal patterns in their data.

Airbyte has introduced column selection for its community on both Open Source and Cloud platforms, allowing users to choose which fields they want to synchronize from their API or database source. This feature is available across all connectors and can be accessed during the connection setup process by selecting columns in a popup. The implementation ensures compatibility with all existing connectors without requiring any changes. Column selection offers benefits such as data minimization, privacy and security, and relevance for specific business analyses. Future updates will include more control over replication behavior when source schemas change.

Airbyte, a data sync platform, has introduced checkpointing to ensure uninterrupted data flow during transient failures. Checkpointing allows the platform to resume any incremental sync from where it left off in the previous attempt. Currently, all sources that support incremental syncs and cloud data warehouse destinations such as Snowflake, BigQuery, and Redshift are checkpointable. The Airbyte Protocol enables this process by using state messages to confirm when data is saved on the destination end. This feature allows for more efficient and reliable data movement, with additional features like deduplication to clean up data on the other side.

In May, Airbyte launched the Connector Builder, a no-code tool that allows users to create new API connectors in as little as 15 minutes. Since its soft launch two months ago, over 100 connectors have been built and deployed to production. The company also made significant changes to the Snowflake Destination by removing the Azure Blob Storage loading option due to issues and lack of certification for this functionality. Airbyte introduced a new content hub featuring various content formats catering to all learning styles and preferences in data engineering. Additionally, they conducted Hack Days where their team built prototypes on interesting projects. The changelog includes new features, security & breaking changes, and bug fixes across different sources and destinations.