August 2022 Summaries

In August's edition of The Drip, Airbyte announced that it is no longer in Alpha phase as thousands of companies are already using the product on-prem and via Airbyte Cloud. SQLite has been added as a Destination, and Postgres is now Generally Available (GA) after validating its reliable experience for users. Additionally, new sources are hitting beta, with more expected to follow. The team reached new milestones and will be supporting and presenting at upcoming industry events such as Big Data London, Big Data Paris, dbt Coalesce, State of Data Engineering, and move(data) conference.

The article discusses the challenges of ensuring data quality compared to code quality, highlighting that detecting, understanding, fixing, and reducing data quality issues are harder than their counterparts in software engineering. It argues that while both involve meeting expectations, data is more complex due to its size, rapid changes, and dependencies. The article also explores the difficulty of debugging data quality issues, as well as strategies for improving data architecture to reduce such issues.

Data security and transfer require careful consideration of various rules and regulations, which can be challenging to navigate. Key factors to consider before moving data include the type of data (personal or sensitive), its origin, destination, purpose, and method of transfer. Understanding these aspects helps determine applicable regulatory requirements and ensures compliance with data protection laws.

Airbyte is a fully-managed platform that enables secure data movement across organizations. With its 40k+ users, it offers various features such as embedding hundreds of integrations into apps, reliable database and API replication at any scale, and embeddings from unstructured data. The platform also allows developers to build new connectors in just 10 minutes and provides a wide range of integrations for marketing, sales, product, finance, engineering, and more. Additionally, Airbyte offers various resources such as guides, community calls, live events, and knowledge bases to help users get started and stay up-to-date with the latest developments in data engineering.

Airbyte contributor Tuan Nguyen shares his experience with the open-source data integration platform and offers tips for first-time contributors. He highlights the Octavia CLI feature, which allows users to manage connection settings programmatically. Tuan emphasizes the importance of community involvement in maintaining a successful open-source project and discusses his passion for contributing to such projects. Additionally, he shares insights on upcoming technology trends like AR/VR and his favorite Airbyte Slack channel, #shameless-plug.

This article discusses the concept of a Data Lake, its importance and how it differs from a Data Warehouse or a Data Lakehouse. It explains that a data lake is a storage system for vast amounts of unstructured and semi-structured data, stored as-is without a specific purpose. The primary components of a data lake include the storage layer, the data lake file format, and the data lake table formats. The article also delves into the differences between these three components and how they can be used to build an open-source Data Lakehouse. It further discusses the market trends in 2022 related to data lakes and provides a step-by-step guide on how to turn a data lake into a data lakehouse. The article also mentions some alternatives or situations where using a data lake might not be suitable.

Reverse ETL, a concept that involves moving data from a warehouse into business applications, is becoming increasingly popular among data teams. It allows non-technical users to access actionable data within their commonly used applications. Unlike traditional ETL, which extracts, transforms, and loads data into a warehouse, Reverse ETL syncs data from the source like a data warehouse to a business application. This process is more challenging than conventional ETL/ELT due to factors such as synchronization mode, data quality and transformations, and failures and job re-execution. The most common use cases for Reverse ETL include marketing, sales, and support.

Airbyte is a fully-managed platform that enables companies to securely move and replicate their data across various systems, with features such as secure data movement, reliable database and API replication, and the ability to embed hundreds of integrations into an app. The platform has been used by over 40k+ companies and offers tools for building new connectors in just 10 minutes. It also provides a community-driven approach, allowing users to learn from others' success, share tips, and get support from a large community of data professionals.

Analytics engineers often face challenges when rewriting SQL data models, such as slow updates, lack of comments, incorrect joins, and duplicated data. Data modeling organizes SQL code to make it usable, and dbt (Data Build Tool) aids in writing efficient, modular code that improves model performance and debuggability. Key practices include creating base models to reference raw data, using correct joins to minimize duplicates, and favoring common table expressions (CTEs) over subqueries for clarity. dbt also supports macros for reusable SQL logic, enhancing code efficiency and readability. By adhering to these best practices, analytics engineers can build fast, dependable, and scalable data models that avoid technical debt and support business growth.

Podcasts offer a convenient way to stay informed and entertained, especially in the data field, yet challenges like feedback for podcasters and discovery for listeners persist. Airbyte has curated a list of top podcasts in categories such as Data Practitioner, Data Thoughtleader Interview, Database/Warehouse, and Best New Podcasts of 2022. The selection highlights podcasts like The Analytics Engineering Podcast and The Data Stack Show, known for their engaging interviews and informative content. Honorable mentions are also noted, offering a rich tapestry of options for listeners interested in data topics. The piece encourages feedback on any missed favorites and promotes Airbyte's own appearances on various podcasts, enhancing its visibility in the data integration community.

An Airbyte user reported that their unsecured instance was compromised, and their connector credentials were stolen. As transparency is a core value at Airbyte, the incident has been highlighted to the community. The company takes security seriously but avoids making too many assumptions on infrastructure due to being an open-source project. It strongly recommends that self-hosted instances not be exposed to the public internet. Data pipelines are particularly vulnerable as they store credentials. In March 2022, Airbyte changed its upgrade flow to no longer require exporting/importing credentials and scrubbed all secrets from output by default. The current version of Airbyte is 0.40. To improve security, the company will implement basic password authentication on the UI, enable external secret storage in Airbyte Core, and implement scanners to detect publicly exposed instances.

In July, Airbyte celebrated its second birthday and made significant progress with connectors such as GitHub, Google Analytics, Instagram, TikTok Marketing, Bing Ads, and S3. The Octavia CLI saw improvements including the ability to set custom HTTP headers on requests, enable normalization or use custom DBT transformations within yaml configurations, and easily switch between different Airbyte instances. Additionally, Per-Stream State was released into OSS, allowing for future incremental sync capabilities. Open Telemetry Metrics were also added to Airbyte OSS, enabling the export of built-in metrics when using Airbyte at scale.

Airbyte is a fully-managed platform that enables companies to securely move their data between different systems and applications, with the ability to embed hundreds of integrations into an app in one go. It provides reliable database and API replication at any scale, as well as embeddings from unstructured data, allowing developers to build new connectors in just 10 minutes. The platform also supports high-volume databases with low latency, making it suitable for various industries such as marketing, sales, product, finance, engineering, and more. Airbyte provides a range of features that make it an attractive solution for companies looking to improve their data replication game, including easily collecting credentials from end-users, learning from other members' success, and choosing the right solutions for their needs. Additionally, Airbyte offers guides, community support, and resources to help users evaluate costs, learn from others, and stay up-to-date with the latest developments in data engineering.

Postgres, an open-source daily active user connector on Airbyte platform, has been used by over 2,000 users and moves more than 200TB of data monthly. The growth rate is around 30% each month. Postgres was the first connector released on the platform in October 2020. Today, Airbyte announces the release of Postgres to general availability with a reliable 99% SLA, live technical support and scalable pricing. This move comes after understanding that there are no other good offerings in the market for this service. The dedicated focus on bringing Postgres to GA has led to improvements such as full syncs, incremental syncs, and Change Data Capture (CDC). Airbyte's Postgres connector is now a game changer with its reliability, live support, thriving community, and cost-effective pricing. The company plans to apply the learnings from this process to bring MySQL and MSSQL source connectors to GA within the next two months.

Elementl has announced General Availability (GA) of their Dagster Cloud, along with Dagster 1.0. The company's Dagster Cloud is now operationally robust and easy to work with, offering an enterprise orchestration platform that prioritizes developer experience. It supports fully serverless or hybrid deployments, native branching, and out-of-the-box CI/CD. Elementl has also introduced a new feature called Branch Deployments. The company's Dagster Day event highlighted the product's solidity and user feedback, emphasizing its developer experience benefits.

Airbyte is a fully-managed platform that enables companies to securely move their data between different systems and applications, with features such as secure data movement, reliable database and API replication, and the ability to embed hundreds of integrations into an app at once. The platform is used by over 40,000 companies and provides tools for data engineering thought leadership, community engagement, and knowledge sharing. With Airbyte, developers can build a new connector in just 10 minutes, make sense of unstructured data with large language models, and deploy their use case in minutes, among other benefits. The platform also offers resources such as guides, webinars, and live events to help users get started and stay up-to-date with the latest developments in data engineering.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to embed hundreds of integrations into their applications in minutes. With its reliable database and API replication capabilities, Airbyte supports high-volume databases with low latency and makes sense of unstructured data using LLMs. The platform also provides an easy way for developers to build new connectors in just 10 minutes, as well as a community-driven approach with guides, knowledge base, and GitHub discussions. Airbyte's success has been recognized by over 40k companies, and its developer advocate team continues to provide support and resources for data engineering thought leadership.

In October 2021, Airbyte introduced a vision for consumption-based pricing with credits, separating how databases and APIs were charged. However, in April 2022, they switched to volume-based consumption due to the unpredictability of compute-based pricing. The new pricing model is based on millions of rows for APIs and GBs for databases and files. Airbyte offers discounts when users buy bulk credit packages upfront. They are also more affordable than competitors using the monthly-active-rows (MAR) model. A 14-day unlimited credit trial is available to help customers estimate their costs.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to get started in minutes. It offers various features such as secure data movement, reliable database and API replication, embeddings from unstructured data, and the ability to build new connectors in just 10 minutes. Airbyte has been used by over 40k companies worldwide and provides a wide range of integrations that can be embedded into apps. The platform also supports high-volume databases with low latency and offers various tools for data engineering thought leadership, such as improving data replication game and deploying use cases in minutes. Additionally, Airbyte provides resources like guides, community calls, and knowledge base to help users on their journey, including a GitHub Discussions forum for questions and ideas, and live events by the Airbyte team.