November 2023 Summaries

move(data) 2023 is a two-day online conference dedicated to the latest in data engineering and AI. The event will feature lightning talks from experts covering various topics such as LLMs, data pipeline optimization, and real-world applications of data engineering strategies. With its focus on practicality and inclusivity, move(data) 2023 aims to provide valuable insights for data professionals across all levels of expertise.

Airbyte Quickstarts are practical examples that demonstrate how different components of a data stack interact, making learning transparent and straightforward. They provide actual code for various data stack interactions in real-world scenarios, allowing users to quickly understand and apply complex data stack interactions. Quickstarts integrate leading tools like Airbyte, dbt, and orchestration solutions such as Airflow, Dagster, or Prefect, and are designed for today's data professionals who need to set up and understand diverse data stacks quickly and effectively. They can be adapted to fit specific needs, providing a solid foundation on which to build and tailor data engineering projects.

The Airbyte Hashnode Hackathon was a successful event with over 60 contributions and 19 quickstarts created. Participants focused on migrating connectors to a low-code framework, improving existing connectors, and creating quickstart guides for using Airbyte with other tools. Winners were announced in two categories: Connectors Improvements and Quickstarts. The top contributors will receive over $20,000 in cash prizes.

Airbyte's Schema Propagation feature helps manage unexpected schema changes in relational databases by allowing users to choose how they want to handle these changes. The DiscoverSchema process compares old and new schemas, detecting any changes at the source and automatically handling them based on connection settings. Users can configure connections with options such as propagating column changes only, propagating all changes, or ignoring and pausing the connection. This feature ensures there are no surprises in data when a change occurs, leading to fewer breaking changes, less worry about validating data, and stronger data contracts.

The ELTP architecture (Extract, Load, Transform, Publish) is introduced as a solution to common design mistakes made when building data pipelines for AI, analytics, or data engineering. Unlike the traditional ETL approach, ELTP separates replication and processing steps, making it more stable and scalable. It also addresses the gap in delivering data to downstream users and systems by adding a 'Publish' step. The Publish framing is a simple way to describe how transformed data can be efficiently delivered to various destinations such as external SaaS applications, file stores, CRM systems, AI vector stores, and other databases. ELTP offers benefits like sending data files to external systems, decentralizing analytic queries, and publishing to downstream applications and indexes.

Airbyte now supports extracting text from documents stored in S3, Azure blob storage, and Google Drive sources. The extracted textual content is emitted as markdown, allowing users to leverage this data in search scenarios and when building language model-powered applications. This feature enables the utilization of valuable unstructured data such as meeting notes, specifications, roadmaps, and descriptions of planned features. Airbyte can extract all valuable data from these documents and send it to a warehouse for further processing. The new experimental "Document File Type Format" allows users to extract text content from PDFs, Word, PowerPoint, and Google documents just like structured data stored in CSV or Avro formats.