October 2023 Summaries

Airbyte's annual conference, move(data) 2023, is set to take place on December 6th and 7th, bringing together data engineers and practitioners from around the world. With over 40 featured speakers and a focus on innovation and dedication in the field of data engineering, this event promises valuable insights and networking opportunities for attendees. The conference will cover topics such as cutting-edge data movement technologies, real-time data integration, software engineering principles in data engineering, data contracts, Data as Code, and the role of large language models in data engineering. Registration is now open for this free online event.

The MongoDB source connector has been significantly upgraded, now supporting a Change Data Capture (CDC) mode and boosting replication speed over 10 MB per second. This enables faster data migrations, reducing downtime and improving data availability for analytics. The enhanced connector can also replicate collections of any size (1 TB+), with improved syncs that are resumable after failure. Additionally, the CDC mode ensures that data reflections are always current, enhancing data-driven decision-making processes.

In recent months, Airbyte has significantly improved the performance of its MySQL source connector, achieving a 3x increase from 4 MB per second to ~11 MBps. The improvements were made possible by applying lessons learned from enhancing the Postgres source connector. Key features include support for initial syncs of any size, reliable handoff to incremental syncs, and compressing state in logical replication (CDC) mode. These enhancements enable efficient data synchronization from MySQL databases while addressing various edge cases. Looking forward, Airbyte aims to leverage parallelization for further performance improvements and maintain its commitment to making the MySQL source connector industry-leading.

Airbyte's Hacktoberfest 2023 edition is an open-source event aimed at improving data integration, celebrating creativity, and fostering global collaboration. Participants can contribute to real-world projects, enhance their skills, network with talented individuals, and win prizes up to $9,000. The event includes categories such as Most Connectors Improvements and Most GitHub Upvotes on Written Quickstarts. It runs from October 1st to November 1st, 2023, with winners announced on November 20th. This hackathon is not just about winning; it's also about learning, growing, and being part of the open-source ecosystem that drives change.