December 2022 Summaries

In 2022, Airbyte achieved significant milestones such as reaching 9,000 stars on GitHub and having over 300 connectors available. Community contributions included writing articles and tutorials, with the most popular being "Snowflake Data Warehouse Architecture" by Madison Mae and "Build a real-time data analytics pipeline with Airbyte, Kafka, and Pinot" by Dunith. The company also hosted its first virtual conference, move(data), which attracted 3,000 attendees from over 72 countries worldwide. Looking ahead to 2023, Airbyte plans to focus on improving speed and reliability of the platform, expanding its community contributors program, enhancing tooling for contributors, and making progress on its split control/data plane.

Fully-managed Airbyte is a data integration platform that allows companies to securely move their data across different systems in minutes. It supports 40,000+ organizations and provides reliable database and API replication at any scale. The platform also enables the embedding of hundreds of integrations into an app, making it easy for developers to integrate various data sources and services. Additionally, Airbyte uses large language models (LLMs) to make sense of unstructured data, and its high-volume DBs can handle low latency. With a strong focus on community engagement, Airbyte offers guides, knowledge base, GitHub discussions, and live events to support users in their journey. The platform is designed for various industries, including marketing, sales, product, finance, engineering, and more.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to replicate their databases and APIs in minutes. With over 40,000 users, it provides integrations with hundreds of applications, supports high-volume databases with low latency, and makes sense of unstructured data using LLMs. Airbyte also offers a range of features such as embeddings from unstructured data, building new connectors in just 10 minutes, and providing easily collectible credentials for end-users. Additionally, the platform has a strong community aspect, with over 20,000 members sharing tips and getting support through GitHub Discussions. It also hosts live events by the Airbyte team and provides access to its knowledge base and 30,000+ subscribers, making it a go-to solution for data engineering and technology professionals.

Mastodon, an open-source and decentralized social media platform, has seen a surge in its active user base from 300,000 to 2.5 million in the last two months. It offers similar features as Twitter but with a different structure and culture that can be intimidating for new users. This blog post provides tips and guides for data engineers transitioning to Mastodon, covering account setup, finding communities, participating in conversations, and sharing work. Mastodon's decentralized nature, privacy-focused features, and emphasis on community over commercial interests make it an increasingly popular choice among data engineers looking for a more collaborative and inclusive environment.

Airbyte is a fully-managed platform that enables secure data movement for entire organizations, allowing companies to easily replicate their databases and APIs. With thousands of integrations available, users can embed these integrations into their applications in minutes. Airbyte also provides reliable database and API replication at any scale, as well as embeddings from unstructured data. Additionally, the platform offers a range of features such as low-latency high-volume DBs, credential collection, and community support. Through its developer advocacy program, Airbyte aims to help Python developers leverage its power and provide thought leadership in data engineering. The platform also provides resources for users to learn from each other's successes, evaluate costs, and access knowledge bases and communities.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing users to start with their entire org in minutes. With its 100s of integrations, Airbyte provides reliable database and API replication at any scale, as well as embeddings from unstructured data and the ability to build new connectors in just 10 minutes. The platform is used by over 40k companies and offers a range of features including low-latency high-volume DBs, marketing, sales, product, finance, eng & more. Airbyte also provides a knowledge base, GitHub Discussions for questions and ideas, as well as live events by the team to support users in their data engineering journey.

The author, a full-stack web developer, shares their experience of discovering the benefits of thinking like a Data Engineer and utilizing off-the-shelf Data Engineering tools. They identify three types of projects that can benefit from these tools: bulk third-party API integration, timestamps and audit trails, and slow & sequential tasks. The author emphasizes the importance of selecting open-source or free-to-use tools with easy setup, monitoring capabilities, and production-ready features. They provide examples of how Data Engineering tools can streamline processes such as integrating APIs, capturing database changes, and managing complex job dependencies.

This article discusses the fundamental differences between a data warehouse and an operational database, explaining that while they provide similar functionalities, they are optimized for different use cases. Data warehouses are generally optimized for OLAP (online analytical processing) workloads, while operational databases are generally optimized for OLTP (online transactional processing) workloads. The article covers topics such as the architectural differences between these technologies, examples of OLTP and OLAP workloads, benchmark results comparing an OLTP database with an up-and-coming OLAP data warehouse, when to use an operational database or a data warehouse, and how Airbyte can be used to synchronize data from an operational database to a data warehouse as part of an overall data integration strategy.

BigQuery, a popular analytical database on Google Cloud Platform, is designed for heavy analytical queries on large datasets. It separates storage and compute components to allow independent scaling of both. To increase speed and performance, design patterns can be used for optimizing BigQuery storage. Understanding BigQuery storage costs involves recognizing that data is stored column-wise and charged based on whether it's active or inactive. Compute costs are determined by the amount of data scanned during query execution, with more data processed as you select more columns. To investigate performance issues, two visual tools are available: one showing how much data a query will process and another providing diagnostic information about query execution steps. Four ways to optimize storage include partitioning tables, clustering tables, pre-transforming data into materialized views, and denormalizing data.

Airbyte is a fully-managed data integration platform that enables companies to securely move data across their organization in minutes, with 40k+ users worldwide. It offers various features such as secure data movement, reliable database and API replication at any scale, embeddings from unstructured data, and high-volume DBs with low latency. Airbyte also provides a wide range of integrations that can be embedded into applications, making it a powerful tool for Python developers and organizations across multiple industries including marketing, sales, product, finance, and engineering. The platform offers various resources such as guides, community calls, and knowledge bases to help users get started and stay up-to-date with the latest developments in data engineering.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to embed hundreds of integrations into their apps in a single deployment. With its reliable database and API replication capabilities, Airbyte supports high-volume databases with low latency, making it suitable for various industries such as marketing, sales, product, finance, engineering, and more. The platform also provides embeddings from unstructured data and allows users to build new connectors in just 10 minutes. Additionally, Airbyte offers a range of resources, including guides, community forums, and knowledge bases, to help users evaluate costs, learn from others' success, and contribute to the open-source community.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to securely replicate their databases and APIs. With 40k+ users, it provides embeddings from unstructured data, reliable database and API replication at any scale, and allows developers to build new connectors in just 10 minutes. The platform also supports high-volume DBs with low latency, marketing, sales, product, finance, eng, and more integrations. Airbyte makes it easy for users to collect credentials from their end-users, learn from others' success, and choose the right solutions for their needs. It offers guides, community support, and a knowledge base to help users on their journey, as well as opportunities to become a technology or consulting partner.

In this blog post, we discuss 12 key trends in data engineering that will shape the industry in 2023. These include SQL and Python dominating as programming languages, with Rust emerging as a promising contender; abstraction through custom ETL being replaced by data connectors; streaming data becoming more popular than batch processing; architecture improvements such as the lakehouse and semantic layer combining strengths; trust in data quality and observability becoming essential; usability focusing on data products and data contracts; openness with open-source tools dominating; standards simplified by DuckDB; roles expanding knowledge into data engineering; collaboration through DataOps reducing data silos and improving teamwork; adoption of data engineering regardless of industry or business size; and community interaction on various platforms.

Airbyte is a fully-managed platform that allows companies to securely move their data across different systems and applications, with the ability to embed hundreds of integrations into a single app. The platform supports large-scale database and API replication, reliable data embeddings from unstructured data, and enables users to build custom connectors in just 10 minutes. Airbyte also provides tools for data engineering thought leadership, improving data replication, and deploying use cases in minutes. With a strong focus on community engagement, Airbyte offers resources such as guides, knowledge bases, GitHub discussions, and live events, serving over 30,000 subscribers and 20,000+ members of its data community. The platform is used by over 40,000 companies and provides a range of features for marketing, sales, product, finance, engineering teams, and more.

Airbyte Cloud has integrated with dbt Cloud, allowing users to schedule dbt transformations directly within Airbyte Cloud. To start, add a dbt Cloud service token in workspace settings and associate the desired transformation with your connection. This seamless integration enables triggering of transformations from ELT pipelines.

The Airbyte team has announced several updates during their first conference dedicated to data practitioners. They have reached 600+ code contributors, 2,500+ daily active users, and over 300 connectors in the past two years since inception. In 2023, they plan to launch a Free Connector Program for Alpha/Beta connectors with their Airbyte Cloud product offering. Additionally, they will introduce a connector builder that enables users to create connectors in under 10 minutes and a low-code CDK for creating connectors in under 30 minutes. Furthermore, companies can now bring their own infrastructure by running the data plane in their virtual private cloud while Airbyte hosts the control plane for configuration management and scheduling tasks.

Yiyang Li, an open source contributor and maintainer for Airbyte, shares his experience with the platform and offers tips for first-time contributors and those looking to become maintainers. He discovered Airbyte while building a new analytics tool and found it valuable due to its adaptive, powerful, and flexible nature. As a maintainer, Yiyang has learned a lot about the deployment cycle, code standard, and CI tools. His top tip for using Airbyte is to use the default setup and run it in your local docker before considering more complex configurations or pricing models. For those interested in contributing, he suggests being patient with the three-level learning curve related to schema language, connectors platform, and data in the schema. Yiyang also emphasizes the importance of providing specific feedback, being kind and empathetic, and trusting mentors within the community.

Airbyte is a fully-managed platform that enables secure data movement across organizations in minutes, with features such as secure data movement, embeddings from unstructured data, reliable database and API replication, and low-latency high-volume databases. It also provides a wide range of integrations with 100s of apps, making it easy to embed these integrations into custom applications. Airbyte is used by over 40k companies worldwide and has a large community of developers and users who contribute to its development and share knowledge on data engineering topics.

The Hacktoberfest event was a great success, with contributors from all over the world participating in creating 103 new connectors, which is a 145% increase compared to last year's 42 connectors. Out of these, 75 sources used the low-code CDK, while 20 and 1 source utilized Python and Java CDK respectively. The winners for Hacktoberfest 2022 were Haithem Souala (first place), Isaac Harris-Holt (second place), and Elliot Trabac (third place). A total of $70,000 in cash prizes will be given out to those who had their PR merged by the deadline. Missed participants can still get involved through Airbyte's contributor and maintainer program.

In November, Airbyte concluded its Hacktoberfest contest and launched new connectors, including those for Apache Doris, Heap Analytics, Redpanda, and more. The company also released an MVP version of the connector builder, which allows users to build low-code connectors. Additionally, Airbyte OSS benefited from improvements in its core features, such as progress bars, while Airbyte Cloud was launched in Europe with a hybrid model for data protection.

The State of Data Engineering survey results were released, showcasing top individual and ecosystem data newsletters for 2022. Individual Writers publish long-form original essays and explainers, while Ecosystem Aggregator newsletters curate the best work from others. The top five individual data newsletters are Seattle Data Guy, Benn Stancil, Metadata Weekly, Technically, and Learn Analytics Engineering. The top five data ecosystem newsletters are Towards Data Science, Data Engineering Weekly, The Analytics Engineering Roundup, Blef.fr Data News, and All Hands on Data.

Airbyte, a data infrastructure software company, has launched its EU version of Airbyte Cloud after ensuring GDPR compliance and adherence to ISO 27001 standards. This multi-region cloud solution validates the functionality of their "dual plane" architecture, which separates the oversight of data pipelines from the pipelines themselves. The next steps for Airbyte include expanding into more complex regulatory frameworks and developing a hybrid version of its Open Source and Cloud software that will allow customers to process data on a cloud provider or within their own network. This hybrid model aims to offer clients greater control over who processes their data assets and when, where, and how that processing occurs.