January 2023 Summaries

Open source has become a crucial part of software development, with over 96% of global organizations using it. It provides control, flexibility, and agility to developers, enabling them to customize software according to their specific needs. Open-source software also accelerates time to value, offers better reliability and security, drives innovation, and prevents vendor lock-in. However, industries where users are not technical or lack overlap between use cases and contributor profiles have been slower to adopt open source. While free tiers can be a great way to test out new technologies, open-source solutions offer more control, flexibility, and community support. In the data infrastructure industry, only the data warehousing segment hasn't been significantly impacted by open source.

Madison Schott, an analytics engineer with a passion for teaching and writing, shares her journey into data engineering and blogging. She started by sharing fashion and travel experiences on her college blog before moving to health, wellness, and eventually data topics. Her goal is to make complex concepts easy to understand through her writings. In the past six months, she has been particularly proud of her ebook on analytics engineering and maintaining consistency with her weekly newsletter. Madison loves open-source data tools and finds them accessible for anyone interested in learning these skills. She enjoys reading Brene Brown's "Atlas of the Heart" and recommends Pizzolato Sparkling Rose as a favorite wine.

Analytics engineering is a role that combines aspects of data analysis and data engineering, making it an attractive option for individuals with diverse skill sets. Analytics engineers use SQL and Python to manage the data pipeline from ingestion to transformation, ensuring high-quality data through documentation, testing, and validation. They work closely with business teams to define KPIs and produce data models that can be used by data analysts to create reports and dashboards for stakeholders. Analytics engineers are responsible for setting standards across the data pipeline and discovering new tools and ways of doing things. If these tasks interest you, analytics engineering could be a suitable career path.

Airbyte is an open-source data integration engine that consolidates data in warehouses, lakes, and databases. It offers a Free Connector Program with 300+ connectors for users to try for free. The platform also provides various products such as Airbyte Open Source, Airbyte Cloud, and Airbyte Enterprise, catering to different needs. Users can benefit from solutions like database replication, marketing analytics, sales & customer analytics, product analytics, engineering analytics, and finance & ops analytics. Additionally, Airbyte has a community of over 10,000 data professionals who contribute to the platform's development and share knowledge through various resources such as blogs, tutorials, webinars, and more.

Airbyte has launched its Free Connector Program, making all alpha and beta connectors on Airbyte Cloud free of charge. This move aims to increase the usage of data connectors and improve their reliability. Alpha connectors have a 90% sync success rate, while beta connectors have a 93% sync success rate. The program is part of Airbyte's mission to commoditize data integration by providing thousands of reliable connectors. By encouraging users to adopt these connectors, Airbyte hopes to identify edge cases and bring more connectors to General Availability (GA) status with a >99% sync success rate.

Airbyte is an open-source data integration engine that enables users to consolidate their data in various storage systems such as warehouses, lakes, and databases. It offers a Free Connector Program with 300+ connectors for unlocking all your data. The platform supports extract & load operations from multiple sources to numerous destinations, as well as transformation capabilities using dbt/SQL. Airbyte also provides a Connector Development Kit for building new connectors in under 30 minutes and an API for embedding integrations into applications. It offers three product offerings: Open Source, Cloud, and Enterprise, each catering to different needs and requirements. The community is active with over 10,000 members contributing to the growth of data integration solutions.

Airbyte is an open-source data integration engine that helps users consolidate their data in various storage systems such as warehouses, lakes, and databases. It offers a Free Connector Program with 300+ connectors to unlock all your data. The platform also provides a variety of products including Airbyte Open Source for deep custom needs, Airbyte Cloud for effortless custom connectors, and Airbyte Enterprise for security and compliance in handling large volumes of data. Airbyte's solutions cater to various industries such as database replication, marketing analytics, sales & customer analytics, product analytics, engineering analytics, finance & ops analytics, and more. The platform also has a strong community with over 10,000 members contributing to its growth and development.

Airbyte is an open-source data integration engine that consolidates your data in warehouses, lakes, and databases. It offers a Free Connector Program with 300+ connectors for unlocking all your data. The platform includes features like extract & load, transform, API & Powered by Airbyte, and Connector Development Kit. Airbyte provides solutions for database replication, marketing analytics, sales & customer analytics, product analytics, engineering analytics, and finance & ops analytics. It also offers a community with contributors, events, and support channels.

Airbyte is an open-source data integration engine that helps users consolidate their data in various storage systems. It offers a Free Connector Program, allowing users to try out its 300+ connectors for free. The platform supports extracting and loading data from multiple sources to different destinations, as well as transforming it using SQL or dbt. Airbyte also provides a Connector Development Kit for building new connectors in under 30 minutes. Its community includes over 10,000 members who share tips and receive support through various channels such as Slack, Discourse, and YouTube.

Airbyte is an open-source data integration engine that helps users consolidate their data in various storage systems such as warehouses, lakes, and databases. It offers a Free Connector Program with 300+ connectors for unlocking all your data. The platform also provides a 14-day free trial of Airbyte Cloud. Users can join the vibrant community on Slack to get help from fellow members and access troubleshooting insights, GitHub repositories, and more resources.

Airbyte is a fully-managed platform that enables secure data movement for organizations, allowing users to embed hundreds of integrations into their apps in minutes. The platform offers reliable database and API replication at any scale, as well as embeddings from unstructured data, making it easy to build new connectors in just 10 minutes. With Airbyte, developers can make sense of unstructured data using large language models, manage high-volume databases with low latency, and easily collect credentials from end-users. The platform also offers a community-driven approach, with guides, knowledge base, GitHub discussions, and live events for users to learn from each other's success and stay up-to-date on the latest developments in data engineering. Overall, Airbyte is designed to help organizations streamline their data management processes and improve their data replication game.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to embed hundreds of integrations into their applications in a single step. With its reliable database and API replication capabilities, Airbyte supports high-volume databases with low latency and can handle large-scale data movements. The platform also provides embeddings from unstructured data, enabling users to build new connectors in just 10 minutes. Additionally, Airbyte offers a range of resources and guides to help users evaluate their costs, choose the right solutions, and improve their data replication game. With its large community of over 30,000 subscribers, Airbyte provides access to knowledge base, GitHub discussions, and live events by the team, as well as opportunities for users to contribute to the open-source project and become technology or consulting partners.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to embed hundreds of integrations into their applications in minutes. With its reliable database and API replication capabilities, Airbyte supports high-volume databases with low latency, making it an attractive solution for various industries such as marketing, sales, product, finance, engineering, and more. The platform also provides embeddings from unstructured data, enables users to build new connectors in just 10 minutes, and offers a vast knowledge base and community support, including live events and GitHub discussions. By choosing Airbyte, organizations can simplify their data replication processes, reduce costs, and improve their overall data engineering capabilities.

Airbyte is a fully-managed platform that allows companies to securely move their data between different systems in minutes, with features such as secure data movement, reliable database and API replication, and embeddings from unstructured data. It also provides a range of integrations with popular tools and services, including Python development frameworks, and supports high-volume databases with low latency. Additionally, Airbyte offers a community-driven approach with guides, knowledge base, and live events, as well as opportunities for developers to contribute to the platform and become technology or consulting partners. The company has gained significant traction with over 40k+ companies using its platform, and has a strong presence in the data engineering and OSS communities.

As of 2023, Airbyte has over 300 connectors available for data integration tasks. The company uses a grading system called "Connector Release Stages" to ensure the reliability and quality of these connectors. Connectors are grouped into four categories: Pending, Alpha, Beta, and Generally Available (GA). Each stage has specific requirements that must be met before moving on to the next one. The main goal of certifying a connector is ensuring that it performs well against various use-cases and test expectations. Airbyte's testing process includes unit tests, integration tests, acceptance tests, and certification for unknown issues. Connectors are released in stages to ensure users only pay for reliable and well-tested connectors. The company plans to grow the number of sources and destinations supported by Airbyte in 2023.

Data security is crucial for businesses that store valuable customer information, and Snowflake offers robust security features to ensure data protection. Key aspects of data governance include data masking, access control, and classification and tagging. With Snowflake's dynamic data masking feature, users can hide sensitive information based on their roles. Access control helps limit who can view or modify data, while tagging and classifying organize data to quickly identify sensitive information. By implementing these best practices, businesses can maintain a secure environment for handling customer data.

Google BigQuery is a powerful and widely used cloud-based analytics tool designed for data-driven enterprises. It was created in 2010 by a team of engineers at Google's Seattle office to leverage the company's advanced internal storage, computing, and analytics tools. Its key features include its scalable design, serverless architecture, and ability to quickly process large datasets using standard SQL. BigQuery is built on top of Dremel, a revolutionary internal Google tool that enables engineers and product managers to perform interactive ad-hoc analysis of read-only nested data on web-scale applications. It also relies on Colossus, a system optimized for reading large volumes of structured data using Capacitor – Google’s columnar storage format – and efficient compression. BigQuery's decoupled storage and computation architecture leverage column-based partitioning to minimize the amount of data read from disk by slot workers. It allows for unlimited data manipulation language (DML) statements, including insert, update, or delete a large number of rows in a table in a single job. BigQuery's storage pricing model is based on the amount of data stored and whether the storage is considered active or long-term. The platform also offers partitioning and clustering features to help improve query performance and make managing and analyzing data easier. Data can be ingested into BigQuery using various methods, including batch ingestion from Cloud Storage, streaming ingestion through tabledata.insertAll function or the BigQuery Storage Write API, and utilizing the Data Transfer Service for importing data from external sources. Querying data in BigQuery is compatible with standard SQL and legacy SQL, and it offers a range of functions and operators for tasks such as text manipulation, date/time calculations, mathematical operations, and JSON extraction. Additionally, BigQuery ML allows users to build and evaluate machine learning models within the platform without needing extensive programming or knowledge of ML frameworks.

Enterprises often store large amounts of unstructured data, which can be challenging to analyze or repurpose due to the lack of tools designed for handling such data. Approximately 70%-90% of enterprise data is unstructured, while only 20% is structured and 5%-10% is semi-structured. Many data analysis and business intelligence tools are tailored for structured data, leading to situations where companies collect vast amounts of data but lack the means to utilize it effectively. To analyze unstructured data, additional engineering efforts may be required to structure the data before Data Analysts can work with it. Unstructured data often needs to go through data wrangling and cleansing processes before it can be used for analysis.

Airbyte is a fully-managed data integration platform that allows companies to securely move their data across different systems and scales, with the ability to embed hundreds of integrations in a single application. It provides reliable database and API replication at any scale, as well as embeddings from unstructured data, making it easier for developers to make sense of complex data sets. With Airbyte, users can build new connectors in just 10 minutes, deploy their use case in minutes, and easily collect credentials from end-users, among other features. The platform is used by over 40,000 companies and has a large community of developers and data engineers who share knowledge and best practices through various channels such as GitHub Discussions, community calls, and live events.

Airbyte is a fully-managed platform that allows users to securely move their data between different applications and services, with the ability to embed hundreds of integrations into an application in just one minute. The platform has been used by over 40,000 companies and provides reliable database and API replication at any scale, as well as embeddings from unstructured data. It also offers a range of features such as low-latency high-volume databases, easy credential collection from end-users, and a community-driven approach to support users in their journey. Additionally, Airbyte provides guides, knowledge base access, and live events to help users learn and grow with the platform. The company is committed to becoming a technology or consulting partner for users who want to integrate Airbyte into their products.

Mid-to-large-sized enterprises often struggle with setting up a Modern Data Stack (MDS) due to the complexity of their existing systems and the need for top-level management support. These companies typically use large ERP, CRM, and other internal systems to hold business data. The adoption of MDS tools is still in its early stages, as most are brand new and have not yet been widely adopted by larger enterprises. However, the open-source Modern Data Stack offers opportunities for these companies, such as lowering legal risks, enabling more users to perform data engineering, and providing a technological advantage over competitors who remain less data-driven.

Airbyte is a fully-managed platform that enables secure data movement across organizations, allowing companies to integrate 100s of integrations into their apps in minutes. With its reliable database and API replication capabilities, Airbyte supports high-volume databases with low latency, making it suitable for various industries such as marketing, sales, product, finance, engineering, and more. The platform also provides embeddings from unstructured data, allows users to build new connectors in 10 minutes, and offers a knowledge base and community support for its 30k+ subscribers. Airbyte's developer advocate program aims to promote data engineering thought leadership and provide resources for developers to learn and contribute to the platform.

In December, Airbyte organized its first move(data) conference with over 3,300 attendees and 40 lightning talks from speakers across the data stack. The event's videos are now available online. Airbyte Cloud now supports custom connectors, allowing users to build their own connectors for unsupported data sources. Additionally, API calls can fully control Airbyte Cloud resources, enabling automation and integration with other systems. Airbyte OSS received an update that displays throughput and progress bars during syncs, making it easier to track sync status and understand how data is being transferred. The changelog includes new sources, promotions, features for existing connectors, UI updates, bug fixes, and breaking changes.

The article explores core open-source tools needed for a company to become data-driven, focusing on integration, transformation, orchestration, analytics, and ML tools as part of the latest open data stack. It introduces the Modern Data Stack (MDS) and explains how it is extendable like lego blocks, usually consisting of data integration, a transformation tool, an Orchestrator, and a Business Intelligence Tool. The Open Data Stack is then defined as a better term for the MDS, focusing on solutions built on open source and open standards covering the data engineering lifecycle. The article presents Airbyte as the number one choice for data integration, highlighting its reliability, extensibility, integrations, and transparency. It also discusses dbt as the king of SQL for data transformation, emphasizing its documentation generation, reusability of SQL statements, testing, source code versioning, added functionality to plain SQL with Jinja Templates, and Python support. For analytics and data visualization (SQL), Metabase is recommended due to its simplicity and ease of setup for non-engineers. It allows users to ask questions about their data and displays answers in formats that make sense, whether a bar chart or a detailed table. The last core data stack tool discussed is Dagster as the data orchestrator, which enforces best practices such as writing declarative, abstracted, idempotent, and type-checked functions to catch errors early. Additional components of the Open Data Stack are also mentioned for inspiration, including semantic layer/metric layer, data quality and data observability, reverse ETL, and data catalogs. The article concludes by encouraging readers to explore these tools in action through a tutorial on configuring Airbyte connections with Python (Dagster).