May 2023 Summaries

During Hack Days at Airbyte, a two-day objective was set to boost the speed of their front-end end-to-end (e2e) test suite. The team optimized the e2e suite using Airbyte's Config API to generate fixtures for sources, destinations, and connections. They aimed to reduce the slowest suite's runtime by about 30%. With previous groundwork in place, they were able to leverage the config API in three suites, remove redundant tests from three suites, and increase coverage in others. The result was an overall decrease in frontend e2e runtime of ~60% and a significant reduction in runtime within the slowest suite (~96%). Some tests were moved or added/removed to improve reliability and debugging capabilities.

Airbyte, a data integration platform, introduced a hackday project to improve the visibility of synchronization processes. The project aims to provide a centralized view of connection events and reduce reliance on logs or internal orchestrator UI for understanding synchronization steps. By adding an API to add and list events, granular error tracking is made possible, making it easier to understand why a connection failed. This POC could also be used for auditing purposes. The hackday project demonstrated that a better way to expose what happens in a connection is feasible and valuable, potentially replacing the existing log view and addressing most investigation needs.

The text introduces a content hub dedicated to all things data engineering, featuring various content formats catering to different learning styles and seniority levels. It covers topics related to data engineering, including databases, orchestration, transformation, modeling, warehousing, AI, and more. Contributions from data professionals are welcomed, with payment offered for accepted articles. The best content will be promoted on social media channels and featured in the newsletter.

Data lineage is a crucial aspect of data management, providing a traceable route that marks the origin, transformations, and final destination of data. It plays a vital role in improving data quality, risk management, regulatory compliance, and optimizing data flows. There are three main types of data lineage: end-to-end, vertical, and horizontal. Implementing data lineage can be challenging due to the complexity of data systems, lack of standardized tools, evolving data regulations, data volume and velocity, and resource intensity. Tools like Collibra, Azure Purview, Open Metadata, MonteCarlo, Alvin, Marquez, and Apache Atlas can aid in implementing data lineage. The future of data lineage includes automation, integration with machine learning and AI, expansion of column-level lineage, greater emphasis on data privacy and security, increased use of visualizations, and cross-functional collaboration.

In this final installment of the "Data Modeling" series, we delve into data architecture patterns and their influence on data modeling. We explore general and specialized patterns, such as batch vs. streaming and lakehouse vs. warehouse, and discuss the role of a semantic layer in complex data architecture. We also survey the landscape of data modeling tools, comparing commercial and open-source options, and ponder the potential of AI in data modeling. To wrap up, we introduce data modeling frameworks like ADAPT™ and BEAM, designed to guide effective data model creation.

The State of Data research is a comprehensive study on the evolving trends and sentiments within the data engineering ecosystem, with 886 participants from various continents and experience levels. Key insights include the dominance of Airbyte in data ingestion for small/medium-sized businesses and enterprises, dbt's popularity in data transformation, Snowflake and BigQuery leading in data warehouses, and a growing interest in reverse ETL tools like Hightouch and Census. The research also highlights popular community influencers and resources such as newsletters, podcasts, and YouTube channels. With plans to expand the survey annually, this study aims to provide valuable insights into the evolving landscape of data engineering.

During the recent Hack Days at Airbyte, a director of engineering developed an iOS application using the company's new Public API. The app allows users to check in on their Airbyte workspaces and manage jobs. It was built with React Native and Expo, which rendered native components for both iOS and Android platforms. The developer praised the progress made by React Native since its early days and highlighted how well it worked with Expo. However, they faced some challenges with CORS on the web but managed to work around them. The code is available for anyone interested in exploring it further.

The Connector Builder, a new self-serve, no-code tool, has been released by Airbyte. It enables users to build new API connectors and directly connect them to any Airbyte destination in as little as 15 minutes without requiring coding experience or development environment. Since its soft launch two months ago, over 100 connectors have been built using the Connector Builder and deployed to production to support critical data movement workloads. The tool is best suited for synchronous HTTP API connectors, but it will evolve to support more components in the future.

On May 16th, 2023, a new program called the "Free Connector Program" was introduced. This program offers limitless data movement with free Alpha and Beta connectors. The details of this initiative are yet to be disclosed but it aims to revolutionize data management by providing seamless integration across various platforms.

The author shares their experience of transitioning from a data engineer to an analytics engineer. They emphasize the importance of following one's interests and exploring different roles within a company. Analytics engineering is described as more customer-oriented, focused on driving growth for the business through data analysis. The author discovered this role after realizing they wanted more connection with the business side of things. Key skills to develop before transitioning include dbt, business communication, and building data pipelines. They also share what they wish they knew before making the switch, such as embracing their existing data engineering skills and not being afraid to explore new roles. The author concludes by encouraging others to take risks in their careers and try out different roles to find fulfillment.

Online Transaction Processing (OLTP) is a critical component of modern data systems, enabling the efficient handling of transaction-based data in real-time. It is essential for supporting business operations that require rapid processing and management of large volumes of data, such as order entry, retail sales, and financial transactions. The process involves a robust data movement infrastructure that allows businesses to transform raw data into actionable insights. OLTP systems are designed to ensure data integrity and consistency, offering high availability and scalability to meet the demands of dynamic business environments.

In this edition of dataNews.filter(), we cover Entity-Centric Data Modeling (ECM) and its benefits, latest updates in data engineering tools and techniques, inspiring reads and discussions, and trending topics on social media platforms. We aim to provide a reliable stream of relevant content for the hypergrowth of the data ecosystem. The format is minimal and succinct, with added thought leadership, providing you with the highest value in the shortest time.

Part 2 of this series focuses on various data modeling approaches and techniques, which play a vital role in designing and structuring data models. These methods help organizations gain valuable insights from their data. The text discusses top-down and bottom-up approaches to data modeling, as well as conceptual, logical, and physical data models. It also covers different data modeling techniques such as dimensional modeling, data vault modeling, and more. Additionally, the text addresses common challenges faced in data modeling and how to mitigate them. By understanding these approaches and techniques, data engineers can better navigate the complexities of data modeling and design systems that cater to their organization's specific needs and goals.

Maintaining a robust data ingestion process requires adopting best practices from software engineering and devOps, including reproducibility, automation and orchestration, and observability. Reproducibility ensures that the same inputs are provided to data jobs consistently, while automation and orchestration provide ways to execute jobs when needed and trigger downstream tasks based on upstream task completion. Observability allows monitoring of jobs to ensure they ran as intended. To achieve these goals, version control tools like git and platforms like GitHub can be used for tracking changes to project code and configurations over time. Additionally, maintaining consistent package versions helps keep the system running as intended. Automating data ingestion processes with scheduling or orchestration tools ensures that jobs run precisely when needed and in the correct order. Observability requires notifications, logging infrastructure, and monitoring of ingested data for quality issues to enable rapid problem-solving when failures occur. By embracing devOps best practices, data teams can reliably deliver high-quality data with lower maintenance overhead and unforeseen issues.

Simon Späti, a Data Engineer and Technical Author at Airbyte, discusses the advancements in the field of analytics engineering and emphasizes the importance of staying abreast of emerging open-source technologies. With over 15 years of experience, Späti highlights his dedication to innovation and entrepreneurship within the data ecosystem. Readers are encouraged to explore Airbyte's new platform and its latest features, with an invitation to join their newsletter for insights into the data stack. The blog also references related posts that cover updates and achievements within Airbyte, including improved performance with Redshift, certification of MotherDuck as a destination, and a significant return on investment identified by Forrester for Airbyte users.

In April, Airbyte launched its API for programmatic use of Airbyte Cloud. The API supports orchestration and the Terraform SDK. Additionally, Airbyte changed its support approach to accommodate the growing number of active users and contributors. Changes include a new blog post on community PRs and updates to response times. In April's changelog, Airbyte introduced several new connectors and features for existing ones. The company also offers a free Alpha and Beta connector program and a 14-day trial of Airbyte Cloud.

The blog post titled "Discover the Modern Transactional Stack" delves into the infrastructure that supports transactional data processing, highlighting its significance in the modern data ecosystem. Authored by Simon Späti, a seasoned Data Engineer and Technical Author at Airbyte with over 15 years of experience, the article emphasizes the importance of staying abreast of innovative and emerging open-source technologies. Additionally, the blog encourages readers to explore Airbyte's new platform, Airbyte Agents, offering early access to its features and inviting them to join the newsletter for insights into the data stack. The post is part of a series of related articles that explore advancements in data processing, including improvements in Airbyte's integration with Redshift and the addition of MotherDuck as a certified destination, as well as a Forrester report highlighting a substantial ROI for Airbyte users.