October 2022 Summaries

The debate between Data Nets and Data Mesh has become a hot topic in the data management world. Proponents of Data Nets argue that this new approach leverages advances in Neural Nets and Generative AI to create an AI-first data stack, while those in favor of Data Mesh contend that it is merely a rehashing of old ideas with some new features. Both sides have their merits, but the key difference lies in how they handle data processing and management. Data Nets are designed to take advantage of the latest advances in artificial intelligence and neural networks to automatically ingest, cleanse, transform, and aggregate data from multiple sources without human intervention. This approach aims to replace traditional data engineering tasks with AI-powered pipelines that can detect and recover from downtime or schema changes in real time. On the other hand, Data Mesh focuses on how data is managed within an organization, while Data Contracts define the interface between different software components. Both of these approaches are limited in scope compared to Data Nets, which offer a more complete picture of what is possible with data today. Some factors to consider when deciding whether or not a Data Net is right for your organization include the need to ingest data from multiple sources and formats, automate the creation and management of data pipelines, monitor and improve their performance over time, detect and recover from data downtime or schema changes in real-time, and generate realistic data mocks for testing purposes. The controversy surrounding Data Nets stems from concerns about precision and flexibility when defining business metrics, reliance on artificial intelligence and neural networks, and potential bias introduced by generative AI. However, proponents argue that Data Nets offer better performance than traditional data stacks due to their use of parallel processing, distributed computing, complete observability, and ability to automatically detect and recover from downtime or schema changes in real time. In conclusion, while both Data Nets and Data Mesh have their merits, the new AI-first approach offered by Data Nets represents a departure from traditional data stacks that could revolutionize how we manage and process data in the future.

The General Data Protection Regulation (GDPR) is a strict privacy and security law that applies to organizations handling data related to people in the European Union. Violations can result in harsh fines, reaching into the tens of millions of euros. Pseudonymisation is a standard requirement for data used in statistical production under GDPR. Airbyte, an ELT tool, has introduced a low-code connector framework that enables users to build source connectors for REST APIs by modifying boilerplate YAML files. This tutorial demonstrates how to incorporate "tweak" transformation functionality into a low-code connector configuration to pseudonymise sensitive fields, remove sensitive fields and records, and limit data sent to the destination.

Rust is a multi-paradigm programming language that has gained popularity for its ability to create highly concurrent, safe, and performant systems. While it may not replace Python outright, there are increasingly many projects trying to integrate Rust into Python/Data Engineering. Rust's strong type and system checks during compile-time make it a good choice for data engineers who want to prevent errors in their code. Additionally, its speed as a compiled language and ability to easily integrate with other languages through foreign function interface (FFI) are beneficial features. However, Python remains the go-to language for data engineering due to its versatility and widespread use in existing tools and frameworks. Rust may continue to grow in popularity for vital and core components of data engineering but is unlikely to replace Python as the primary language in the foreseeable future.

The data engineering role has evolved significantly since its inception, with new technologies and concepts emerging rapidly. From the early days of business intelligence and big data to the modern data stack and cloud-based solutions, the field has matured and become more specialized. Data engineers now focus on strategic tasks such as data modeling, quality, security, management, architecture, and orchestration while adopting software engineering best practices. The future of data engineering points towards increased abstraction, simplification, and maturity by adopting concepts like managed data connectors, specialization roles, improved data management through DataOps, and a closer collaboration between producers and consumers. Despite the changes in tools and practices, data engineering will always be necessary as companies increasingly rely on data for decision-making.

In September, Airbyte released the most number of General Availability (GA) connectors and new sources in a single month. The company also launched its Low-code CDK for Hacktoberfest, offering rewards for contributions to building connectors and reporting bugs. Additionally, several blog posts and YouTube content were published covering topics such as semantic layers and sync intervals. Airbyte's team members will be speaking at dbt Coalesce in New Orleans, and the company is hosting its own move(data) conference on December 7-8. The Drip also highlights new sources and promotions for various connectors.

This article provides an overview of different replication modes supported by Airbyte, a popular open-source data integration tool. It covers the ELT approach to data integration and why it is preferred over ETL. The article also explains various concepts such as sync runs, cursors, primary keys, full refresh sync, incremental sync, CDC replication, and more. Furthermore, it discusses the benefits and trade-offs of each replication mode and helps users choose the best option for their use case.

The company, Airbyte, has launched a survey to understand the sentiments of its community regarding data engineering. The results will help identify industry trends and provide useful information for participants. Some questions that may be answered include whether there is a hiring freeze in data right now, how much money one should be making based on geography, team size, and years of experience, comparison between Airflow, Dagster, and Prefect, favorite industry newsletters and podcasts, etc. The survey will take approximately 15 minutes to complete and is open until October 20, 2022. The results will be announced at the company's conference, move(data), dedicated to data engineering advancement and practice.