July 2024 Summaries

A Data Platform Engineer is an expert who builds and manages integrated data platforms that provide a cohesive system for delivering reliably every time, enabling organizations to proactively harness their data for strategic advantage. These platforms are different from ad hoc ETL setups in that they offer a holistic approach, scalability, decentralization, and data democratization, allowing domain-specific teams to manage their data independently. The role requires expertise in architecture, infrastructure as code, software development best practices, and understanding of the systems they impact, complementing traditional Data Engineers who focus on operational aspects of data management. Data Platform Engineers are senior strategists who navigate team psychology, ensuring everyone is aligned and motivated to make data usage ready for production, and are the unsung heroes shaping the future of their companies by building modern, scalable, and governed data platforms.

The author of this text is an Open Source Software Engineer at Dlt, a Python library that lets you build data pipelines as code. The library uses Apache Arrow to make pipelines faster by representing tabular data in memory more efficiently. The Arrow format is better than native Python objects (list of dictionaries) because it enables offloading computation to Arrow's fast C++ library and prevents processing rows one by one. The author explains how dlt works at a high level, including the three main steps: extract, normalize, and load. They also describe two pipeline "routes" - traditional and Arrow - which differ in how they represent tabular data in memory and persist it to disk. The Arrow route is faster because it uses schema-aware pyarrow objects that can be processed concurrently in C++. The author concludes that using Arrow improves performance significantly, especially in the normalize step, where it can process batches of values concurrently.