October 2024 Summaries

The Dagster 1.9 release, named Spooky, is a significant update that introduces declarative automation, which marks the system as stable and extends it to asset checks. This feature allows users to automatically materialize assets when specific criteria are met, such as waiting for new upstream data. The release also introduces integrations with business intelligence (BI) tools like Tableau, Power BI, Looker, and Sigma, enabling users to orchestrate updates to these assets and simplify data refreshes. Additionally, a toolkit called Airlift is introduced to accelerate migration from Airflow to Dagster, facilitating a three-step process of peer observation, model lineage tracking, and API invocation. The release includes changes to the UI information architecture to make important entities easier to find, such as moving backfills to the runs page and introducing new pages for jobs and automations. Other notable features include asset kinds with labels that show up on asset nodes in the UI and a more general compute_kind attribute. The community is thanked for their efforts in making this release possible, and users are encouraged to provide feedback and participate in future releases.

The long-term impact of AI on data engineering jobs is a story of transformation—both in the skill sets required and the nature of the work itself. Data engineers will need to embrace AI as both a challenge and an enabler: acquiring new AI-related skills to meet the growing demands of the industry while leveraging AI tools to boost productivity and enhance creativity. The future will favor engineers who adapt to these changes, blending traditional data engineering expertise with an evolving AI-first approach. As AI adoption is expected to be personal rather than a broad process change, it's recommended that executives encourage experimentation and sharing of best practices on how AI can boost productivity, allowing team members to figure out their adoption journey. Additionally, AI tools will empower data engineers by fundamentally changing how we work, with coding assistants providing the most value during an engineer's ramp-up phase, while introducing Gen AI into ideation sessions has been shown to stifle innovation in some studies. Ultimately, the advent of Gen AI will only ramp up expectations of what the data engineering function can deliver, requiring a sizable investment from executives and engineers alike.

KIPP, a nationally recognized network of free, open-enrollment, college-preparatory public schools, faced challenges in managing its complex and fragmented data stack. As the sole data engineer for KIPP's Newark, Camden, and Miami regions, Charlie Bini sought to find a solution that could support growth, streamline data operations, and provide visibility into the data stack. He experimented with various tools before adopting Dagster, an emerging orchestrator that promised more visibility and control over data pipelines. With Dagster, KIPP centralized its data processes, bringing together data extraction, transformation, and orchestration into a unified platform. The new data stack empowered the team to work more efficiently, reduced data quality issues, and provided real-time alerts and lineage tracking, making it easier to respond to issues immediately. By adopting Dagster, KIPP transformed its data platform from a fragmented set of ad-hoc tools into a unified, resilient, and scalable solution that resonated with everyone.

The text discusses the challenges of managing data in modern organizations, where data generation outpaces indexing capabilities. Data catalogs are introduced as a solution to unify orchestration and data cataloging, providing a searchable and centralized view of data assets. A built-in catalog integrated into the orchestration platform gives data platform engineers a serious advantage, allowing them to quickly search for datasets, access definitions, transformations, and lineage with real-time metadata, reducing tool fatigue, and fostering collaboration among teams. The text highlights the benefits of using software-defined assets as the source of truth for catalogs, providing versioned and collaborative code, supporting automation, flexibility, and avoiding redundant work. It also emphasizes the importance of breaking silos and onboarding new team members faster with a unified view of data. Finally, it showcases Dagster's capabilities in simplifying data management with a built-in catalog, eliminating the need for standalone tools, and providing real-time accuracy, streamlined management, enhanced visibility, and reduced overhead.

No-code development platforms promise quick pipeline management, orchestration, fast deployment, and quick wins without writing a single line of code. However, even the best no-code platform will fall short when building scalable, maintainable, complex data pipelines and platforms. No-code solutions often fail due to limited flexibility and customization, poor scalability, limited or shallow integration capabilities, security and compliance issues, vendor lock-in, platform-specific skills with limited transferability, knowledge bottlenecks, tech debt accumulation, inconsistent practices, hidden costs and financial overhead, and the need for hybrid code approaches that combine the ease-of-use of no-code platforms with the flexibility and power of traditional coding. Hybrid code solutions are more reliable and flexible for managing data assets, offering a balance between speed, flexibility, and scalability.