January 2024 Summaries

Dagster is a data orchestrator that aims to empower organizations by building productive and trustworthy data platforms. The company's mission is rooted in a set of beliefs about how data pipelines and platforms should be built, which include thinking of data engineering as software engineering, embracing heterogeneity in data platforms, unifying them across the organization, adopting declarative approaches over imperative ones, and centering assets such as data assets. These beliefs have influenced Dagster's design principles, including code being king, embracing non-production environments, multi-tenancy from day one, and making choices that prioritize doing the right thing over ease in the short run. The company encourages users to adopt these principles by providing a tool that makes it easy to develop pipelines in code, declaratively, and link them into a unified whole.

Pedram Navid, the Head of Data Engineering and DevRel at Dagster Labs, explores the role of AI and large language models in data analysis on the podcast "Data Driven". While AI has limitations, it offers new ways to explore and leverage data, transforming the analysis process. AI and LLMs also democratize access to data analysis for non-technical people, enabling them to derive insights from complex datasets without feeling overwhelmed. The discussion highlights the potential of AI and LLMs in data, despite their limitations.

Pedram Navid, Head of Data Engineering and DevRel at Dagster Labs, explores the role of data products and AI in data. With rising data literacy levels, organizations are shifting towards viewing data as a product to enable self-service and empower teams to make business decisions. In this episode of Data Driven, Pedram discusses cutting through the noise of data products.

Dagster 1.6 has been released, bringing several major enhancements to the tool. The UI has seen significant upgrades, including a dark mode feature that matches the user's system theme, and improvements to the asset lineage graph, such as easier visualization and navigation of large graphs. Other new features include report asset materializations from the UI, backfill previews, triggered run tracking, and sensor or schedule tick logs accessible by default. Additionally, three new APIs, MaterializeResult, AssetSpec, and AssetDep, have been marked as stable after being previously in an "experimental" status. Dagster Pipes have also seen improvements, including easier integration with AWS Lambda, report arbitrary messages between pipes processes, and termination forwarding. The development team would like to thank all the community members who contributed to Dagster since the 1.5 release.

Retain.ai, a company that provides software for data management and analytics, has joined forces with Dagster Labs, a developer-centric platform that helps companies manage their data pipelines. The agreement includes key members of the Retain.ai team joining Dagster Labs, including Eric Chernoff, who will be working closely with the Dagster team to accelerate innovation and deliver industry-best solutions for its customers. This move is seen as a strategic partnership that aims to support customers and partners while driving growth through an aggressive product roadmap. The company is not disclosing further details about the transaction at this time, but invites users to reach out with feedback or questions in various channels such as Slack, Github, or by checking their open job listings.

In this recent episode of The Data Stack Show, Sandy Ryza, Lead Engineer at Dagster Labs, shares insights on data cleaning, data engineering processes, and the need for improved tools. He introduces Dagster, an orchestrator that focuses on assets like tables, datasets, and machine learning models, contrasting it with traditional workflow systems. Dagster's integration with dbt is also discussed, as well as the changing dynamics in data roles, the impact of modern tooling, and potential for increased creativity in the field. The conversation highlights the evolving landscape of data orchestration and the importance of innovative tools like Dagster.