March 2024 Summaries

The Data Engineering Lifecycle is a process used by data teams to author, evolve, and maintain data pipelines and the data assets that those pipelines produce. The lifecycle consists of four phases: Development, Verification, Deploying to Production, and Monitoring and Debugging. In the Development phase, code is written and tested to improve or expand data pipelines. In the Verification phase, problems are caught before they reach production through a mix of manual and automatic testing. During this phase, tools like continuous integration (CI) infrastructure and automated unit tests help ensure that changes are thoroughly tested before deployment. The Deploying to Production phase involves getting new code running in production and updating data assets to reflect the changes. Finally, in the Monitoring and Debugging phase, issues are discovered and addressed by fixing upstream data sources or re-executing pipelines. A healthy Data Engineering Lifecycle enables fast shipping and improvement of data pipelines without impacting production data quality or timeliness.

Dagster Cloud is a data management platform designed to support compliance with the Basel Committee on Banking Supervision's Standard Number 239 (BCBS 239), which focuses on effective risk data aggregation and reporting in banks. The standard emphasizes robust systems for aggregating risk data, effective risk reporting mechanisms, strong governance and internal controls, compliance, and implementation. Data engineering teams play a critical role in implementing and maintaining the necessary data management systems to meet these demands. Dagster Cloud offers features that can support BCBS 239 compliance, including ensuring data quality, traceability, governance, documentation, robust data aggregation, change management, scalability, security, and performance. While Dagster provides tools to support compliance, actual compliance will depend on how these tools are implemented within an organization, requiring a combination of technology, processes, and organizational policies.

We are pleased to announce a new integration that will allow data practitioners to easily include OpenAI API calls as part of their data pipelines, providing insights to optimize API calls and credit consumption. This integration enables the use of generative AI in various scenarios such as document summarization, sentiment analysis, and language translation. Dagster Labs has successfully implemented this integration to build a pipeline that handles complex support requests, provides first-stab answers to user questions, auto-categorizes issues, and generates learning summaries on a weekly basis. To manage costs effectively, the platform introduces features like OpenAIResource and with_usage_metadata, ensuring uniform resource utilization across the platform. These features are integrated seamlessly with Dagster Insights for enhanced analytical capabilities, allowing users to monitor and optimize their data pipelines while maximizing the benefits of generative AI.

Nick Schrock, the co-creator of GraphQL and innovative mind behind Dagster Labs, shares his journey from Facebook to solving data and machine learning's pressing issues through Dagster, an open-source data orchestration platform. He explores the integration of machine learning, AI, and data engineering, offering insights on open-source development's role in modern data infrastructure and advice for technical founders. Nick discusses balancing leadership with an engineering vision, sharing challenges and experiences from his journey as a leader in the field of data science and engineering.

The Dagster team has released a new guided course called Dagster + dbt, which is part of their Dagster University offerings. The course covers the integration of Dagster and dbt, a popular data engineering tool, in seven lessons with real-world examples and quizzes to check knowledge. This follows the success of their previous course on Dagster Essentials and aims to make adoption easier for users. The new course builds upon the revamped `dagster-dbt` integration library, which was released last year, offering a more generic and flexible integration that empowers users to customize various aspects of the tool. The course is free and available to everyone, with the option to track progress and learn at one's own pace.

Modern organizations face challenges in utilizing their vast data assets to drive informed decision-making, mainly due to organizational complexity and siloed departments. Data democratization offers a solution by making data accessible to all stakeholders without compromising security or integrity. This approach involves implementing a balanced data platform that balances centralized control with decentralized access, empowering individuals across the organization to leverage data for decision-making. By adopting a data democratization strategy, organizations can enhance data quality, streamline operations, and foster a culture of collaboration and innovation. The key benefits include improved data accessibility and reduced silos, establishment of a single source of truth, tribal knowledge sharing and documentation, and enhanced innovation and time-saving via unified data platforms. Organizations can assess their current data landscape, implement robust data governance, select the right tools, foster a data-driven culture, and continuously monitor progress to successfully democratize their data.