October 2023 Summaries

Nick Schrock, the founder and CTO of Dagster Labs, sat down with Jason Davis, CEO and co-founder of Simon Data, to discuss how to work effectively with data teams in a podcast episode called "Data Unlocked". Nick has extensive experience working for top companies such as Microsoft, Facebook, and Care Evolution before founding his own company. In the conversation, he and Jason discussed three main questions about collaboration between data engineering and go-to-market teams, including effective collaboration strategies and steps to improve cross-functional teamwork. The podcast is available for listening or viewing on YouTube, and viewers are encouraged to provide feedback through various channels, including the Dagster community Slack and GitHub discussions.

The Craft Of Open Source - a Flagsmith podcast` discusses the challenges of data management in today's world and how companies like Dagster Labs are addressing these issues with data pipelines and orchestrators. Pete Hunt, CEO of Dagster Labs, joins Ben Rometsch to share his career journey and explain how Dagster is revolutionizing data orchestration. With a recent $33 million Series B funding round, Dagster Labs is poised for continued growth and innovation in the field of data science. The podcast also invites listeners to provide feedback and get involved with the community through various channels such as Slack and Github.

CI/CD (Continuous Integration and Continuous Deployment) is a concept central to software development, now also applied in data engineering to automate the testing, integration, and deployment of data pipelines. It merges development, testing, and operational workflows into a unified, automated process, ensuring high-quality data assets and reliable data infrastructure. Data engineers use tools like Git, GitHub Actions, Bitbucket Pipelines, and Buildkite Pipelines to streamline tasks, reduce human errors, and ensure data pipeline reliability. The integration of Git with CI/CD solutions enables the automation of repetitive tasks, ensures data quality, and focuses on optimizing data pipelines. By adopting best practices in Git, such as handling large data files, using pull requests, code reviews, and atomic commits, data engineers can foster collaboration, efficiency, and robustness in their workflows.

The Tech Trek podcast features an interview with Pete Hunt, CEO of Dagster Labs, a company revolutionizing the data industry through open-source data orchestration. Dagster aims to make sense of messy data by creating pipelines that can efficiently process and transform large datasets. The podcast highlights the importance of open-source nature in addressing challenges in the data orchestration market.

Dagster External Assets provide a solution to enable the adoption of Dagster as a "single pane of glass" for data platforms without requiring a wholesale migration of all scheduling and orchestration infrastructure. This feature enables teams to model data assets that are not scheduled and orchestrated by Dagster, allowing them to leverage Dagster's metadata capabilities independently from its orchestrator. External Assets can be declared in a declarative fashion within Dagster, providing a clean separation between the orchestration and metadata layers. The primary goal of External Assets is to enable seamless integration with existing platforms and stakeholders without requiring synchronous migration of all infrastructure. This feature provides value by enabling system of record for metadata, alerting, cross-cutting data lineage, data quality, and scheduling capabilities, while also providing an incremental pathway for adoption.

Dagster Pipes is a protocol introduced by Dagster that standardizes the process of invoking and retrieving structured results from external computations, allowing for lower complexity burdens for both infrastructure engineers and stakeholders writing business logic. It decouples orchestration and execution in Dagster while retaining orchestration and metadata-driven features, enabling users to launch code in external environments with minimal additional dependencies and get parameter passing, streaming logging, and structured metadata back into Dagster's lineage, observability, and data quality tools. Pipes is designed for teams that have invested countless engineer-hours into their existing data platform infrastructure, allowing them to adopt capabilities on top of software that works without a risky, costly rip-and-replace approach. The protocol is pluggable on a per-capability basis, supporting first-class multi-language support in Dagster and enabling users to write business logic and transformations in any programming language, including Rust, Julia, R, Java, and others. With Dagster Pipes, users can bring every computation that produces data into the fold, apply observability, lineage, data quality, consumption management, and great developer workflow across the entire platform.

The text discusses the challenges of data ingestion, a crucial part of the data engineering lifecycle. It highlights the limitations of traditional approaches, such as rolling one's own solution or relying on heavyweight third-party frameworks, and notes that many teams struggle with the complexity of building reliable ingestion pipelines. The author introduces Dagster Embedded ELT, a new library that aims to simplify data ingestion by providing pre-built assets and resources around lightweight frameworks. This allows users to build ELT pipelines with Dagster without duplicating the complexity of heavier ingestion frameworks. The library is designed to address key pain points in data ingestion, including observability, error handling, state management, data quality, type conversions, schema drift, and more. By providing a simpler and more cost-effective solution, Dagster Embedded ELT aims to make data engineering easier and more accessible.

Dagster University is a new resource introduced by the Dagster team to empower users with in-depth, self-guided learning. The first course, Dagster Essentials, provides a solid foundation for beginners and covers essential concepts to building in Dagster. The course includes detailed explanations, quizzes, and practice problems, and upon completion, users receive a certificate of completion. Additionally, the Dagster Open Platform is now available as an open-source project, providing guidance and references for users months into using Dagster. This platform applies best practices in data engineering and Dagster to build complex asset-based pipelines. The Dagster team also offers other resources, including Python for beginners, a data engineering glossary, and access to the Dagster documentation.

Dagster Insights is a new feature announced by Dagster, a modern data orchestration platform. It aims to improve visibility into data operations and cost management for teams. The feature introduces a metrics store and API, tooling for capturing operational metadata, and visualizations within the UI to help teams understand their data platform's performance. With Dagster Insights, teams can retrieve operational metadata from pipelines and downstream systems, attribute it to data processes inside the orchestrator, and aggregate operational metadata in the orchestrator. The feature is initially available to Cloud customers only and will be expanded to include additional data sources and capabilities in the coming weeks. Dagster Insights is an experimental feature that can be enabled on request, and teams are encouraged to provide feedback and suggestions for improvement.

The text discusses the importance of data quality checks in ensuring that data is delivered high-quality and meets the organization's standards. It introduces Dagster Asset Checks, a new feature in Dagster 1.5, which allows users to define data quality checks for their assets. These checks can be executed inline with the pipeline or on their own, providing flexibility and control over when and how they are run. The feature aims to provide a unified view of pipeline health, enabling users to understand what data is expected to exist, how it's partitioned, and its freshness. Additionally, Asset Checks facilitate anomaly detection by comparing current values to historical trends, allowing for more accurate alerting and investigation. The text highlights the benefits of integrating data quality checks with orchestration, providing a single system that can execute and track updates to data, understand what data should look like, and verify its quality.

The podcast episode features Nick Schrock, Founder of Dagster Labs and Creator of the open-source orchestration platform Dagster. He discusses the complexity and available tools in the data infrastructure space, leading him to found Dagster Labs. The conversation also touches on the challenges and fragmentation in the data engineering industry, the need for better abstraction layers, and the importance of data orchestration in the future of data infrastructure and engineering.

Dagster 1.5 introduces several major enhancements to the platform, including Embed data quality steps in your data pipeline with Dagster Asset Checks, Expand Dagster's span-of-control with Dagster Pipes, and Monitor Cloud costs with Dagster Insights. Additionally, Dagster University has been created to help beginners build a solid foundation for Dagster, while the UI performance improvements provide a smoother experience for users. The 1.5 release also includes contributions from the community, such as improved handling of failed jobs and bug fixes.