September 2023 Summaries

The write-audit-publish (WAP) design pattern is a software engineering approach used in data pipelines to ensure data quality and reliability. It involves staging, validating, and committing data to production, with a focus on accuracy, consistency, and trustworthiness. The WAP workflow consists of three phases: Write, Audit, and Publish, where data is first written to a staging area, subjected to rigorous quality checks in the audit phase, and only validated data is transitioned to the production environment in the publish phase. This approach addresses the problem of data integrity by providing a structured approach to ensuring data quality and reliability. The WAP pattern has proven integral to improving data quality and can be used in various scenarios, including batch processing and streaming real-time data. By adopting WAP design patterns, data engineers can enhance data integrity, increase data security, improve reliability, and provide operational flexibility and scalability.

Dagster is launching its Launch Week, starting October 9th, to announce new features and capabilities that aim to "escape the Modern Data Trap." The Modern Data Trap refers to the challenges faced by data engineers in managing the complexity of modern data stacks, including assumptions of homogeneity, disconnected tools, high costs, inflexibility, and a lack of software engineering best practices. Dagster believes that its orchestrator should be a true control plane for the Modern Data Stack, integrating deep features directly into the product to empower data engineers. The Launch Week agenda includes various talks and announcements, showcasing new bundled features and integrations aimed at improving developer experience.

Dagster Labs, the company behind the open source orchestration platform Dagster, has raised $50M from investors including Sequoia, Index, and Georgian Partners. The founder of Dagster, Nick Schrock, recently shared his insights on bringing software engineering principles to the data space in an episode of the Open Source Startup Podcast. In this podcast, Schrock discussed what a great developer experience means for data engineers, how to think about launching the cloud version of an open source project, and more.

Pedram Navid, a data engineer, shares his experience with Dagster, a company building a product purpose-built for users. He was drawn to Dagster after hearing Nick Schrock discuss its approach to orchestration on the Data Engineering Podcast. Pedram has been working with Dagster for over a year, integrating it into projects and finding the team's mix of performance and compassion impressive. He considers himself lucky to be part of Dagster Labs, which is pursuing more than just good enough tooling in data engineering. The company values user feedback and encourages people to reach out with questions or comments.

A Dagster-powered spam filter was built using a Bag of Words model, which uses lists of spam and non-spam messages to determine whether a message is spam or not. The model was trained on an SMS spam dataset from Kaggle and used to score new messages against a test dataset. A job was created in Dagster to rebuild the model every month by integrating user messages from the application and comparing it to the previous month's model. The updated model was saved to a file and stored in a database table to keep track of the current model, which is compared to the previous model to promote the new one if it scores better. The API endpoint was created using Flask to expose the spam filter to users, and testing was done using cURL to try different messages to see if they get flagged as spam.

Dagster Labs CEO Pete Hunt recently joined a podcast discussing the beginnings of Dagster and its journey so far. He was also interviewed on another podcast, where he shared his blueprint for engineering excellence. Additionally, Dagster has open roles available for those interested in working with them, and users can provide feedback through various channels such as the Dagster community Slack or GitHub discussions. Sandy Ryza, a data scientist, discussed the impact of data scientists on the creation of next-generation data orchestration tools in another podcast. Dagster Labs is also involved in other podcasts, including Tech Talks Daily and A Geek Leader, where they share insights on open-source, machine learning, and the future of Dagster.

Data orchestration is a centralized utility that controls the execution and sequencing of interdependent operations, offering a single location for managing visibility and error handling. Data platform engineers can utilize data orchestration to manage complexity by integrating multiple technologies in their systems. The creator of Dagster, Nick Schrock, shares his perspective on the state of data orchestration technology and its application in various environments. He discusses the multi-faceted nature of data in an organization, highlighting the need for capabilities and interfaces tailored to that specific context. This centralized utility helps inform its implementation by providing a single location for managing complexity and visibility.

Factory patterns are a valuable tool for data engineers, especially when dealing with different types of data sources or complex object creation. They simplify code and make it more reusable, scalable, and maintainable. Design patterns in Python work as templates that can be applied to recurring tasks or problems, providing structured and efficient solutions to common challenges in data processing and integration tasks. By using design patterns like the factory pattern, data engineers can prevent specific problems as their project scales, promote centralizing configuration, standardize testing, and allow flexibility while adhering to consistency. Factory patterns are categorized as creational because they create objects, returning different objects based on certain conditional statements or parameters. They separate object creation from main application logic, making code cleaner and easier to understand. In Python, implementing factory patterns is streamlined due to its dynamic typing and first-class functions. The factory pattern is particularly useful in web scraping when handling multiple file formats and can be used to streamline data extraction from APIs. It allows for declarative configuration, reusability, and consistency in producing assets. By mastering factory patterns in Python, data engineers can simplify their code and make it more maintainable, ensuring that data is managed and optimized for best results.