April 2025 Summaries

In modern data work, the gap between development and production environments presents significant challenges, often causing data pipelines that run smoothly in development to fail in production due to variations in data volumes, schemas, or system dependencies. Continuous Integration and Continuous Delivery (CI/CD) practices address these issues by implementing structured approaches that include automated testing, deployment, and monitoring processes. This enables data engineers to detect issues earlier, deploy with greater confidence, and adapt quickly to changing requirements. CI/CD originated from software development to streamline integration and reduce conflicts between developers' work, and it extends to Continuous Delivery and Continuous Deployment, where code is kept in a deployable state and can be automatically deployed to production. In data engineering, CI/CD must accommodate large volumes of stateful data, changing schemas, and data quality issues, requiring isolated environments for development, testing, and production to prevent test operations from affecting production data. The practice of CI/CD encourages modular pipeline architecture, allowing for scalability, adaptability, and testability, and involves strategies such as "shift left" for early testing during development and "shift right" for operational resilience post-deployment. This approach leads to a continuous feedback loop, improving pipeline reliability and allowing teams to deploy changes with confidence. Implementing CI/CD is a gradual process, starting with basic automation and expanding as teams gain experience, supported by platforms like Jenkins, GitHub Actions, and GitLab CI. The successful adoption of CI/CD depends on cultural shifts within teams towards frequent integration, comprehensive testing, and automation, ultimately resulting in less stress, quicker value delivery, and improved system reliability.

Prefect has introduced new self-serve plans, Starter and Team, to accommodate growing teams transitioning from personal projects to more robust production environments, addressing the gap between their Hobby and Enterprise tiers. The new plans offer flexible compute options, including a fully managed Prefect Serverless platform, allowing teams to choose between managed infrastructure, their own setup, or a combination of both. This flexibility is complemented by seat-based pricing, ensuring predictable costs without concerns over usage fees, which can restrict task management on other platforms. Prefect Serverless provides execution credits used only during runtime, optimizing cost efficiency. The new plans enable teams to scale their workflows seamlessly, with an easy transition path from Hobby to Starter and Team, and ultimately to Pro/Enterprise for advanced security and control. Starting from April 15, these plans are accessible, allowing users to upgrade based on their requirements and scale their operations confidently without altering their workflow processes.

WHOOP is a wearable technology company that provides users with insights into their health by tracking metrics such as sleep, strain, recovery, and stress through a personalized analytics platform. Initially, the company faced challenges with its internally developed data workflow systems, which were difficult to scale and troubleshoot. To address these issues, WHOOP adopted Prefect Cloud for orchestrating data and machine learning workflows, significantly reducing incident counts and improving mean time to recovery. This transition allowed engineers to focus more on innovation rather than orchestration, enhancing workflow observability and expanding responsibility to other teams. With Prefect's seamless integration with AWS and Kubernetes, WHOOP did not need to alter its existing architecture, and the partnership with Prefect has been characterized by clear communication and support. The company continues to expand its data platform capabilities with Prefect, aiming for increased reliability and speed as it scales.

Prefect has announced Prefect Serverless, a fully managed compute platform that allows for infrastructure-free workflow execution within Prefect Cloud, addressing the needs of teams that require quick and reliable workflow management without the complexities of infrastructure setup. This new offering builds on the success of Prefect 3.0, which saw a massive increase in adoption, and complements Prefect's existing hybrid execution model. Prefect Serverless is designed for teams that either lack dedicated platform engineering resources or prefer to focus on workflow creation rather than infrastructure management. It offers features such as immediate deployment, effortless scaling, usage-based pricing, and enterprise-grade security with minimal configuration. Prefect Cloud users can choose between the hybrid model, which offers complete control over compute environments, and the serverless model, or use both depending on their specific needs, thereby providing flexibility and catering to a broad range of organizational requirements.

Gus Buchele's team of software engineers orchestrated a significant transformation of their cloud-native platform, composed of hundreds of microservices, by upgrading from Prefect 1.0 to Prefect 2.0. This transition was driven by the need for a more modular, scalable, and observable workflow orchestration system that could handle dynamic workloads and strict reliability expectations. By deploying their system on Google Kubernetes Engine (GKE) and utilizing Kubernetes workers, they achieved dynamic infrastructure scaling, reduced deployment times from several hours to just 20–30 minutes, and prevented API and database overloads. The new architecture allowed for better concurrency control and debugging capabilities, reducing cascading failures and improving system maintainability. This enhancement enabled the team to confidently tackle large-scale data initiatives and manage complex workflows, setting the stage for future upgrades to Prefect 3.0 for even deeper observability and advanced scaling features.

The tutorial guides users on utilizing the Prefect Terraform Provider to manage Prefect deployments, emphasizing the benefits of using Terraform for scalable and consistent infrastructure management. It provides a detailed walkthrough on defining and configuring the Prefect provider for authentication with Prefect Cloud, setting up work pools for workflow execution, retrieving flow code from sources like GitHub, and scheduling workflows for automatic execution. The tutorial highlights the distinction between Prefect deployments, which represent workflows server-side without storing code, and how Terraform aids in centralizing configuration, ensuring environment parity, and integrating seamlessly with CI pipelines. The approach is particularly advantageous for infrastructure engineers looking to deploy workflows consistently across multiple teams and environments, with Terraform offering a declarative interface to manage Prefect resources like deployments, work pools, and more. Using Terraform in conjunction with Prefect ensures reproducibility, centralized management, and version-controlled configurations, making it easier to scale and maintain consistent workflow deployments.