January 2022 Summaries

The text explores the creation of dynamic image processing workflows using Conductor, focusing on generating images in multiple formats and sizes through runtime task splitting. Initially, image processing workflows were limited to a fixed number of images due to hardcoded paths, but Conductor's dynamic fork feature now allows for more flexibility by determining the number of parallel processes at runtime. The workflow involves three main tasks: reading input parameters to define dynamic tasks, creating parallel tasks dynamically using the FORK_JOIN_DYNAMIC, and rejoining completed tasks. A detailed explanation is provided on how to configure these tasks using JSON and Java, including the creation of dynamic task names based on output formats and sizes. The process culminates in spawning several image conversion tasks that result in various image formats and sizes, with plans to introduce subworkflows in future posts for additional tasks like uploading images to Amazon S3.

The blog post discusses the implementation of a Netflix Conductor workflow designed to handle image processing by using a FORK operation to generate multiple image versions in parallel. The workflow takes an image as input and creates two different formats, jpg and webp, using separate parallel processes that run asynchronously to ensure efficiency. This demonstration illustrates the use of the FORK task to split workflows into multiple streams, allowing simultaneous execution of tasks before synchronizing the results with a JOIN task. The necessary tasks such as image conversion, resizing, and uploading to AWS S3 are defined and managed within the Conductor environment. The workflow and tasks, along with sample code, are available in the orkesworkers GitHub repository, enabling users to replicate and test the process locally. The post highlights the flexibility and speed of parallel processing in workflows and concludes with a teaser for an upcoming post that will explore using subworkflows instead of parallel tasks.

Conductor is a microservices orchestration platform developed by Netflix and released under the Apache 2.0 Open Source License, designed to manage the complexities and failures inherent in large, interconnected cloud systems. It enables users to orchestrate workflows composed of numerous tasks, where each task functions as a stateless worker that processes inputs and outputs, often running outside the Conductor server. Conductor is built with the understanding that systems can degrade or fail, and it allows for the creation of stateful applications that can manage such issues without requiring additional coding. Users can configure tasks to respond to availability events like failures, timeouts, and rate limits, with parameters available to dictate retry logic and timeout policies, ensuring that tasks can handle execution failures through mechanisms like fixed delays or exponential backoff.

This text outlines a method for automating image processing using Conductor to orchestrate microservices, which allows for resizing images and saving them to an Amazon S3 bucket. The workflow consists of two main tasks: the first involves resizing an image using ImageMagick and saving it locally, while the second uploads the resized image to an S3 bucket and outputs the image URL. The setup requires running Conductor locally with prerequisites such as Java JDK and NPM Yarn, cloning the Conductor repository, and configuring AWS for storage. The process exemplifies the flexibility of a microservice architecture, enabling easy swapping or extension of components, and is tested through an API call that modifies an example image. The overall workflow is managed through Conductor, with tasks executed by Java applications that interact with ImageMagick and AWS, and results are accessible through a local server interface.