Overriding a Kestra Plugin for Custom Requirements: Why & How?

Kestra offers an extensible plugin architecture that allows users to build custom plugins to address specific business needs, as exemplified by a company that integrated Soda tests with its internal data quality platform. The company developed a customized plugin to aggregate test metrics from Soda and communicate with external services through Kafka, as well as interact with internal APIs via HTTP requests. Initially, they used YAML to construct a Kestra Flow to execute Soda tests and send metrics to Kafka topics, but opted to convert the flow to Java for better handling of complex logic and API interactions. This approach highlights the flexibility and maintainability of building components in Kestra, whether through YAML, Java, or Python, enabling users to create modular and reusable components tailored to their workflows.