We Let an LLM Open Pull Requests Against Your .mergify.yml. The Prompt Was the Easy Part.

Mergify has introduced a new "Fix with Mergify" feature that employs a language learning model (LLM) to automatically correct errors in the .mergify.yml configuration files, opening pull requests with the suggested fixes. The real innovation lies not in the AI itself but in the robust system of checks and balances surrounding it, including schema validation, access controls, and usage quotas, which ensure that any changes proposed by the AI are safe and controlled. This system is designed to prevent potential damage by validating outputs through a strict schema and involving human oversight before any changes are merged. The engineering focus is on creating a secure framework that constrains the LLM's output, rather than the model's internal processes, ensuring that any proposed changes are valid and reviewed by a human. While the model can be easily swapped without affecting these safeguards, the emphasis is on maintaining security and correctness through thorough validation and controlled access, making the AI's role more about enhancing efficiency than compromising safety.