January 2026 Summaries

Iterating on skills for agents like Codex poses challenges in determining whether changes improve performance or merely alter behavior, with regressions such as skipped steps or leftover files complicating the process. A skill, comprising prompts and instructions for a large language model (LLM), is best improved by evaluating it like any LLM prompt through "evals," which assess if the model's outputs and process align with intended goals. This involves defining measurable success criteria, such as outcome, process, style, and efficiency goals, and using a pattern akin to lightweight end-to-end tests to validate performance. The post illustrates this with a skill for setting up a React demo app, emphasizing the importance of clear naming and descriptions in the SKILL.md file for reliable skill invocation. Manual testing reveals hidden assumptions, and a small, targeted prompt set helps catch regressions early. Evals leverage both deterministic checks and qualitative, rubric-based grading to ensure skills meet both functional and stylistic expectations. As skills mature, evals can be expanded with additional checks to maintain and improve performance consistency, shifting from intuitive improvements to evidence-based validations.

Skyscanner enhanced OpenAI's Codex CLI by integrating it with JetBrains IDEs using the Model Context Protocol (MCP) server, significantly boosting its effectiveness as an AI pair programmer. This integration allows Codex to access the rich context of the development environment, enabling it to identify errors and run configurations like human developers, which accelerates development without compromising quality. For instance, Codex can immediately detect compilation issues, reducing iteration time and enhancing problem-solving capabilities. The integration also streamlines operations by allowing Codex to autonomously execute predefined testing and formatting tasks, thereby minimizing manual intervention and increasing efficiency. Ultimately, this approach has transformed Codex from a standalone tool into a seamlessly integrated component of Skyscanner's development ecosystem, aligning more closely with existing workflows and improving trust in its output.