October 2025 Summaries

Kubernetes has transitioned from a novel technology to a reliable infrastructure tool, and recent updates to the Buildkite Agent Stack for Kubernetes aim to enhance this stability. These updates include a simplified installation process, improved scalability under high loads, better error reporting, and enhanced observability. The system now uses REST APIs for configuration, reducing setup complexity and increasing security. Scalability improvements ensure efficient handling of high job volumes, while error reporting enhancements allow for more precise troubleshooting. Observability is streamlined with a more straightforward integration path, and new configurations offer greater customization options. Future developments promise further improvements in scheduling, job state visibility, and fine-grained configuration management. The transition to version 0.29.2+ is designed to be seamless, emphasizing simplicity and security for users.

On October 20, 2025, a major AWS outage in the us-east-1 region led to significant service disruptions for Buildkite customers, marked by high latency and increased error rates, primarily due to the inability to provision additional server capacity during peak usage hours. While initially customers experienced minimal impact, the situation worsened as traffic increased, resulting in varied impacts on job dispatch and notifications due to the sharded architecture. The worst-affected customers faced delays of over an hour, while REST and GraphQL APIs also suffered increased latency and error rates, affecting users depending on these for job execution. In response, Buildkite paused all deployments, utilized Statuspage for updates, and attempted to mitigate issues by redistributing loads across shards. Despite challenges in communication due to the outage, the on-call team managed to stabilize performance by the evening as AWS began allowing limited compute provisioning, with normal service levels resuming by late evening. The incident highlighted the need for enhanced resilience strategies and alternative communication avenues for handling such region-wide disruptions.

Earlier this year, Buildkite released its MCP server, initially uncertain of its potential use cases, but quickly discovered significant insights through customer engagement. The server, designed to integrate AI agents into CI/CD workflows, presented challenges with large, noisy build logs that overwhelmed AI agents, leading to inefficiencies. In response, Buildkite introduced a fully managed remote MCP server secured with OAuth, eliminating the need for local installations and reducing maintenance overhead. Additionally, they enhanced log fetching by converting logs into Apache Parquet format for efficient querying and caching. New tools, like wait_for_build, reduce unnecessary polling, optimizing network activity and token usage. Alongside these updates, the documentation was improved to support AI-assisted workflows, making it easier for users to incorporate the server into their processes. These advancements aim to streamline user experiences and foster continuous improvement in delivery processes, while encouraging community contributions for further development.

Buildkite has introduced a new feature for Test Engine called workflows, aimed at enhancing the efficiency and management of software testing by automatically detecting and responding to flaky tests. This feature allows developers to configure monitors that identify flaky behavior using heuristics like transition count, passed on retry, and probabilistic flakiness. Automated actions can be triggered to manage these tests, such as labeling, muting, skipping, or sending notifications via Slack or webhooks, as well as creating issues in tools like Linear. Tag filters can be used to specify which executions a monitor evaluates, providing tailored responses for different test scenarios. By addressing common challenges with flaky tests, workflows help teams maintain high confidence in their testing processes, offering a sophisticated and customizable approach that integrates seamlessly with existing tools and practices. Available in public preview, workflows can be utilized by customers on Pro and Enterprise plans, allowing up to three workflows per test suite.