August 2016 Summaries

On August 22, 2016, Buildkite experienced a severe outage due to a series of technical and operational mishaps, including misconfigured PagerDuty settings, database performance issues following a rushed AWS infrastructure downgrade, and AWS IAM issues. The outage was compounded by a failure in health checks and replacement server deployments, which were based on outdated AMIs, leading to a cycle of server failures. The team acknowledged their oversight in not conducting proper load testing and monitoring of AWS credits, resulting in hasty decisions that impacted service availability. Despite the challenges, Buildkite expressed gratitude for customer understanding and committed to learning from the incident to prevent future occurrences, emphasizing the importance of improved technical strategies and team coordination.

GraphQL is a versatile standard for defining, querying, and documenting APIs in an intuitive manner, offering a user-friendly query language and integrated documentation. The guide outlines how Buildkite utilizes GraphQL for front-end development, demonstrating how users can create scheduled builds, a feature not yet available via the web UI, using GraphQL queries and mutations. It introduces GraphiQL, an open-source GraphQL explorer that facilitates testing and exploring APIs with features like auto-complete and inline documentation. Users can execute queries to retrieve user information or build details, and leverage variables for more dynamic and reusable queries. The text also explains the implementation of GraphQL mutations to automate tasks like creating build schedules with required fields such as pipelineID, label, and cronline specified in Cron syntax. Examples include running queries and mutations via cURL, emphasizing GraphQL's simplicity in using HTTP requests to send and retrieve JSON data. The document concludes with additional resources for learning more about GraphQL.