October 2023 Summaries

Buildkite Pipelines is a CI/CD tool that is gaining popularity as a preferred alternative to CircleCI, offering a hybrid approach that separates the control plane from compute, enhancing security and flexibility. The Buildkite Migration tool has been developed to facilitate the transition from CircleCI to Buildkite by translating CircleCI workflow definitions into Buildkite pipeline definitions. Key similarities between the two platforms include the use of YAML for configuration, job and step definitions, and support for parallelism and concurrency, while differences lie in terminology and certain functionalities like orbs and plugins. The migration tool simplifies the transition process, allowing users to maintain their existing configurations and adapt them for Buildkite, with the added benefits of unlimited concurrency, agent flexibility, and a strong emphasis on security. The tool is open-source and under active development, with ongoing improvements and support for translating more features planned. Buildkite emphasizes user control over security and private data, addressing growing concerns in the industry regarding data privacy and security, which has been a driving factor for many companies to reconsider their CI/CD requirements and migrate to Buildkite. The platform offers a 30-day free trial for new users to explore its features.

The text discusses three hosting options for CI/CD tools: managed, self-hosted, and hybrid. Managed CI/CD is a cloud-based fully managed platform that abstracts away most infrastructure details. Self-hosted CI/CD requires the user to manage their own control plane and compute, offering more control but also requiring more effort. Hybrid CI/CD combines elements of both managed and self-hosted options, allowing users to run the compute resources while using a SaaS product for management. Each approach has its advantages and disadvantages in terms of security, speed, convenience, scalability, and cost. The text provides a decision tree based on these factors to help users choose the right CI/CD tool for their project.

CI/CD systems are valuable targets for malicious actors due to their access to code, internal systems, and secrets. To address potential security risks, it is crucial to establish boundaries using hardware and software configurations. This involves setting up granular access controls, maintaining open-source projects with care, and considering sensitivity and regulatory requirements when assigning access levels. Buildkite offers several mechanisms for enforcing security boundaries, such as clusters, queues, and tags. By implementing these measures, organizations can create a safe and secure software engineering environment while still benefiting from the advantages of CI/CD systems.

The blog discusses various engineering techniques that can help teams write better software more quickly, using the spirit of the Agile Manifesto as a guide. It covers pair programming, checklists, and standards and templates. Pair programming is about emphasizing individuals and interactions rather than rigidly sticking to methodologies. Checklists ensure compliance and safety in high-pressure fields like Aviation, Health, and Food Safety, and can be useful in software engineering for business-critical functions that would cause more problems if not done correctly. Standardizing your interactions focuses on where humans interact, such as branch names, code review flows, and incoming tasks. Templates reduce cognitive load, encourage good practices, and mean less re-inventing the wheel. The most beneficial parts of agile are being flexible, listening to the people around you, and empathizing with them.

Choosing a CI/CD tool for your project involves understanding the different hosting options—managed, self-hosted, and hybrid—and evaluating them based on factors like security, speed, convenience, scalability, and cost. Managed CI/CD tools, such as CircleCI and GitHub Actions, offer ease of setup and use but may lack performance flexibility, making them suitable for simple projects. Self-hosted tools, like Jenkins and TeamCity, provide strong security and performance control but require significant infrastructure expertise, fitting well with large companies needing strict compliance. The hybrid approach, exemplified by Buildkite and CircleCI self-hosted runners, offers a balance, allowing companies to manage compute resources while utilizing a SaaS control plane for orchestration, making it ideal for complex use cases. The decision depends on the specific needs and resources of your project, with security being a primary consideration when selecting the right CI/CD tool. Many teams start with managed solutions and move towards hybrid setups as their projects grow, emphasizing the importance of aligning your choice with your project's unique requirements.

The text describes an issue with file permissions in a Docker container used for running jobs. A Bash script was initially created to fix the file permissions, but it had security vulnerabilities due to potential manipulation of symlinks by attackers. The solution involved using syscalls like openat2 and fchownat to ensure that the path given to chown is a subpath of a trusted directory while preventing any symlinks. This approach effectively creates a tiny per-open-call chroot jail, ensuring security in file permission management within Docker containers.

In a narrative exploring the evolution of file management and security issues in modern computing, a team grapples with challenges related to file permissions in Docker containers managed by Buildkite agents on AWS EC2 instances. Initially, they employ a Bash script with sudo permissions to correct file ownership issues, but this approach is plagued by inefficiencies and vulnerabilities, including the potential for symlink exploitation. Despite attempts to safeguard the process through checks and potential use of chroot jails, security concerns persist due to TOCTOU (time-of-check to time-of-use) vulnerabilities, highlighting the difficulty of securely managing file permissions with untrusted inputs. Ultimately, they discover a solution in the form of the Linux kernel's openat2 system call, which allows for secure path resolution by ensuring subpaths are free of symlinks, akin to a per-open-call chroot jail, and propose implementing a custom recursive chown using this technique to mitigate the security risks.

This blog discusses best practices for running CI/CD workloads on Kubernetes, focusing on security, observability, frugality, and flexibility concerns. Security measures include implementing role-based access controls and restricting service communication to prevent privilege escalation. Observability is crucial for understanding trends and patterns over time, as well as identifying performance issues. Frugality involves managing costs by separating clusters for build, test, and deploy, using spot instances or different instance sizes, and setting resource limits on CPU and memory. Finally, flexibility allows for rethinking CI/CD workloads and leveraging open-source tools like Kustomize, Helm, and Flagger to customize the cluster and deployment techniques.

Kubernetes is an open source container orchestration engine that automates deployment, scaling, and management of containerized applications. Helm is a package manager for Kubernetes that simplifies the creation, packaging, configuration, and deployment of Kubernetes applications by combining configuration files into a single reusable package. In CI/CD, Kubernetes can be used to manage build agents, rollback mechanisms, and deploying applications in containers. This blog provides step-by-step instructions on setting up an autoscaling CI/CD pipeline using Buildkite Pipelines with Kubernetes and Helm on any cloud platform.

The text explores best practices for deploying Continuous Integration/Continuous Delivery (CI/CD) on Kubernetes, highlighting the benefits of improved scalability, cost efficiency, and reliability. It emphasizes the importance of security by advocating for role-based access controls and audit logging to monitor inter-service communication and prevent privilege escalation. Observability is also crucial, with the need for comprehensive monitoring of Kubernetes clusters to track trends and troubleshoot issues effectively. Frugality is addressed, suggesting the separation of clusters for different stages like build, test, and deploy, and leveraging spot instances to manage costs. Flexibility is a notable advantage of Kubernetes, allowing for more dynamic and parallelized CI/CD workflows, supported by open-source tools such as Kustomize, Helm, and Flagger, which enhance customization and deployment strategies. Overall, the text suggests that by applying these best practices, teams can create a secure, observable, and cost-effective CI/CD environment on Kubernetes.