September 2023 Summaries

Buildkite has transitioned to using UUIDv7 as the primary key for all new tables, moving away from the dual-key system of sequential integer IDs and random UUIDs. This change addresses the performance issues associated with non-time-ordered UUIDs and the complications of ensuring uniqueness with sequential integer IDs in distributed databases. UUIDv7 offers time-ordered identifiers that maintain the standard UUID format, allowing for improved database index locality and reduced write and read performance overhead. The decision to adopt UUIDv7 was influenced by its compatibility with existing systems and its potential to become a future standard, providing a streamlined identifier solution that eliminates the need for coordinated key generation and simplifies application logic. While Buildkite evaluated other implementations, such as Instagram's ShardingID and Shopify's composite primary key, the stability and standardization potential of UUIDv7 ultimately led to its adoption, despite the slight storage overhead compared to other solutions.

Buildkite has adopted the UUIDv7 standard as their primary key of choice, offering efficient indexing and external use. The company previously used sequential primary keys for indexing and UUID secondary keys for external use. However, the upcoming UUIDv7 standard provides time-ordered primary keys that can be utilized for both purposes. UUIDs are unique identifiers generated independently without coordination with other parties, making them preferable in sharded database environments. The adoption of UUIDv7 as the primary key eliminates the need for coordinated identifier generation and simplifies application logic.

Continuous Integration and Continuous Deployment (CI/CD) has revolutionized the software development process, enabling teams to deliver high-quality software rapidly. However, as organizations accelerate their software delivery capabilities, they must adapt how they manage security within their software ecosystems. This involves integrating security measures into CI/CD pipelines and ensuring continuous compliance and governance throughout the software lifecycle. Security, compliance, and governance can work in harmony with continuous integration and deployment by designing accountability into the architecture of the CI/CD pipeline, incorporating secure boundaries for CI/CD workflows and pipelines, and implementing standardized security practices and tools. Engineering leaders and security teams must collaborate to decide on acceptable risk tolerance levels and define secure policies and practices to be integrated as standard across all project development lifecycles. Challenges in implementing compliance and governance in CI/CD include speed vs. security, fragmented tooling, human error, open-source contributions, and managing roles and responsibilities. To address these challenges, organizations should use version-controlled code-based configuration, generate and store immutable artifacts, audit trails, metadata, and logs, create secure boundaries for CI/CD workflows and pipelines, and design human systems and processes for fixing issues and vulnerabilities. In conclusion, striking the right balance between efficiency and security is crucial in today's fast-paced software development landscape. By integrating continuous compliance and governance measures into delivery and deployment processes, developers can ship their code securely without delay, and organizations can avoid becoming the next big headline due to a security breach.

The adoption of Continuous Integration and Continuous Deployment (CI/CD) has revolutionized software development by facilitating rapid and high-quality software delivery, yet it poses challenges in integrating security, compliance, and governance. As companies strive to balance efficiency with security, the necessity of merging these aspects into the CI/CD pipeline becomes crucial. This integration involves implementing practices such as static application security testing, automated compliance checks, and immutable metadata storage, all while maintaining developer agility and minimizing manual oversight. The evolving cybersecurity landscape, highlighted by initiatives like the Biden-Harris administration's National Cybersecurity Strategy, underscores the need for shared responsibility between software producers and vendors in safeguarding sensitive data. Reports like the DORA 2022 Accelerate State of DevOps emphasize the importance of frameworks such as the Secure Software Development Framework (SSDF) and Supply Chain Levels for Software Artifacts (SLSA) to enhance software supply chain integrity. Overall, fostering collaboration between engineering and security teams is essential to manage risks effectively, ensuring that security measures are both thorough and seamlessly integrated into the development process.

This guide covers best practices for transitioning from local Terraform workflows to shared CI/CD pipelines, including version control, project structure, managing configuration for different environments, managing secrets, state and state locking, handling concurrency, version constraints, code quality enforcement, policies for governance and compliance, pre-preparing the build environment, using a shared plugin cache, approval gates, and managing CI/CD with Terraform. It emphasizes the importance of choosing practices that fit organically into your operations and evolve alongside you as your team's needs and project requirements change.

Engineers often start with Terraform's CLI for managing infrastructure locally, but as projects scale, transitioning to a shared CI/CD pipeline becomes crucial for collaboration, security, and efficiency. This comprehensive guide outlines best practices for managing Terraform workflows in CI/CD environments, emphasizing version control, project structuring, and configuration management for different environments. It highlights the importance of secret management, advocating for the use of centralized secret managers to avoid security risks. The guide also covers state management, advocating for remote state storage and state locking to prevent concurrency issues and data exposure. It stresses the importance of enforcing code quality through formatting, linting, and validation, and discusses the need for governance and compliance through tools like Open Policy Agent. By preparing the build environment in advance and using shared plugin caches, teams can optimize performance and consistency. The guide also touches on the use of approval gates for sensitive operations and advises leveraging Terraform not only for infrastructure but also for managing CI/CD pipelines, ultimately encouraging teams to adapt practices that align with their unique needs and project requirements.

The text discusses the challenges and solutions for utilizing Buildkite, a CI/CD tool, across various cloud platforms, particularly beyond AWS, where it is most seamlessly integrated. The author introduces the agent-stack-k8s, a method to deploy Buildkite agents using Kubernetes (k8s), which allows users to set up autoscaling CI/CD agents on any cloud platform, including Google Cloud Platform (GCP) and Azure, by leveraging Kubernetes clusters. The text guides users on setting up k8s clusters in multiple cloud environments, including AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE), and provides insights into configuring these clusters with the necessary secrets and installing the Buildkite agent-stack-k8s using Helm charts. The author also emphasizes the importance of best practices such as version pinning and the use of specific queue tags for different workloads, while humorously advising readers to clean up their resources post-experimentation to avoid unnecessary cloud expenses.