August 2022 Summaries

DevOps teams possess inherent advantages when undergoing audits due to their seamless processes and automation capabilities, which streamline compliance verification. Audits, whether internal or external, ensure that organizations meet various standards, from government regulations to internal governance rules, necessitating meticulous tracking of data and processes. Historically, audits were labor-intensive, but technological advancements have simplified preparations and execution, although increasing security concerns have led to more frequent audits. DevOps facilitates the audit process by maintaining detailed, auditable records of every step in software development, including code commits and reviews, which are critical for compliance across industries with varying regulations. The continuous nature of DevOps aligns with the need for continuous audits, as automation helps track each step and verify the integrity of the code, thereby providing auditors with a clear and accessible trail of development activities.

In the 2022 Global DevSecOps Survey, over 5,000 practitioners highlighted a significant shift in DevOps roles towards an integrated DevSecOps approach, with developers, security teams, and operations professionals increasingly sharing responsibilities. Developers are now deeply involved in operational tasks, with 38% responsible for production monitoring and 53% taking on security responsibilities within their organizations. Security teams are becoming more collaborative, with 29% involved in cross-functional teams and 35% being more hands-on with DevOps projects. Operations professionals, while maintaining their focus on cloud and infrastructure management, are also expanding their roles to include DevOps coaching, automation, compliance oversight, and platform engineering. This reshaping of roles, driven by a need for tighter team integration, reflects a broader trend towards collaborative and versatile DevOps practices, even amidst technical and macroeconomic challenges.

The 2022 Global DevSecOps Survey, involving over 5,000 DevOps practitioners, reveals a shift in the perceived reasons for software release delays, with respondents identifying five key factors: code development, code review, security analysis, test data management, and testing, diverging from previous years when testing was primarily blamed. Code reviews remain challenging due to their labor-intensive nature and cultural issues, although they are highly valued, with artificial intelligence increasingly being used to facilitate the process. Security has emerged as a top investment priority, with DevOps teams intensifying efforts in security testing, despite ongoing frustrations with developers identifying bugs too late. Test data management has become more complex due to increased automation and testing, yet many teams lack systems to effectively process and utilize test data. While software testing continues to be a critical yet elusive component, the adoption of AI and machine learning is growing, suggesting potential improvements in testing efficiency and effectiveness.

Small and medium-sized businesses (SMBs) can significantly enhance their survival and growth prospects by adopting an end-to-end DevOps platform, which streamlines operations and mitigates common challenges such as limited resources and high failure rates. Such a platform automates repetitive tasks, integrates security from the outset, and facilitates collaboration across departments, thereby maximizing the efficiency of limited IT staff. Unlike managing a complex and costly toolchain, a single DevOps application reduces expenses related to licenses and maintenance, while boosting development efficiency by over 87%, according to a Forrester Consulting study. By embedding security throughout the software lifecycle, businesses not only safeguard their software but also enhance customer trust and brand reputation. Consequently, as SMBs face stiff competition and resource constraints, simplifying their toolsets with a comprehensive DevOps platform helps them operate more effectively and compete against larger enterprises.

Threats to the software supply chain are prompting a significant shift in DevOps, with organizations under pressure to integrate security deeply into their software development life cycles and comply with various federal and industry mandates. This has led to the emergence of a DevSecOps strategy, which governs the protection of code, applications, and infrastructure across the software supply chain while emphasizing automation to minimize human error and malicious activity. The concept of shifting security left in development has gained traction, with security becoming an integral part of DevOps culture. The Biden administration has heightened the urgency for improving software supply chain security, influencing standards bodies and prompting organizations to align their practices with mandates from entities like NIST and SLSA. A DevOps platform can aid in this by providing end-to-end visibility, consistent policy application, and a reduced attack surface through a simplified toolchain. GitLab, a leader in DevSecOps, automates security testing within the CI pipeline and partners with other entities to enhance security measures, such as generating SBOMs and protecting against malicious modules, thereby enabling organizations to balance speed and risk management effectively.

The text discusses the complexities and considerations involved in migrating to a DevOps platform, emphasizing the benefits of consolidating toolchains to reduce the time and resources spent on tool management. It highlights the findings from a 2022 Global DevSecOps Survey, which revealed that a significant portion of developers' time is devoted to toolchain maintenance, leading to a strong desire for consolidation. The text outlines a three-step plan for migration, which involves identifying strategic goals, evaluating existing tools, and considering the potential business disruption caused by replacing existing tools. It stresses that migration processes should be tailored to a company's specific needs and provides guidance on evaluating timelines, costs, team support, data handling, and automation integration. Additionally, it recommends documenting the migration process, addressing security considerations, and defining goals for a successful migration to ensure teams are well-prepared and aligned with the new system.

Many companies are moving towards DevOps to enhance software development efficiency and security, but often rely on a fragmented collection of tools rather than a unified platform, leading to complexity and inefficiencies. This DIY approach results in a tangled web of tools that complicates the workflow, creates silos, and incurs what is termed a "DevOps tax," where significant time is wasted on integrating and maintaining toolchains instead of focusing on core development tasks. An end-to-end DevOps platform offers a solution by eliminating the need for multiple tools, thus simplifying processes, enhancing collaboration, and integrating security into every development stage. Such platforms not only streamline the entire software development lifecycle—from planning to deployment and monitoring—but also foster transparency and shared responsibility among team members. This shift is expected to provide a rapid return on investment by reducing costs associated with maintaining a complex toolchain and improving overall project visibility and efficiency.

GitLab has launched a Capture the Flag (CTF) challenge aimed at identifying permission-related vulnerabilities with high business impact but lower CVSS scores, offering a $20,000 USD bounty to the first person who successfully captures a flag by bypassing access control without user interaction. The challenge involves finding a flag within a private project on GitLab.com, and participants must submit a detailed report of their findings through HackerOne to claim the reward. The initiative is designed to enhance security by exposing real-world vulnerabilities and is distinct from other CTFs as it lacks a known solution and is ongoing, with the program being updated on HackerOne once the flag is captured. The use of leaked administrator-privileged tokens is excluded from the CTF but may still qualify for the maximum bounty payout under GitLab's regular bug bounty program. Participants are encouraged to stay informed about the challenge status and CTF availability by subscribing to updates on HackerOne.

APIs encompass more than just interfaces, incorporating various development lifecycle elements like source code, definitions, tests, and feedback, all crucial for successful implementation. In collaboration with GitLab, Postman introduced a git integration allowing users to connect APIs in Postman to their GitLab cloud repositories, enhancing seamless collaboration for API teams. This integration brings version control to Postman, synchronizing branches and versions between GitLab and Postman, and enables elements like schema and collections to coexist with source code, thereby facilitating cohesive workflows. It also redefines the perception of APIs as a comprehensive entity rather than disparate components, thus providing a holistic view critical for robust API programs. Additionally, the integration reduces the risk of using outdated APIs by ensuring users work with current versions, supported by real-time collaborative tools for maintaining synchronization. This partnership underscores Postman's commitment to serving API-first organizations by improving productivity, product quality, and ecosystem strength.

The 2022 Global DevSecOps Survey highlights a significant shift towards prioritizing security within DevOps practices, as evidenced by over 5,000 practitioners emphasizing security as the main reason for implementing DevOps platforms and the top investment priority for the year. This change marks a departure from past tendencies where security was often sidelined for faster software releases. The survey reveals that 53% of developers feel fully responsible for security, an increase from previous years, and that nearly half of operations professionals also assume full security responsibility. Despite these advances, challenges remain, such as difficulties in security testing accessibility and the persistent gap between security and development teams regarding bug identification and prioritization. While security professionals rate their current organizational security posture positively, a notable portion feels unprepared for future challenges, with AI emerging as a promising tool to bolster security efforts, overtaking soft skills in importance. The survey also explores other pivotal themes like automation, AI, and compliance, offering a comprehensive view of the evolving DevSecOps landscape.

Amid an increase in credential stuffing attacks, GitLab has enhanced user authentication processes by integrating Arkose Protect, a security solution from Arkose Labs, into its login flow to help mitigate these threats. Although multifactor authentication (MFA) is recommended and reduces the likelihood of such attacks, it is optional for users, prompting GitLab to implement additional protective measures. The integration, which went live on April 29, 2022, involves assessing user sessions with a machine learning model that classifies risk as high, medium, or low. Users deemed low risk proceed without disruption, while those at higher risk complete an enhanced CAPTCHA. These measures have significantly reduced account takeovers, spam, and crypto mining abuses by over 40%. GitLab's anti-abuse team continues to refine its strategies and plans to introduce a holistic user scoring engine to further secure the platform while minimizing inconvenience for legitimate users.

Software testing is identified as the most challenging aspect of DevOps, often causing release delays due to its complexity and the perception that it takes too long. GitLab emphasizes the importance of using the right metrics to evaluate testing processes, suggesting a focus on the age of unresolved bugs rather than just the speed of resolving new ones to avoid incentivizing superficial fixes. Addressing flaky tests and fostering a manual testing mindset that mimics real user behavior are crucial for improving test reliability and developer satisfaction. Automation, particularly with the integration of AI and machine learning, is highlighted as a means to streamline the testing process, reduce delays, and enhance collaboration between QA and development teams. A comprehensive testing strategy that includes accessibility and advanced techniques like fuzzing is essential for a successful DevOps environment, suggesting that modernizing testing approaches can mitigate frustration and improve release efficiency.

Zero trust, first introduced by Forrester Research in 2010, has evolved from being a focus for enterprise security to becoming integral to modern DevOps practices, aligning well with the DevSecOps approach that emphasizes shifting security measures left in the development lifecycle. This framework mandates that all users—human or automated—must be authenticated, authorized, and continuously validated to access resources, a concept that has gained traction and even become a requirement in some contexts, as highlighted by references in U.S. cybersecurity policies and NIST standards. Despite its growing adoption, confusion persists as zero trust is often mischaracterized as a product rather than a strategic approach, with some early supporters failing to adapt to advancements like cloud technology and AI. GitLab has embraced zero trust by integrating its principles into its DevOps platform, offering capabilities such as granular role-based access, continuous authentication, and real-time monitoring to support organizations in implementing comprehensive zero trust strategies. However, the complexity of authentication in automated environments requires sophisticated solutions, such as mutual authentication or automated multifactor authentication tools, to address challenges posed by automation and ensure secure data access.

GitLab's transition to its own SSH daemon (SSHD) stemmed from the need for enhanced functionality and control over SSH connections, which the existing combination of OpenSSH Server and GitLab Shell couldn't fully provide. The move was motivated by a community-contributed lightweight alternative that offered benefits for containerized deployments and better performance management. Key improvements included enabling the PROXY protocol for accurate IP address tracking, ensuring compatibility with Kubernetes for graceful shutdowns, collecting detailed metrics for monitoring, and reducing the attack surface by using a restricted set of SSH features. Despite the benefits, the transition posed significant challenges, including security risks and compatibility issues with existing features, which were mitigated through thorough security reviews and gradual rollouts. The process also required addressing limitations in the golang.org/x/crypto library and implementing OpenSSH options to maintain necessary features. Incremental rollouts and seeking diverse perspectives were crucial in navigating the complexities of this shift and ensuring a successful implementation.

Transitioning to a DevOps platform requires careful preparation to ensure IT teams handle the migration with confidence and efficiency, fundamentally altering their workflow and offering benefits such as reduced tool-management costs, enhanced security, accelerated software creation, and a collaborative environment. Gaining organization-wide buy-in from the C-suite to the workforce is essential, as resistance can stem from anxiety about change. Identifying and empowering champions from within teams to lead the migration can inspire others, while clear communication about roles and expectations can ease tensions. Training and hands-on experience with sample projects prior to migration are crucial for building confidence and reducing fears of failure. The move to a DevOps platform promises improved collaboration, streamlined processes, and enhanced security and compliance, ultimately leading to increased job satisfaction and the ability to focus on more impactful projects.

DevOps teams are increasingly tasked with integrating compliance into their processes to ensure secure software development, aligning with standards such as NIST, ISO, GDPR, and others. This integration involves balancing security and compliance, where compliance sets regulatory requirements and security implements them. Traditionally, compliance was a cumbersome process, but with the rise of cloud-native applications and complex infrastructures, automation and collaboration within DevOps are essential. Automation, particularly in CI/CD pipelines, can streamline compliance by eliminating manual checklists, providing audit trails, and creating "Golden Images" for guidance. Establishing a system of record (SOR) also aids in tracking compliance changes. As software supply chain security becomes a focal point, encompassing security scanning and policy automation, organizations must continuously maintain compliance and security measures. GitLab supports this by offering solutions to achieve continuous compliance and enhance software supply chain security.

GitLab.com's transition to Ubuntu 20.04 for its Postgres database clusters involves a complex process aimed at enhancing performance and reliability, necessitating a planned service downtime. The upgrade from Ubuntu 16.04 is critical due to changes in glibc affecting Postgres indexes, potentially corrupting them, which requires a maintenance window on September 3, 2022, from 11:00 UTC to 14:00 UTC. During this period, GitLab.com will undergo a complete service disruption to ensure data integrity and proper functioning post-upgrade. The strategy involves using Postgres streaming replication to transfer data to new Ubuntu 20.04 clusters and reindexing potentially corrupt indexes, with particular focus on B-Tree, GiST, Hash, and GIN indexes, to minimize downtime and customer impact. Enhancements in performance include a significant improvement in the speed of the amcheck script to detect corrupt B-Tree indexes and advancements in reindexing speed, achieved by parallelizing processes and optimizing Postgres parameters. The upgrade process has been carefully planned, with measures in place to maintain business continuity and address potential index corruption efficiently.

Securing the software supply chain requires continuous improvement and adaptation to evolving security frameworks, with a growing emphasis on attestation and the presentation of software bills of materials (SBOM). Attestation, a verified statement about software artifacts, is integral to compliance standards such as SLSA Level 2, and has gained prominence in response to high-profile security breaches. Automating attestation processes, as demonstrated by GitLab's Release 15.1, can enhance security by reducing manual errors and costs while ensuring that software development and build environments remain uncompromised. The next step in this evolving landscape involves integrating code signing to further bolster confidence in software artifacts and encouraging widespread adoption of attestation practices across open-source communities. This shift towards automated and standardized attestation is a significant move towards fortifying software supply chain security, aligning with initiatives like GitLab Ultimate that offer comprehensive security and compliance features.

GitLab has been exploring virtual reality (VR) as a team-building tool to foster connections among its all-remote workforce, inspired by a CTO's participation in a virtual networking session. An initial trial involved VR enthusiasts using Oculus Quest 2 headsets to test games like Wander, a VR version of Google Street View, and ForeVR Bowl, a virtual bowling game. The experiences highlighted both the potential of VR for interactive and engaging team activities and challenges like user identification, game limitations, and hardware accessibility. While VR offers an innovative way to connect globally dispersed team members, considerations such as the cost and availability of VR headsets in different regions remain crucial for equitable access.

Part 2 of the series on decomposing the GitLab backend database delves into the migration process and its outcomes, detailing the decision to opt for a full downtime approach over a zero downtime one due to the lack of an easy rollback strategy and negligible business requirements for zero downtime. The migration involved pausing GitLab services and blocking user-level traffic to allow for comprehensive testing, resulting in a successful migration completed in 93 minutes. The team conducted several rehearsals in a staging environment to identify and resolve potential issues, which proved critical to the migration's success. Post-migration performance improvements included reduced CPU utilization and vacuuming saturation, increased headroom, and decreased query duration, leading to enhanced database efficiency. The team's meticulous preparation, including a production readiness review, ensured a smooth transition and provided confidence in tackling potential challenges.

In a strategic move to address scalability issues, GitLab successfully transitioned its monolithic Postgres database into two separate entities—Main and CI—over the course of a year, detailed in their blog series "Decomposing the GitLab Backend Database." Initially motivated by the limitations of a single database, the project was initiated by a database sharding team formed in early 2021, which identified CI tables as ideal for separation due to their significant write traffic. The endeavor involved complex technical challenges, including designing mechanisms like "loose foreign keys" to manage cascading deletes and implementing mirroring for certain data dependencies. The process was broken into seven phases, allowing for incremental changes and testing, ultimately ensuring a smooth transition with minimal risk and clear prioritization of tasks across development and infrastructure teams. This phased approach enabled GitLab to confidently reconfigure their system for a dual-database setup, demonstrating a robust framework for large-scale architectural changes in software infrastructure.

Part 3 of a blog series discusses the intricate challenges and unexpected surprises encountered during GitLab's database migration. The complexities of taking GitLab.com offline involved shutting down various systems like Kubernetes pods and cron jobs, which proved to be complex due to GitLab's extensive infrastructure. Despite thorough rehearsals, unforeseen issues such as autovacuum processes on CI tables and unexpected database connections required real-time problem-solving. The migration also faced latency issues due to cascading replication, which increased the load on primary databases and required careful rebalancing of PGBouncer connections to avoid CPU saturation. Incremental adjustments were made to manage connection limits and ensure stability, with detailed analysis guiding each step. Despite these challenges, GitLab successfully achieved its objectives, enhancing scalability without affecting developer productivity, thanks to the abstraction provided by Rails models. The blog emphasizes GitLab's commitment to transparency and invites further engagement from readers to explore the project's detailed documentation.

GitLab Federal, LLC has announced the appointment of Janelle Romano and Patty Molthen to its Federal Advisory Board, highlighting their extensive experience and leadership in government and defense sectors. Romano, currently Vice President of Cyberspace Operations at CACI and former Chief of Critical Networks Defense at the National Security Agency, brings nearly three decades of government experience to the board. She is enthusiastic about contributing to GitLab's mission to deliver a streamlined, open core application that supports rapid innovation and integration in government operations. Molthen, an independent consultant with over 20 years of experience working with the Department of Veterans Affairs and the Department of Defense, owns CM2 Group and has been involved with various organizations in the Washington, D.C. area. She is eager to leverage her expertise in healthcare informatics, IT policy, and contracting support to enhance military healthcare and secure product delivery. These appointments align with GitLab's commitment to diversity and inclusion, joining existing board members to further drive the adoption of innovative technologies in the public sector.

In July 2022, a proof of concept for Continuous Integration (CI) workflows was introduced during the Verify Stage Hackathon, aiming to enhance automation in software development processes by using event-based services to trigger pipelines. This concept allows users, particularly those utilizing GitLab CI, to leverage the flexibility of pipelines beyond traditional use cases, such as automatically running pipelines upon certain events like issue creation or merge request state changes, using definitions in the .gitlab-ci.yml file and existing webhooks. The initiative is part of broader GitLab workflow and automation goals, with future plans to formally instrument CI workflows and incorporate Platform Events or Cloud Events to expand event-triggered automations beyond current webhook capabilities. Feedback is encouraged from users to refine and expand these capabilities.

The introduction of a Pull Request generator for GitLab within the Argo CD project's ApplicationSet enables the dynamic provisioning of review environments directly from merge requests, streamlining the code review process through a GitOps workflow. This enhancement allows code reviewers and designers to efficiently evaluate changes in Kubernetes clusters from within the merge request, eliminating the delays traditionally associated with coordinating multiple team environments and addressing issues post-deployment. The ApplicationSet offers flexibility by enabling unprivileged users to deploy applications, deploying to multiple clusters simultaneously, and managing applications from a single monorepo. The GitLab Pull Request Generator uses the GitLab API to automatically detect new merge requests and create review environments based on specific filters, such as labels and merge request states, using templating options like merge request ID and branch name. This setup involves configuring necessary tools, setting up GitLab and Kubernetes environments, and creating secrets and ApplicationSets in Argo CD. Once configured, new applications based on merge requests can be provisioned and accessed directly, with the environment being automatically cleaned up upon closing the merge request, thus enhancing efficiency and collaboration in GitLab and Argo CD workflows.