July 2022 Summaries

In July 2022, the Verify Stage product and engineering teams at GitLab conducted a "Hackathon Week" to inspire innovation and address various challenges. The event began with brainstorming sessions to generate ideas for potential solutions and enhancements, followed by collaborative efforts to refine these concepts. Critical projects saw engineers take the lead in organizing tasks and ensuring designs were ready to facilitate successful outcomes. The team prioritized codebase improvements by deleting redundant code, offering a prize for the most deletions, and delivered several Proof of Concepts (POCs) that led to new features or enhancements, such as easier navigation of job logs, improved job performance data display, and streamlined debugging processes. The Hackathon also focused on future initiatives like fault-tolerant runners and CI workflows, while emphasizing the importance of reducing technical debt through code cleanup. This collaborative effort resulted in over 1,300 lines of code and 4,421 MB of database data being removed, illustrating the team's commitment to efficiency and innovation. The event concluded with an invitation for contributions to ongoing and future projects, fostering a culture of continuous improvement and community involvement.

R2Devops, in partnership with GitLab, aims to simplify the CI/CD onboarding process by creating a collaborative hub of open-source CI/CD jobs, allowing developers to easily add and use jobs from a shared library, thus saving time and effort in pipeline creation. By leveraging GitLab's features, such as the Include keyword and CI/CD variables, R2Devops enables developers to customize and integrate jobs seamlessly into their projects without rewriting code from scratch. This initiative aligns with both organizations' commitment to open source and transparency, promoting collaboration and continuous improvement in development processes. Through R2Devops, developers can contribute their own jobs to the library, enhancing the community-driven aspect of CI/CD development, and can engage with the community via Discord to provide feedback and participate in the evolution of this open-source solution.

Johan Carlsson, a part-time hacker from Gothenburg, Sweden, quickly climbed to the number seven spot on HackerOne's Top 10 list, despite balancing a full-time web development job and family commitments. With a background in computer science and fine arts, Johan ventured into IT security and bug bounties through a course in ethical hacking during his final university semester. He emphasizes persistence, genuine interest in IT security, and adopting the mindset of an attacker as keys to his success. Johan's approach to bug hunting includes studying GitLab's release notes, revisiting past bugs, and engaging with the GitLab issue tracker to uncover overlooked vulnerabilities. He advocates for a more personalized triage experience in bug bounty programs to enhance motivation and engagement. GitLab's program, while efficient and standardized, aims to balance efficiency with collaboration, transparency, and inclusivity. Johan's journey and insights offer a roadmap for newcomers to the bug hunting field, highlighting the importance of curiosity and continuous learning.

GitLab's user experience research highlights the importance of addressing workflow disruptions caused by bugs, slow performance, and bad design within its complex product. The research involves engaging with users to identify issues, particularly in branch and merge request operations, and transforming these findings into potential contributions for the community. GitLab encourages participation in solving identified problems, offering guidance through development resources and support channels. The benefits of contributing to open-source projects are underscored, emphasizing learning, public recognition, and career advancement opportunities. Contributors are rewarded with GitLab swag and can aspire to join the GitLab Core team, experiencing personal and professional growth while enhancing their understanding of the platform.

GitLab can be easily installed on most Linux distributions, but accessing it from outside a private network typically requires router configuration, which poses security risks and challenges, especially if the ISP does not provide a static IP address. Tailscale, leveraging the WireGuard protocol, simplifies setting up a VPN by providing a user-friendly client for various platforms, allowing seamless and secure remote access to services like GitLab without complex router configurations. By using Tailscale, one can assign a static IP within the network, set up DNS records, and ensure secure access through HTTPS encryption using Let's Encrypt certificates. This approach enables running GitLab in a home network while maintaining secure and hassle-free remote connectivity.

Hiring and retaining software engineers is a challenging task for companies, as they often switch jobs within two years. GitLab addresses this issue by focusing on an effective onboarding process to reduce the time engineers spend in the adaptation phase, aiming for them to ship production code within a week and be fully onboarded in under three months. This strategy helps in quicker integration and reduces turnover by ensuring a positive onboarding experience. GitLab's process begins before the new hire's first day, involving the creation of an onboarding issue that serves as a central communication tool, assigning an onboarding buddy to guide the new engineer, and encouraging practices such as pairing and regular coffee chats to build relationships and ease the transition. Tailoring the onboarding experience to the individual and establishing feedback loops are crucial to their success, with the ultimate goal being to integrate the engineer efficiently and enable them to contribute effectively to the team.

Product managers at GitLab engage in three primary types of user calls—discovery, roadmap discussions, and solution validation—to better understand user needs and improve product offerings. Discovery calls focus on understanding user pain points and require a structured yet flexible approach, often involving scripts and targeting both active GitLab users and potential users from relevant communities. Roadmap discussions, initiated by sales or account teams, aim to strengthen customer relationships by transparently discussing product development plans, though they offer limited insights into user needs. Solution validation calls, which are more focused than discovery calls, involve testing specific prototypes with users to gather feedback, requiring minimal guidance from product managers to accurately assess user interaction with the product. Regular engagement with users through these calls, alongside asynchronous feedback, is crucial for continuous learning and refining product strategies.

GitLab.com has implemented a system to monitor its services using Service Level Indicators (SLIs) and Service Level Objectives (SLOs) to promptly address issues before they impact users. However, the previous setup led to excessive paging of the Site Reliability Engineering (SRE) team during service-wide or site-wide outages, causing stress and distraction. To address this, GitLab introduced a more efficient alert management system using Prometheus and Alertmanager, which groups alerts by service and incorporates service dependencies. This change reduces the number of pages the SRE team receives, allowing them to focus on resolving issues more effectively. The system now ensures that if a foundational service like the database degrades, it prevents cascading alerts for dependent services, thus improving the quality of life for on-call engineers by consolidating alerts and decreasing unnecessary notifications. This approach has already resulted in a significant reduction in the number of pages received during outages.

The text examines the concept of "Worse is Better," a software development philosophy suggesting that simpler and initially less optimal solutions can ultimately outperform complex designs by allowing for quicker feedback and iteration. GitLab's success is attributed to this approach, favoring "boring technology" that prioritizes simplicity and adaptability over rigid, complex structures. This philosophy contrasts with the challenges of developing plugin interfaces, which can slow innovation due to their complexity and need for careful design. The open core model, despite lacking defined boundaries, benefits end users by allowing immediate customization and integration, fostering a cycle where rapid adaptation enhances functionality and user satisfaction. This approach, exemplified by GitLab's use of Ruby on Rails, balances flexibility with solid engineering practices, facilitating an environment where good design arises organically and boundaries, though present, are adaptable.

GitLab's compliance management capabilities aim to seamlessly integrate security and compliance into development and deployment processes from the outset, emphasizing the importance of designing processes to be secure and compliant initially rather than retrofitting them later. The platform offers several built-in features to help organizations meet compliance requirements and enhance security, many of which are available for free. Key features include enabling Multi-Factor Authentication (MFA) to reduce account access risks, reviewing and controlling privileged access to adhere to the principle of least privilege, using role-based access to protect branches, and configuring merge request approval settings to ensure proper change management. Furthermore, GitLab provides audit event configurations to monitor changes within the platform, helping organizations maintain compliance by offering detailed logs and records for auditing purposes. These features collectively assist in reducing breach risks, lowering costs, and ensuring alignment with regulatory requirements.

GitLab, founded a decade ago on the Git version control system, addresses the challenges organizations face with large monolithic repositories, known as monorepos, which can cause slowdowns and complexity. Despite some skepticism, monorepos offer advantages such as reducing team silos, standardizing tooling and processes, simplifying dependency management, and providing unified CI/CD processes. GitLab suggests several strategies to manage monorepos effectively, including using CODEOWNERS for efficient merge request approvals, employing Git LFS for better handling of large files, implementing partial clone operations to reduce download times, utilizing parent-child pipelines to streamline processes, and conducting incremental backups to minimize downtime. While these tips have facilitated transitions to GitLab, the platform continues to seek improvements and user feedback to enhance support for large repositories and expand features like SAST scanner support and namespace reporting for monorepos.

The blog post provides a detailed walkthrough of migrating from Atlassian's Bamboo Server to GitLab CI/CD, focusing on user experience and the practical steps involved. It illustrates this process through a fictional scenario involving a software engineer, John Doe, and his team as they manage components of a larger product, using GitLab to test, release, and deploy applications in various preview environments. The narrative describes how infrastructure as code (IaC) templates are managed and updated, emphasizing GitLab's capabilities and the GitOps approach, which leverages Git for tracking changes and facilitating user self-service. Despite its advantages, such as smooth migration and reduced dependency on Ops teams, the solution also has some limitations, including the lack of a centralized dashboard for managing environments, which would require additional development.

Software is integral to modern life, and as organizations strive to deliver it faster for enhanced customer experiences, maintaining security becomes crucial. This has led to the adoption of the National Institute of Standards and Technology (NIST) Risk Management Framework (RMF) by U.S. federal operations to ensure security vulnerabilities are managed as risks, although achieving full compliance is challenging. The process involves continuous monitoring and documentation through the Plan of Actions and Milestones (POA&M), which can be burdensome. GitLab offers solutions by providing security and compliance scanners that evaluate source code, finding vulnerabilities in new and existing code, container images, and third-party licenses. It aggregates findings, allowing security teams to manage and triage vulnerabilities efficiently. With features like one-click issue conversion, Epic management, labels, and Issue Boards, GitLab enhances transparency and efficiency in vulnerability management. This unified system facilitates continuous monitoring and mitigation, offering valuable insights for ongoing improvement and supporting compliance with NIST standards.

The text outlines a detailed process of migrating a CI/CD structure from Atlassian Bamboo Server to GitLab CI/CD, specifically for a multi-component application. The current setup on Bamboo Server involves a sophisticated orchestration system for over 50 components, each stored in separate Git repositories, with distinct build and deployment pipelines. The migration task aims to maintain the existing deployment scheme while utilizing GitLab's features, ensuring self-service management of environments by team members and consistency in Infrastructure as Code (IaC) across different environments. The text describes the structure of the GitLab setup, including groups and projects for applications, shared CI configurations, and infrastructure management using Terraform. The demo repository provides Terraform files to replicate this setup in GitLab, allowing users to observe the process in action. The document also highlights the importance of maintaining security and monitoring aspects, even though they are omitted from the demo for simplicity. The second part of the series promises to delve deeper into the practical implementation of these concepts using GitLab CI/CD.

David Heinemeier Hansson developed Ruby on Rails by combining the approachability of PHP with the structural integrity of Java, creating a framework that is both user-friendly and well-structured. This innovative approach parallels the evolution of operating systems, where usability and stability were traditionally seen as mutually exclusive until NeXT combined them, proving that these qualities can coexist. GitLab co-founder Dmitriy Zaporozhets embraced Rails for its balance of accessibility and architecture, allowing GitLab to develop a modular monolith—a single, cohesive program that avoids the pitfalls of microservices, such as complexity and performance issues. This structure supports GitLab's Open Core business model by fostering contributions from a broad community while maintaining a clear division between open and closed components. Rails' architecture enables GitLab to remain adaptable without overcommitting to rigid, future-proofing strategies, thereby enhancing innovation and scalability.

Software testing, a crucial yet often challenging step in the software development lifecycle, is undergoing a transformation with the rise of test automation and new technologies like artificial intelligence and machine learning. Despite its importance, testing has historically been a bottleneck in development, leading to significant financial losses, as highlighted by a $1.6 trillion cost from software failures in the U.S. in 2019 alone. To improve testing efficiency, DevOps teams are encouraged to integrate manual testing principles into automated testing, focus on business goals, and embrace a data-driven approach. Certification programs and training can further enhance skills, while a deep understanding of AI/ML can optimize test automation processes. By prioritizing relevant metrics, such as pipeline stability and bug resolution time, organizations can better manage the increased data from automation and enhance their testing capabilities, offering testers a chance to elevate their roles within DevOps.