May 2020 Summaries

Participating in the CEO Shadow Program at GitLab provided invaluable insights and lessons that became crucial during the onset of the COVID-19 pandemic. The author, initially apprehensive about working in an office and being away from their young daughter, found the experience transformative, particularly in embracing iterative processes and effective communication. The pandemic's disruption highlighted the importance of flexibility, leading to a blend of personal and professional adjustments, such as setting family-focused schedules and integrating family into work life. The program's timing was fortuitous, offering skills that enhanced productivity and adaptability amidst chaos. Emphasizing asynchronous work and minimizing unnecessary meetings allowed for a more balanced life. The experience reinforced a sense of community within GitLab, underscoring the importance of mutual support and learning during challenging times, and encouraged participation for its profound impact on personal and professional growth.

The 2022 Global DevSecOps Survey, gathering insights from over 5,000 DevOps professionals, highlights both the benefits and challenges of DevOps practices. While a successful DevOps strategy can accelerate code release, with 83% of 2020 respondents affirming faster deployment, persistent challenges such as testing, planning, and code review cause delays. Testing remains a significant hurdle, with many teams struggling with insufficient testing and tricky automation, although those achieving test automation report clear benefits. Planning issues often stem from hybrid development methodologies, leading to inefficiencies, while code reviews, though critical for quality and security, are often seen as time-consuming due to cumbersome processes. Despite these obstacles, there are emerging solutions, such as test-driven development and AI-powered testing, that offer promising improvements in workflow efficiency.

GitLab CI/CD offers a streamlined process for code changes by utilizing its Web Integrated Development Environment (IDE), which allows users to submit and commit code changes, trigger a continuous integration pipeline, and deploy applications efficiently. Introduced in GitLab 10.7, this Web IDE is designed to facilitate contributions from both developers and non-developers by simplifying the process of editing files and managing code changes directly within the GitLab interface, all without needing to use the command line. The integration of GitLab's Auto DevOps automatically configures the pipeline, which includes stages such as building Docker images, testing, and scanning for vulnerabilities, all of which are documented in merge requests. This process encourages collaboration and enhances code quality by allowing users to visualize changes through Review Apps, analyze test results, and address any detected vulnerabilities before deploying the application to production.

Sean Smith, a senior software engineer at F5 Networks, shared insights on managing GitLab CI and Kubernetes runners at GitLab Commit San Francisco, highlighting the challenges and solutions related to CI job management. F5 Networks, which handles approximately 350,000 to 400,000 CI jobs monthly, faced an incident where a coding error caused exponential job growth, threatening to overwhelm their system. To mitigate such risks, F5 implemented strategies like splitting their GitLab instance across multiple servers and using Kubernetes and Docker runners to manage workloads. Sean emphasized the importance of setting sensible resource limits to prevent excessive consumption by a few users and recommended using monitoring tools like Grafana or Prometheus for resource management. Additionally, he advised employing labels for better workload management in Kubernetes, while cautioning about their limitations. These measures, alongside establishing a robust alert system, helped F5 avoid outages and maintain system stability, illustrating the critical balance between resource constraints and user demands in DevOps practices.

During a two-week period, GitLab's Head of Remote participated as a co-shadow in the CEO Shadow program, attending 110 Zoom calls with GitLab's CEO amidst the global COVID-19 pandemic. This experience highlighted GitLab's unwavering commitment to its values, which remained consistent across various interactions despite external challenges. The program underscored the importance of transparency, democratized communication in video calls, and the benefits of proposing structured ideas over unorganized thoughts. Participants gained valuable insights into the CEO's role, the significance of community contributions, and the efficiency of remote work. The program also fostered personal connections and demonstrated the adaptability of GitLab's operations during a time when remote work was becoming more prevalent. The CEO's passion for the product and his emphasis on community impact were evident throughout the meetings, reinforcing the company's core value that "everyone can contribute." The shadow experience was not only informative but also provided a unique perspective on leadership, responsibility, and the evolving nature of work in a remote environment.

GitLab's Version 12 Year in Review highlights the company's ongoing commitment to speed, agility, and reliability, demonstrated by over 100 consecutive monthly releases, each bringing significant features and improvements. The company has fostered a collaborative environment with almost 200 community contributions per release, totaling 2,158 contributions from versions 12.0 to 12.10, and has expanded its focus to include developers, security, and operations equally in the DevSecOps model. GitLab has introduced enhancements like Requirements Management, Design Management, and a Compliance Dashboard to support a wide range of users, while providing analytics tools like Productivity Analytics and Code Review Analytics to help developers meet business demands efficiently. Security innovations include a comprehensive Security Dashboard and Container Network Security to manage and mitigate risks effectively. To support operational efficiency, GitLab offers tools such as the Environments Dashboard and Merge Trains, along with integrations like HashiCorp Vault for secure key management and an EKS integration for simplified cluster creation. As GitLab continues to evolve, it remains dedicated to transforming creative work from read-only to read-write, asserting its platform as valuable for not only developers and DevOps teams but for a broader audience.

GitLab provides a comprehensive and user-friendly platform for collaboration, with a focus on ensuring security, especially for self-managed installations exposed to the internet. Administrators can enhance security by leveraging numerous features, such as setting strong root passwords, employing environment variables, and enforcing strict access controls through settings like disabling open sign-up and requiring two-factor authentication. Additional measures include configuring SSH access with preferred encryption keys like RSA and ED25519, and implementing passwordless SSH authentication to strengthen security further. GitLab also offers options for restricting user access and defining visibility settings to protect sensitive data, while network performance can be optimized by adjusting user and IP rate limits and managing webhooks responsibly. The platform encourages customization to balance security with business needs, allowing administrators to fine-tune configurations to suit their organizational requirements effectively.

GitLab has been addressing a popular feature request from its community to introduce a dark mode across its user interface, starting with the Web IDE, which is a browser-based code editor that facilitates file changes and merge requests. The initiative aims to enhance accessibility, especially for users with vision impairments such as photophobia, and aligns with GitLab's philosophy of implementing small changes and fast iterations. The design process involves careful consideration of maintaining visual consistency between light and dark modes, particularly regarding syntax highlighting and UI element states. GitLab's remote team, employing tools like Figma for asynchronous collaboration, is working on refining the dark mode by iteratively releasing updates and addressing design challenges such as element contrast, hierarchy, and graphical adjustments. The endeavor has been informed by resources like Apple's Human Interface Guidelines and Material Design's dark theme section, and GitLab encourages community contributions to further enhance the Web IDE's themability and integrate improvements into their Pajamas design system.

The 2022 Global DevSecOps Survey reveals significant insights from over 5,000 DevOps professionals, highlighting the evolving roles within software development as influenced by DevOps practices. The findings indicate that 83% of practitioners release code faster, with a notable 60% deploying code frequently, while developers increasingly take on responsibilities traditionally associated with operations, such as infrastructure management and security, reflecting a "shift left" approach where security and testing are integrated earlier in the development process. This integration is fostering closer collaboration between developers and security teams, with 65% of security professionals acknowledging their involvement in cross-functional teams, although challenges remain in terms of early bug detection and resolution. The survey also identifies testing as a frequent cause of release delays, despite nearly three-quarters of organizations moving testing earlier in the development cycle, with approaches like test-driven development and real-time testing gaining traction. Overall, respondents express a strong sense of preparedness for the future, with ongoing learning and adaptation being key themes, as DevOps continues to transform the landscape of software development and operations.

Meltano is positioning itself as an open-source alternative to proprietary ELT tools, focusing on data integration and transformation pipelines. Unlike existing solutions, Meltano aims to democratize data integration by providing a self-managed, cost-effective option that circumvents the "pay to play" model of current vendors. It is designed to enable companies to handle their data integration needs without relying on expensive proprietary services, thereby leveling the playing field for businesses globally, especially those with budget constraints or data sovereignty concerns. By leveraging the Singer specification for data extraction and loading, Meltano seeks to foster a collaborative community to develop and maintain a comprehensive suite of extractors and loaders, similar to GitLab's approach in the DevOps space. Although the project is still evolving, with a focus on improving ease of use and reliability, Meltano already supports dbt-based transformations and offers a basic web interface for managing data pipelines. The project invites contributions from the community, particularly from those experienced with the Singer ecosystem, to enhance its capabilities and achieve its vision of becoming a viable open-source ELT solution.

Dominic Couture, a senior application security engineer at GitLab, discusses the intricacies of his role, which involves scrutinizing GitLab's code for vulnerabilities, optimizing processes to catch potential security issues, and maintaining secure coding guidelines. He highlights the challenges of monitoring numerous new features and prioritizing the most security-critical ones while acknowledging the occasional oversight of vulnerabilities. Couture emphasizes the importance of automation in enhancing security, as well as the principle of least privilege in protecting APIs. He also addresses misconceptions about the security effectiveness of VPNs, advocating instead for password managers and multi-factor authentication. Transparency at GitLab, while posing potential security risks, ultimately strengthens the software by enabling external researchers to identify vulnerabilities effectively. Outside of work, Couture enjoys outdoor activities, such as skiing and ultramarathon running, and remains engaged in the security community through bug bounty programs and staying informed via Twitter and various security blogs.

Illumina Consulting Group, a Maryland-based data analytics software startup, pivoted from its usual intelligence community projects to develop a symptom tracking tool called DoIhaveit.net in response to COVID-19. The initiative stemmed from the recognition that early detection of the virus was hindered by a lack of primary information on who might be infected. The tool provides users with localized COVID-19 snapshots based on zip codes and collects anonymous health survey data to offer advice from the Centers for Disease Control. Designed to be adaptable, the tool allows public health teams to adjust parameters in response to changing infection rates, with the ultimate goal of preparing healthcare systems for potential surges. DoIhaveit.net is free for municipalities, and its developers aim to expand its use across the United States, emphasizing that the project is a public service without profit motives.

Kushal Koolwal, a senior manager at Arm Inc., highlights the collaboration between GitLab and Arm, announced at Arm TechCon 2019, aimed at enhancing support for Arm architecture within GitLab's CI/CD tool. This partnership is part of Arm's mission to streamline cloud-native development by expanding the software ecosystem and providing comprehensive developer tools. A significant development in this partnership includes the introduction of official support for Arm-based GitLab runners in multiple formats, such as binary, rpm/deb packages, and Docker images. This advancement, coinciding with the AWS Graviton2 launch, enables developers to efficiently work within their custom environments. GitLab's commitment to this partnership is further demonstrated by their involvement in the Arm Neoverse developer program and the release of a demo for AWS Graviton2 M6g Instance deployment, showcasing the integration of Arm architectures into GitLab's DevOps solutions.

GitLab employs distinct approaches to manage software releases for its self-managed users and GitLab.com, focusing on timely updates and security. Self-managed users receive monthly updates on the 22nd, incorporating community-driven features and bug fixes, while GitLab.com benefits from a continuous delivery process called auto-deployments, which ensures rapid updates through automated pipelines. The GitLab Delivery team, responsible for both release types, strives to minimize manual intervention and reduce the mean time to production (MTTP) by automating processes and improving tool efficiency. The release manager plays a crucial role in assessing bug severity and determining the need for patch releases, especially for high-severity issues that significantly impact users. While self-managed releases require more manual actions, GitLab.com updates leverage CI/CD technologies for frequent and automated deployments. The Delivery team aims to enhance release velocity and streamline processes, with goals such as transitioning to a Kubernetes-based infrastructure and reducing MTTP from 50 hours to eight hours. GitLab encourages community contributions and feedback to further improve its delivery practices.

GitLab has introduced a new feature called Directed Acyclic Graph (DAG) that enhances CI/CD pipeline efficiency by allowing jobs to run out of order based on dependencies rather than rigid stage sequences. This feature is available for free on both GitLab.com and self-managed versions, and it significantly reduces pipeline execution time by enabling jobs to commence as soon as their direct dependencies are completed, irrespective of other ongoing jobs in the same stage. The DAG feature is particularly beneficial in scenarios such as monorepo patterns, where different components like iOS, Android, and web apps can be independently built, tested, and deployed, and in pipelines with lengthy tests that benefit from early initiation. DAG's implementation involves using a new keyword, needs, in the .gitlab-ci.yml configuration file to specify job dependencies, allowing for a more flexible and time-efficient pipeline execution.

Collaboration is crucial in DevSecOps, and achieving it requires visibility into both team activities and the processes that generate their work. Toolchain complexity can significantly slow development, as evidenced by a Forrester survey where 45% of IT professionals reported using three or more tools for software delivery, with two-thirds of those using eleven or more tools per toolchain. Managing these tools can be cumbersome, leading to challenges such as brittle pipelines and context switching, which detract focus from core tasks like application building and infrastructure optimization. GitLab addresses these challenges by offering a comprehensive DevOps platform that integrates CI/CD into a single application, eliminating siloes, enhancing visibility, and increasing efficiency. This unified approach allows teams to build, test, and deploy collaboratively without the hindrances of multiple tools, thus streamlining communication and problem-solving across the software development lifecycle.

Autoscaling GitLab Runner offers a scalable and cost-efficient solution for executing CI/CD jobs on cloud-hosted virtual machines, and with the release of GitLab 12.10, it now supports AWS Fargate managed containers. This advancement allows users to leverage container orchestration solutions, such as AWS Elastic Kubernetes Service and Google Kubernetes Engine, for executing GitLab CI/CD jobs. The GitLab Runner Custom executor enables the creation of drivers for non-natively supported platforms, and the newly developed GitLab AWS Fargate driver uses the AWS Fargate run-task action to schedule tasks, with the pipeline job executing on a container built from a specified image. However, AWS Fargate's inability to support privileged mode poses challenges for Docker-in-Docker operations, though GitLab is committed to iterating on solutions for these limitations. With this update, GitLab provides a diverse array of options for autoscaling CI jobs across major cloud platforms, including cloud-delivered virtual machines and various container orchestration platforms, reinforcing its aim to offer the most comprehensive solution for executing CI jobs at scale.

GitLab emphasizes the principle that everyone can contribute and aims to streamline the development process by delivering test data to developers promptly. The introduction of Automated Accessibility testing in GitLab is part of this effort, allowing developers to identify accessibility issues quickly after deploying changes to a review application. A template has been created to simplify the integration of accessibility testing into GitLab pipelines, which can be included in the project's .gitlab-ci.yml file. This setup aims to minimize cycle time by running the accessibility job after deployment to staging. The accessibility scan results are made visible on the Merge Request page using the artifacts:expose_as keyword, providing developers with easy access to detailed reports from the Pa11y engine. To broaden accessibility report sharing across the organization, the GitLab Pages feature can be used to publish the report consistently at the same URL. After some trial and error, adjustments were made to the .gitlab-ci.yml file to successfully deploy and publish the report via GitLab Pages, enhancing visibility and accessibility for team members.

GitLab Metrics, also known as Application Performance Metrics, is a tool designed for developers and DevOps engineers to monitor and understand the performance impact of their changes within Kubernetes clusters deployed via GitLab. Integrated seamlessly into the DevOps process, it leverages Prometheus to automatically detect, monitor, and display both default and custom application metrics within the GitLab interface. This integration allows for enhanced visibility into system behavior without leaving GitLab, and it supports custom dashboards and alerts. As part of the broader GitLab Monitoring solution, GitLab Metrics aims to reduce production incidents and improve system reliability. GitLab is committed to internal testing and feedback, known as dogfooding, to enhance its metrics capabilities and plans to eventually rely solely on GitLab and Prometheus for monitoring, phasing out Grafana. Users can easily integrate existing Prometheus instances with GitLab or use Prometheus as a managed application, and they are encouraged to contribute feedback and feature requests to further refine the tool.

Automating release generation is essential for streamlining the release management process, particularly as users increasingly deploy with GitLab and leverage more paid features. The challenges faced by users include the need to avoid leaving the command-line interface for release tasks, associating tag creation with pipelines, and simplifying release creation within GitLab's CI/CD processes. The GitLab Release Management team explored several options, ultimately deciding to develop an independent command-line interface (CLI) tool, release-cli, written in Go, that facilitates release creation via API calls. This tool integrates into the CI/CD pipeline by allowing users to include release creation as a job in the YAML configuration file, simplifying the process of creating releases and enhancing user experience. The release-cli is designed to efficiently interact with the GitLab system, retrieving and executing release steps through the Rails API. Future enhancements aim to expand the tool's capabilities, such as exposing YAML as steps, supporting file and binary uploads, and automatically generating release notes.

The blog post explores the concept of iteration in software development, drawing parallels between fighter jets and software engineering, as inspired by John Boyd's Law of Iteration, which posits that speed of iteration trumps quality. This principle is embraced by GitLab, particularly within their Monitor:Health team, where a culture of fast, iterative development is fostered through a strong bias for action, low sense of shame, and emphasis on documentation to facilitate communication and efficiency. The team values delivering rapid results, understanding that mistakes are part of the process, and they prioritize creating an environment where developers can iterate quickly, leveraging feedback loops to refine their work. The focus on iteration helps the team maintain high merge request rates, ultimately enhancing product quality and delivering better customer experiences.

GitHub has introduced several new features, enhancing its platform and underscoring the ongoing competition between Git-based tools like GitHub and GitLab. Among the new offerings is GitHub Codespaces, which provides a fully functional Visual Studio Code experience within GitHub, enabling developers to contribute easily to open source projects through an Azure Cloud environment. Additionally, GitHub has launched Insights for analyzing the software development lifecycle and Code Scanning for enhanced security measures. In comparison, GitLab already offers a range of browser-based tools, lifecycle analytics, and comprehensive security features, positioning itself as an integrated solution that combines development, security, and operations. The announcement encourages the community to engage in the #GitChallenge by comparing the functionalities of GitHub and GitLab, sharing reviews on social media or review sites, and receiving swag as feedback incentives.

GitLab 13.0, set to release on May 22, 2020, introduces a range of updates, including new features and significant deprecations aimed at enhancing its functionality and performance. Key changes involve the Auto DevOps default PostgreSQL chart version, auto deployment settings in response to Kubernetes updates, and a shift from only/except syntax to rules in configuration templates to improve job execution logic. Additionally, the update enforces an offset-based pagination limit of 50,000 for the /projects API endpoint and removes snippet content from searches for improved version control. The JSON common security report format is refined with the introduction of a new id field to replace the deprecated cve field, and several API endpoints and features, such as the token attribute in Runner's details and certain GitLab Runner commands, are deprecated. The release also marks the end of support for Windows 1803 and Fedora 29, changes in URL path structures, and a mandatory transition to the License Scanning template for License Compliance, reflecting a broader effort to align with market terminology and improve user clarity.

GitLab has integrated the ability to create review apps for Android applications using Appetize.io and fastlane, enhancing their continuous integration and deployment process. By updating the .gitlab-ci.yml file, developers can deploy Android apps to Appetize.io, allowing them to run in a browser and be reviewed easily. The integration requires setting up new lanes in fastlane and utilizing Appetize.io API tokens, with environment URLs dynamically generated using dotenv files. Enhancements to the process, such as updating existing apps instead of creating new ones, are achieved by saving the Appetize public key as an artifact and leveraging GitLab's API for retrieving build artifacts. The process concludes with a cleanup step that deletes old apps once they are no longer needed, ensuring efficient resource management. This setup demonstrates GitLab's capability to manage mobile app review workflows, not just for web projects, and could potentially extend to iOS or React Native applications.

GitLab emphasizes the importance of setting specific growth targets and performance indicators to align expectations and measure success effectively. The text discusses various growth strategies for SaaS startups, highlighting the WoW Rule, which suggests a formula for revenue growth: Month over Month (MoM) growth equals 6% plus 14% divided by the last month's ARR in millions. It contrasts with the T2D3 rule, which involves tripling and doubling annual revenue, popularized by Neeraj Agarwal of Battery Ventures. The text also references advice from YC Combinator, which recommends aiming for 5-7% weekly growth, and explains how the WoW Rule suggests a slightly slower but steady growth trajectory to achieve similar results over time. The discussion includes milestones like reaching $100M and $1B ARR, emphasizing the consistent expectation of at least 6% monthly growth, while also mentioning the Hypergrowth rule for spending to sustain growth.

Deploying an application onto Amazon EKS is streamlined by GitLab's Auto DevOps, which simplifies the process for developers by automating the detection, building, testing, deployment, and monitoring of applications with minimal manual setup. This tutorial explains the deployment of a simple Ruby "Hello, World" application onto a GitLab-managed Amazon EKS cluster, highlighting the ease of integration at the group level in GitLab. With Auto DevOps, developers need only push their code, and the system manages the rest, creating a seamless workflow that enhances productivity by eliminating the need for manual cluster setups. The tutorial also demonstrates how changes to the application can be managed using GitLab's Review Apps, which allow for testing in a temporary environment before final deployment, emphasizing the efficiency and simplicity of using GitLab's Auto DevOps for application lifecycle management on Amazon EKS.