January 2021 Summaries

GitLab's Security team, particularly the Red Team, emphasizes transparency and collaboration, reflecting the company's core values even in challenging areas like security. The team openly shares their practices, such as through public projects like Tech Notes and live AMAs, to foster a culture of trust and openness. Their approach involves conducting assumed compromise scenarios, leveraging both known and custom-developed tactics to stay ahead of potential threats, and using tools like GitLab and Vectr for managing testing intelligence. The Red Team collaborates closely with other security and application groups, ensuring operations are transparent and inclusive to break the stigma that "red teamers" are adversaries. Conflict resolution within the team is managed through asynchronous collaboration, and GitLab's remote work policy facilitates diversity in hiring. The company's executive management supports the Red Team's efforts, recognizing their value in reducing risk and supporting career growth paths, including non-traditional ones, thereby reinforcing GitLab's commitment to transparency and continuous learning.

Transitioning from an office environment to an all-remote work setup presents unique challenges and requires adjustments for engineers and managers alike, as highlighted by GitLab team members. While the core responsibilities remain largely unchanged, the methods of communication and collaboration undergo significant shifts, emphasizing the importance of clear and structured communication channels. Remote work demands a bias toward overcommunication, rigorous documentation standards, and the establishment of trust within teams to overcome the lack of physical cues and informal exchanges that naturally occur in office settings. GitLab managers stress the need for intentional efforts to maintain team engagement and interpersonal connections, utilizing tools such as video calls and informal chat channels to replicate the social aspects of in-person work. The shift also necessitates a move from synchronous meetings to asynchronous communication, leveraging documentation as a central practice to ensure everyone remains aligned across different time zones, ultimately allowing for more flexible and efficient workflows.

Proxy server environment variables such as http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy, and NO_PROXY play a critical role in how web clients interact with proxy servers, yet there is no standardized way to handle them, leading to inconsistencies across different tools and languages. These variables dictate which proxy server to use and define exceptions for certain hosts, with no_proxy being particularly important for excluding specific domains or IP addresses from proxy usage. Historically, support for these variables dates back to early web clients like CERN's libwww and continues to evolve, with different implementations choosing either lowercase or uppercase forms, which can cause issues in multi-language applications. Notably, discrepancies in handling these variables, particularly with case sensitivity and the use of CIDR blocks, can lead to significant issues, as demonstrated by a GitLab customer's experience where different services interpreted the proxy configuration differently, causing failures in web-based operations while command-line operations succeeded. This highlights the necessity for understanding these variations and calls for a standardized approach to managing proxy settings in diverse programming environments to prevent operational disruptions.

GitLab is transitioning from its Bronze/Starter tier to a new three-tier subscription model, which includes Free, Premium, and Ultimate options. This decision reflects GitLab's evolution into a comprehensive DevOps platform, offering features beyond source code management and continuous integration, which some Bronze/Starter customers initially used it for. The Bronze/Starter tier is being phased out to allow GitLab to focus resources on enhancing usability, performance, and security across its offerings. Current Bronze/Starter customers can continue with their existing plan until their subscription ends, and they are being offered transition options, including renewing at current rates for one more year or upgrading to Premium at a discounted price. The Free tier now includes many features previously exclusive to the Bronze/Starter tier, making it a viable option for new users, while GitLab Premium offers advanced capabilities such as enterprise-grade performance, team-level project management, and priority support. The change in tier names aims to provide consistency between SaaS and self-managed deployment options, ensuring clarity for users.

VS Code has become an integral tool in enhancing the code review process for GitLab users, particularly with the GitLab VS Code Extension that allows developers to review merge requests directly within the editor. Initially, GitLab introduced the Web IDE to facilitate quick code changes and commits via a browser, but the need to integrate code reviews into an editor where developers spend most of their time led to the development of the VS Code extension. This extension, which originated as a community project, utilizes the VS Code Extension API to display comments and merge request changes, enhancing the code review experience by allowing users to maintain their workflow within the editor. Although the current functionality is read-only, future updates aim to incorporate interactive features such as editing comments, adding reactions, and creating reviews, bridging the gap between the GitLab web interface and VS Code. This development promises a seamless transition for users who wish to perform both code creation and review within the same environment.

GitLab's Auto DevOps offers out-of-the-box continuous integration/continuous delivery (CI/CD) templates that automate the process of building, testing, deploying, and monitoring applications, enabling developers to focus more on innovation rather than pipeline maintenance. By automatically detecting source code and adhering to best practices, Auto DevOps facilitates a streamlined, efficient workflow that includes stages for building, testing, staging, and production deployment, with an emphasis on early defect detection and security checks. Users can easily enable and customize Auto DevOps through project settings or environment variables, allowing for variations such as skipping certain tests or incorporating canary deployments. The system also supports Auto Review Apps, which provide ephemeral environments for stakeholder review of changes before merging. This automation not only enhances the productivity and speed of releases but also allows for incremental deployments to minimize the risk of outages, ultimately accelerating product delivery and helping businesses maintain a competitive edge in the digital marketplace.

As GitLab faced challenges in scaling its infrastructure, particularly with Git, the company developed Gitaly to improve Git access speed by bypassing the need for network-attached filesystem access. The initial solution involved eventual consistency, where repositories had three copies to ensure read access in case of primary server failure, which was rolled out in version 13.0. The next phase involved achieving strong consistency, ensuring all copies were up to date simultaneously, a feature dependent on Git's awareness of transactions, and rolled out for most write requests. To address the cost implications of tripling storage requirements, GitLab improved performance by enabling any of the three copies to handle read requests, with the feature becoming generally available in version 13.8. Praefect, a component introduced to manage traffic and ensure high availability, is being gradually adopted by customers, with GitLab planning further features like HA Distributed Reads and variable replication factors for future releases.

Open source projects offer numerous benefits beyond contributing code, such as enhancing personal learning when documentation is outdated or incomplete. The author shares a personal experience with Middleman, a static site generator, where the need to modify the preview server's log output led to exploring the Middleman source code to discover lifecycle callbacks not fully documented. They emphasize the importance of having a specific task when navigating unfamiliar codebases and describe the process of identifying callbacks through code exploration and documentation review. The experience highlights how understanding the source code can improve productivity and facilitate contributions to open source projects by identifying undocumented features and sharing them with the community to enhance collective knowledge.

The text presents an analogy between the DevOps journey and a space expedition, emphasizing the continuous improvement aspect of DevOps and the importance of collaboration and feedback in both the "Inner Loop" and "Outer Loop" stages. The Inner Loop focuses on the efficiency of software creation through practices like continuous integration, deployment, and DevSecOps, while the Outer Loop aims to ensure product effectiveness by integrating user feedback and advancing product value capabilities. The text highlights challenges faced in the Outer Loop, such as tool integration and feedback processing, and envisions a future where an organized system empowers all team members, eliminating the need for product managers. The future potential of DevOps is underscored by the development of platforms, like GitLab, that aim to streamline these processes and eliminate silos, creating a more seamless and data-driven approach to product development and delivery.

At GitLab, the challenge of efficiently managing simultaneous contributions from hundreds of contributors daily led to optimizing their pipeline processes, which initially involved over 90 varied jobs. With over 700 pipelines running, each representing a change from team members or community contributors, ensuring rapid completion is crucial for maintaining productivity. In December 2020, merge request pipelines averaged 53.8 minutes to complete, prompting efforts to reduce this time. One significant bottleneck was the frontend-fixtures job, which previously took 20 minutes but was optimized to run in parallel using the GitLab CI's parallel keyword, reducing its duration by about three minutes. Further improvements were made by distributing test files among parallel jobs using the Knapsack gem, which resulted in an additional reduction of approximately 3.5 minutes. These optimizations highlight the importance of identifying bottlenecks and leveraging parallel processing to enhance pipeline efficiency.

GitLab's commitment to iterative development allows for continuous product improvements, though it can sometimes obscure the overall progress made. In 2020, the Threat Insights team at GitLab tackled significant challenges, including the global pandemic and a nascent Vulnerability Management area that lacked a dedicated team. Despite their limited background in vulnerability management, the team undertook a crucial architectural upgrade to GitLab's vulnerability data storage, leading to the release of Standalone Vulnerability Objects in GitLab 13.0. This marked a significant milestone, moving the maturity of Vulnerability Management to Minimal and setting the stage for rapid feature development. Between May and December, the team introduced numerous new features, including over 40 enhancements, technical improvements, and a shift to GraphQL-first development, culminating in Vulnerability Management reaching the Viable maturity level. This progress underscores the team's dedication and adaptability, highlighting substantial achievements within a challenging year and setting an optimistic tone for further advancements in 2021.

Originally published on the GitLab Unfiltered blog and republished in February 2021, this blog post discusses an issue encountered during the rollout of the gitlab-pages PROXYv2 project involving HAProxy's server-state-file directive, which causes the persistence of old port numbers across restarts. This resulted in failed connections as HAProxy attempted to send traffic to outdated ports despite configuration changes in haproxy.cfg. The problem was traced to HAProxy's state file, which retained previous server states, including port numbers, and overrode new configurations upon reloading. A workaround was devised by changing the backend server names in the configuration file, which prevented HAProxy from matching them with the old state, thereby allowing the updated port numbers to take effect. The blog also delves into troubleshooting steps, including traffic capture analysis and the use of tools like tcpdump and opensnoop, and explains the mystery behind unexpected "QUIT" messages found in traffic captures, which were identified as part of the PROXY protocol's version 2 binary header format.

The blog post provides a step-by-step guide on setting up a demo project in GitLab to utilize their Secure scanning functionality, specifically focusing on Dependency Scanning. Users are encouraged to start by creating a project using a template, configuring the Dependency Scanning in the pipeline, and reviewing the results of the scan through both merge requests and the Security Dashboard. The guide highlights that the process is simple and quick, taking approximately 15 minutes, and emphasizes that public projects, educational institutions, and open-source projects can access these tools for free. The walkthrough aims to familiarize users with the process of identifying and managing vulnerabilities within their projects using GitLab's Secure offerings.

GitLab has introduced Learn@GitLab, a centralized learning portal designed to consolidate its educational resources and make it easier for users to find and engage with content about using GitLab. This initiative addresses the challenge of navigating multiple sources of information by offering a streamlined platform where users can access high-quality, self-driven demos, videos, and tutorials. The portal covers various technical topics like DevOps platforms, version control, collaboration, and continuous integration. Learn@GitLab aims to highlight the benefits of a single DevOps platform, such as reduced toolchain complexity, and educate users on integrating security into CI/CD pipelines and understanding developer workflows with Kubernetes. While other educational content will remain available on GitLab's site, Learn@GitLab serves as a primary entry point for users to independently educate themselves about GitLab's capabilities and solutions.

GitLab's security organization features a diverse team of women who share insights into their roles, projects, and career advice in a profile series. With women making up 21% of GitLab's security department, the company is actively working towards inclusivity through hiring models and training programs. The profiles highlight various roles, including Security Risk and Compliance, Risk Management, Security Automation, and Incident Response, showcasing projects like SOC audit reviews, anti-spam services, and third-party risk management. The women discuss their interests in areas like Zero Trust architecture, cloud security, and data privacy, emphasizing the importance of mentorship, continuous learning, and adaptability in the evolving tech landscape. They offer advice for aspiring security professionals, focusing on building a support network, understanding organizational dynamics, and staying updated on security trends, while also encouraging those from non-traditional backgrounds to pursue tech careers.

GitLab's engineering stories highlight the company's commitment to incremental changes and iteration, which are fundamental to their development process. Key endeavors include their migration from Azure to Google Cloud Platform (GCP), which resulted in a 61% improvement in GitLab.com's availability, and a significant PostgreSQL upgrade executed in May 2020. GitLab's transition to Kubernetes and the debugging of critical bugs, such as one in the Docker client library, illustrate the complex challenges they tackle. Additionally, GitLab has focused on enhancing tools like their Web IDE and developing a popular VS Code extension. They faced challenges integrating Elasticsearch for advanced code search but learned from initial setbacks to improve their approach. The company also emphasizes the importance of robust monitoring, as exemplified by their use of Prometheus for anomaly detection, and the reimagining of technical interviews to better reflect job requirements. Finally, GitLab's security team continuously works on identifying vulnerabilities to protect against potential breaches, reinforcing their proactive stance on security.

At GitLab, effective communication is crucial for maintaining efficiency and collaboration in an all-remote environment, where team members are globally distributed. The text explores the challenges of communication, especially in design contexts, where lengthy explanations can hinder productive discussions. It provides practical tips for improving communication, such as prioritizing important information, keeping messages concise, balancing context with brevity, using multiple communication formats, making clear and direct calls-to-action, and employing simple language. These strategies aim to enhance understanding and engagement, ultimately improving the quality of communication and collaboration within remote teams. The text also emphasizes the continuous nature of refining communication skills and offers a structured approach for requesting feedback or presenting work.

During a period of transition at GitLab, the author assumed the role of acting Product Design Manager for CI/CD after their manager left, while also being promoted to Staff Product Designer. This dual responsibility required balancing leadership duties with individual contributions, leading to an understanding that growth often involves discomfort and new challenges. The author focused on defining success in their new role, managing schedules to avoid burnout, and embracing the facilitative nature of management as opposed to direct involvement in design tasks. They learned the importance of being a supportive leader, which involved fostering team relationships and prioritizing team achievements over individual accomplishments. The experience highlighted the distinct differences between individual contributor and managerial roles, emphasizing that management requires redefining personal metrics of success and involves a different kind of satisfaction derived from enabling others' growth. The author shared their journey and insights to provide guidance to those considering a transition into management, reflecting on the challenges and rewards of the role and expressing gratitude for the opportunities provided by GitLab.

GitLab encountered a security issue where commits from their private security repository were inadvertently exposed in the public repository due to a design flaw in their mirroring feature. This flaw was rooted in the fetch-based behavior of GitLab's push mirror setup, which made private Git objects known to the public repository. The initial solution involved pausing the mirroring during security fixes, but this caused the private repository to lag behind public development, risking merge conflicts. A more effective solution was proposed by using Git's ls-remote command for branches, similar to how it was used for tags, preventing the commits from being stored on-disk in the public repository. This change was cautiously tested with a feature flag, and after successful experiments, it was implemented without any reported errors. However, an annoying bug arose when ls-remote's execution order caused transient errors due to the time taken to complete, which was resolved by swapping the order of reference gathering. This fix finally closed the issue, ensuring a secure and efficient mirroring process.