June 2015 Summaries

GitLab's monthly release process relies on an unsung hero known as the release manager, who is tasked with coordinating and ensuring the timely release of the latest GitLab version on the 22nd of each month. The role involves managing thousands of changes from a large developer community, ensuring all components like GitLab CE, EE, CI, and Omnibus packages are ready, and addressing any regressions or necessary fixes. Despite improvements over the years, such as automation and a more continuous integration approach, the job remains unpopular due to its complexity and responsibility. Selecting a new release manager is challenging, often met with reluctance, despite attempts to make the role more appealing through simplified procedures and potential rewards. The blog post serves as a tribute to these release managers, acknowledging their crucial yet often invisible contributions to GitLab's seamless operations.

An employee at GitLab shares their experiences and insights about working in a remote role, highlighting the challenges and benefits of this work style. Initially, communication was difficult due to the lack of in-person interaction, but the employee learned to effectively use tools like Slack and to not fear asking questions. Working from home allowed for fewer distractions compared to an office environment, with tools like noise-canceling headphones proving beneficial. The flexibility of remote work enabled the employee to adopt a comfortable, yet disciplined, routine that includes working in pajamas during morning calls. Trust plays a crucial role in remote work, fostering motivation and a desire to prove reliability. Finding a workspace with a stable internet connection and ergonomic setup is essential for maintaining productivity and well-being. The greatest advantage of remote work for the employee is the ability to travel and work from any location, providing a balance between professional responsibilities and personal life, including spending more time with their pet.

GitLab addressed the Logjam vulnerability by updating its SSL cipher suites, opting for stronger security measures while balancing compatibility with older Java-based clients. Initially using 1024-bit DH groups for broader client support, GitLab decided to implement 2048-bit DHE params for enhanced security, although this risked excluding users with outdated Java clients. After evaluating the drawbacks of DHE, such as reduced speed and limited browser support, GitLab removed DHE suites from its configurations, aligning with practices seen on Google sites. This change resulted in improved SSL labs scores from B to A and maintained forward secrecy for most major browsers through ECDHE, although it compromised forward secrecy for a small number of older systems like Android 2.3.7, Java 6, and OpenSSL 0.9.8. The updates also included revised configurations for both omnibus-gitlab packages and source installations, ensuring that the majority of users benefit from the heightened security.

GitLab Enterprise Edition 7.12 introduces a new feature called Merge Request Approvals, which allows users to require multiple reviewers to sign off on a merge request before it can be merged. This feature enables a more controlled review process by setting a numeric value for the number of approvals needed, ensuring that every merge request is reviewed by several people. Users can enable this feature by adjusting the "Approvals required" setting in the project settings, and once the specified number of approvals is met, the merge request can be merged as usual. This addition aims to enhance collaborative workflow and quality assurance within teams.

GitLab announced changes to its Enterprise Edition subscription pricing, aiming to better reflect the value of each offering while accommodating different organizational needs. As of June 12, 2015, the Basic Subscription will cost $390 annually for a 10-user pack, effectively $39 per user per year, with existing subscribers receiving a temporary 25% discount on renewals. Meanwhile, the Plus Subscription is reduced to $14,900 for a 100-user pack, equating to $149 per user annually, with current subscribers receiving prorated refunds for the price difference. The Standard Subscription remains unchanged at $4,900 per 100-user pack. GitLab intends for these adjustments to maintain its status as an affordable enterprise-grade development platform, as the pricing changes address the previous disparity between the Basic and Plus plans. Existing Basic Subscribers will benefit from the old pricing until June 12, 2016, after which the new rates will apply to all.

Installing GitLab from source requires careful attention to the version of Git used, particularly in the context of the security vulnerability CVE-2014-9390, which does not affect GitLab's Omnibus packages but can impact servers delivering compromised repositories to users. Though GitLab itself is unaffected by this vulnerability, users compiling Git from source must ensure their Git version is protected against it, as the previously recommended Git 2.1.2 is not secure. Administrators using the source installation method should verify their Git version, especially if their GitLab server operates with /usr/local/bin/git, to prevent the risk of hosting or distributing 'poisoned' repositories.

The transition from GitLab CI jobs to a single .gitlab-ci.yml file aims to streamline the process of defining continuous integration (CI) jobs by placing them in the root of the repository, rather than using the GitLab CI interface. This change involves GitLab sending webhooks along with the .gitlab-ci.yml contents to the CI Coordinator, which creates builds executed by Runners. To avoid the complexity and confusion of maintaining two systems, the old method of defining jobs will be completely phased out, and existing jobs will be automatically converted to the new YAML syntax. This migration process is simplified, as users can download the converted YAML file from project settings and paste it into their repository. An example .gitlab-ci.yml for a Ruby on Rails project demonstrates the new structure, which includes sections for scripts, parallel jobs, and deployment tasks, emphasizing simplicity and memorability in scripting. Community involvement is encouraged, as GitLab values contributions to its ongoing development.

GitLab utilizes Unicorn, a pre-forking Ruby web server, to manage web requests for both its Community and Enterprise Editions, leveraging its multi-process architecture to optimize CPU usage and enhance fault tolerance. The Unicorn 'master' process initiates a clean Ruby environment and spawns worker processes to handle requests, replacing any that crash or time out, ensuring uninterrupted service. To manage memory leaks inherent in long-running processes, GitLab employs the unicorn-worker-killer gem, which performs memory checks on Unicorn workers after every 16 requests, terminating and replacing those that exceed a set memory limit without dropping user requests. This approach, while resulting in frequent worker restarts, particularly on some GitLab sites, is designed to maintain service reliability and prevent memory leaks from impacting user experience.

GitLab.com experienced an outage on May 29, 2015, due to a backup script malfunction that caused the filesystem on the backend server to remain frozen, leading to a service disruption that lasted approximately 94 minutes. The outage was compounded by the recent infrastructure upgrade, which added complexity to the system, and by insufficient training and documentation for on-call engineers. The recovery process involved diagnosing issues with the NFS share and backend server, ultimately requiring a reboot to restore functionality. In response, GitLab removed the freeze/unfreeze steps from the backup script, implemented a secondary backup strategy for SQL data, and prioritized training through regular operations drills to enhance the preparedness of their engineering team.