September 2021 Summaries

The Global DevSecOps Survey highlights that testing is a primary cause of release delays due to insufficient automated testing and excessive manual tests conducted late in the software development process. Despite advancements in DevOps, integrating automated testing remains challenging, with many teams lacking automation and developers struggling to access test results within their IDEs, which hampers productivity. While there has been progress, such as an increase in fully automated teams from 2020 to 2021, access to essential test data has not significantly improved. Developers emphasize the importance of receiving test results during their workflow to address bugs promptly, advocating for more automation, AI/ML, and interconnected DevOps platforms. Glympse exemplifies the benefits of adopting a DevOps platform, transitioning from using multiple tools to significantly reducing deployment times and fatigue. Teams that have successfully integrated automated testing report increased confidence, efficiency, and the ability to release updates more reliably, demonstrating the substantial benefits of overcoming these challenges.

GitLab has enhanced its user profiles by introducing new settings that allow users to personalize their profiles more effectively, focusing on inclusivity and accessibility. Users can now add pronouns to their profiles, which are displayed next to their name in public views and when hovering over their name in issues or merge requests, promoting the correct use of pronouns in communications. Additionally, a pronunciation guide can be added to help team members from diverse backgrounds correctly pronounce each other's names. The local time feature has also been updated to display the user's local time, aiding team members in understanding each other's availability. Future updates plan to include the user's timezone in the snapshot view of the profile, and GitLab encourages community contributions through issue requests or direct code submissions to further enhance these profile features.

In an investigation to resolve mysterious database stalls on GitLab.com, it was discovered that SQL SAVEPOINT queries within long transactions were causing performance issues on database replicas, leading to 500 errors for users. The problem, dubbed "Nessie," was traced to PostgreSQL's handling of subtransactions, where a single SAVEPOINT during a long transaction could trigger suboverflow, leading to cache misses and high disk I/O. This was exacerbated by replication differences between primary and replica databases. To mitigate this, GitLab eliminated all SAVEPOINT queries from their code, which resolved the issue. The investigation was aided by extending observability tools and leveraging community expertise, highlighting PostgreSQL's robust code but also its limitations in handling subtransactions. The team considered but ultimately opted against using unofficial PostgreSQL patches that could allow larger subtransaction caches, focusing instead on code modifications to prevent the issue from recurring.

Semantic versioning (SemVer) is widely regarded as the standard for tracking software version states, although in practice, many ecosystems have developed their own dialects, complicating automation in dependency scanning and vulnerability assessments. GitLab's Dependency Scanning (DS) feature identifies software vulnerabilities using these diverse SemVer dialects, leveraging the GitLab Advisory Database, which is updated daily. The fragmentation of SemVer has necessitated the creation of semver_dialects, a tool that processes semantic versions generically and language-independently, facilitating automated advisory generation. This tool, now open-sourced, employs linear interval arithmetic to normalize and compare semantic versions across different ecosystems, simplifying the management of version constraints. Through semver_dialects, GitLab enhances its ability to automate the generation of advisories by translating native semantic version constraints into a common representation, thus improving efficiency in identifying non-affected versions and providing remediation information for vulnerable packages.

Configuring Sidekiq in a GitLab deployment, particularly at scale or in special cases, requires nuanced adjustments to address workload distribution and job characteristics. The blog discusses the challenges faced by GitLab's Demo Systems team, who encountered issues with the default Sidekiq configuration during training sessions, leading them to develop a dedicated Sidekiq virtual machine for specific tasks like project imports. The importance of identifying pain points through metrics is emphasized, suggesting the use of tools like gitlab-exporter for enhanced visibility into queue sizes and job distribution. The text provides two main strategies for customizing Sidekiq—using queue selectors or routing rules—to optimize processing power, improve workload management, and reduce Redis load. It also highlights the necessity of monitoring Redis CPU usage to preempt potential saturation issues and offers guidance on migrating active GitLab deployments to new configurations with minimal disruption. The document underscores the need for careful planning and the use of metrics to tailor Sidekiq configurations effectively, ensuring that workloads are processed efficiently and without unnecessary delays.

With a global shortage of software developers, especially in the DevOps domain, companies need innovative strategies to attract and retain talent, as salary alone is often insufficient. The U.S. Bureau of Labor Statistics projects a 22% increase in employment opportunities for developers and testers from 2020 to 2030, necessitating nearly 190,000 new jobs annually. To stand out, organizations can streamline their toolsets by adopting a unified DevOps platform that integrates essential tools, thereby enhancing productivity and job satisfaction. Career education is another key aspect, as many developers self-educate but value employer-provided training, particularly in areas like AI/ML and advanced programming languages. Flexibility in work arrangements, such as remote or part-time options, and encouraging involvement in open-source projects are also important for job satisfaction. Ultimately, fostering a strong DevOps culture can lead to happier developers, benefiting both the organization and its workforce.

Effectively communicating customer needs to product teams is crucial for a product's success, as demonstrated by GitLab's practices, which emphasize transparency and collaboration. GitLab employs public issue trackers to manage feature requests, allowing both customers and community members to observe and contribute to the product development process. When a customer requests a feature, technical account managers log the request in these trackers with generic customer details, while the specifics remain confidential in a CRM system like Salesforce. Product managers then review and prioritize these requests, using labels and milestones to track progress. Engaging customers in this manner fosters trust and enables direct customer input, which can guide prioritization and development. GitLab encourages community contributions to feature development, aligning with their motto "Everyone Can Contribute," and offers support through Merge Request Coaches. This approach not only helps in tracking and delivering features effectively but also builds empathy and connection between developers and end users, ensuring that customer feedback is seamlessly integrated into the product team's workflow.

GitLab's DevOps platform enhances collaboration by offering developers tools like the Web Editor and Web IDE, which now include a real-time preview feature for Markdown editing, introduced in GitLab 14.2. This feature allows users to see changes in Markdown content side-by-side as they write, reducing the need for disruptive context-switching between editing and previewing. Markdown, a lightweight markup language created by John Gruber and Aaron Swartz in 2004, is favored for its simplicity and platform independence, allowing users to format text without needing HTML knowledge. It is widely used by writers and developers for creating web content, notes, and more, with GitLab enabling the easy integration of Markdown into Git repositories. The real-time preview feature aims to streamline the writing process, making it more accessible for users with varying technical expertise, while fostering an inclusive and collaborative environment where everyone can contribute.

Integrating accessibility early in the software development lifecycle (SDLC) is crucial for creating inclusive products and avoiding technical debt, a challenge that can be efficiently addressed through the use of a DevOps platform like GitLab. Despite legal requirements and the potential to broaden user bases, accessibility is often overlooked or added as a final test rather than being embedded into the development process from the start. Educating developers about accessibility tools such as screen readers and fostering empathy can drive the shift left approach, allowing teams to proactively address accessibility issues and implement adaptable solutions that meet diverse user needs. By incorporating accessibility considerations early, companies can enhance their product offerings, reduce costly reworks, and unlock additional business value, benefiting both users with disabilities and those with specific workflow preferences.

QPage, a company providing recruitment solutions for SMEs, transitioned to GitLab's DevOps Platform from a local CI/CD system to enhance their deployment process, driven by their co-founder Pouya Lotfi's prior positive experiences with GitLab. The move to GitLab allowed QPage to leverage integrations such as GitLab-Kubernetes, Auto DevOps, and JIRA, streamlining their continuous integration and deployment pipelines. Developers at QPage, familiar with GitLab or GitHub, find its documentation helpful, reducing reliance on individual expertise for problem-solving. The integration with Docker and use of cloud providers AWS and Digital Ocean have further optimized their deployment, dividing it into staging and production phases. GitLab has significantly improved QPage's operational efficiency, reducing deployment times from 6-8 hours to just 15-20 minutes, while enhancing visibility, collaboration, and cost-effectiveness in managing their primary and 29 sub-products.

Infrastructure as Code (IaC) on public cloud platforms can be efficiently organized using tags and labels, and tools like Terratag can automate this process to enhance management and reduce human error. Developed by env0, Terratag integrates with Terraform and the GitLab CI/CD platform, simplifying the tagging and labeling of cloud resources across major providers like AWS, Google Cloud Platform, and Microsoft Azure. GitLab offers various tools for managing Terraform deployments, including remote state management and private module registries, which make it easier to run CI/CD processes at scale. Terratag automatically tags all resources and sub-modules in Terraform code, facilitating cost management, organization, and reporting. This automation addresses the challenges of manual tagging, such as maintaining standards and managing metadata, and supports the seamless use of tags for various operational and business purposes. The blog post demonstrates practical applications of this tool with GitLab CI/CD, emphasizing the technical and cost benefits of effective labeling.

Weet, an asynchronous video communication tool, overcame its localization challenges by integrating GitLab and Lokalise into its workflow, enabling a more efficient and seamless app development process. Initially hindered by cumbersome localization methods that slowed development and introduced bugs, Weet transitioned to using Lokalise, which integrates with GitLab to automate and streamline the translation of text strings, allowing developers to focus on coding while language experts handle translations. This integration has allowed Weet to localize new releases swiftly, significantly improving their development cycle from days to less than an hour with minimal quality checks. By splitting localization data into multiple projects and using a consistent naming pattern for keys, Weet has maintained clarity and simplicity in their process, paving the way for future expansions such as mobile apps and additional languages. This seamless localization process has contributed to Weet's success, as evidenced by its recent recognition on Product Hunt.

The GitLab Agent for Kubernetes (agentk) is a key component for integrating GitLab and Kubernetes, facilitating GitOps operations via secure, cloud-native methods. It communicates with the GitLab Agent Server (KAS) to apply changes from GitLab-managed manifest files to Kubernetes clusters. By deploying agentk with specific namespace access, users can limit its permissions, enhancing security. The guide outlines the deployment of agentk on a Kubernetes cluster, using tools like Google Kubernetes Engine, and details the setup of service accounts, roles, and role bindings to control access. It also explains how agentk applies changes, manages applications in specified namespaces, and logs deployment activities, offering a more secure GitOps workflow by restricting agent access to selected namespaces.

Organizations are increasingly adopting a hybrid approach to application development and deployment, integrating open source software, DevOps, and cloud computing, which presents challenges for traditionally siloed mainframe development teams using disparate toolsets. GitLab seeks to address this issue by offering a unified DevOps platform that enhances support for IBM z/OS applications, promoting common workflows that align with Agile processes to ensure consistency across hybrid environments. Security and compliance remain crucial, with DevSecOps methodologies embedding security testing into the software development lifecycle, thus enhancing code quality and deployment efficiency. Collaboration and automation are emphasized through CI/CD pipelines, enabling developers to deploy applications at scale while integrating security and development workflows to break down team silos. GitLab Ultimate's integration with both container platforms and IBM Z environments provides flexibility in application deployment, supporting Agile practices and enhancing automation within organizations.

Code reviews are essential in software development but can be challenging, so enhancements to a DevOps platform can streamline and improve this process. Effective code reviews require understanding the full context of proposed changes, including their impact on code quality, security, performance, and maintainability. The integration of features such as merge requests centralizes change management, providing necessary information for informed decision-making. GitLab has introduced several features to simplify code reviews, such as allowing them to be conducted within familiar development environments like Visual Studio Code, integrating Gitpod for quick development setups, and embedding code quality notices directly within merge request diffs. Additional tools include file-by-file reviews, the ability to mark files as reviewed, and distinguishing between "reviewers" and "assignees" in merge requests to clarify responsibilities and expedite the review process. These improvements aim to make code reviews quicker, more effective, and less burdensome, with ongoing updates and community contributions enhancing the platform further.

In summer 2021, GitLab's Vulnerability Research and Static Analysis teams embarked on a Google Summer of Code (GSoC) project to develop a framework for transitioning from various language-specific Static Application Security Testing (SAST) tools to Semgrep, a language-agnostic SAST tool. This project aimed to reduce the maintenance burden and inflexibility associated with multiple SAST tools by creating Semgrep rule-sets equivalent to existing analyzers, ensuring they produce comparable results. A central rule repository was established to manage these rules and their corresponding test cases, with GitLab CI/CD automating the testing and validation process. The framework facilitated the replacement of the C/C++ analyzer Flawfinder with a Semgrep rule-set, using automated gap analysis to measure and ensure parity between the original tools and the new configurations. This approach enabled the iterative refinement of rules to achieve full coverage and parity, effectively simplifying SAST tool management and enhancing GitLab's vulnerability detection capabilities.

The 2022 Global DevSecOps Survey highlights the growing importance of DevOps platforms in streamlining software development processes, as evidenced by insights from over 5,000 DevOps professionals. Development teams are increasingly adopting DevOps platforms to release safer software more quickly, with these platforms being among the top choices for enhancing DevOps practices alongside CI/CD, test automation, and DevSecOps. The DevOps market, valued at $6 billion in 2020, is projected to grow significantly, driven by the effectiveness of DevOps in increasing release speed and revenue growth. However, managing multiple tools can lead to a "DevOps tax," where teams spend considerable time on toolchain maintenance. A unified DevOps platform offers a solution by providing end-to-end visibility and integration, reducing maintenance time, and improving performance metrics such as mean time to recovery and time to market. Adoption of such platforms has resulted in improved collaboration and automation, as seen in companies like BI Worldwide and Glympse, which have successfully streamlined processes and enhanced security measures by consolidating tools into a single platform.

Drive-by attacks exploit vulnerabilities in browsers and local network services to execute malicious actions on a user's computer, bypassing traditional security measures like firewalls and antivirus software. These attacks typically involve websites that contain harmful JavaScript code targeting specific vulnerabilities within the browser or accessible network services. A notable example includes how multiple vulnerabilities in the GitLab Development Kit were chained to achieve remote code execution on developer laptops. The attack leveraged permissive CORS headers and improper content-type validation to execute arbitrary commands. To mitigate such risks, it's crucial to implement strong protections against cross-origin requests, regularly inspect network services for vulnerabilities, and consider using virtual machines or containers to isolate potentially insecure environments. Additionally, securing browsers through plugins and segmenting them in virtual environments can add extra layers of protection against drive-by attacks.

The 2022 Global DevSecOps Survey reveals insights from over 5,000 DevOps professionals, highlighting the importance and challenges of code review in the DevOps process. Despite being a critical factor for ensuring code quality and security, code reviews are often a source of frustration and delay due to the need for context-switching, collaboration, and subject matter expertise. Survey respondents noted that code reviews could be time-consuming, with a lack of reviewers and communication issues often exacerbating delays. However, the majority still find them "very valuable," and many teams are exploring ways to improve the process, such as using smaller merge requests, automated pipelines, and collaborative pull requests. Developers expressed a desire to increase the frequency of code reviews, indicating their recognition of its significance in maintaining high standards in code quality. GitLab, for instance, has focused on refining its code review practices as part of continuous improvement efforts within its DevOps Platform.

GitLab CEO Sid Sijbrandij's blog post discusses the evolution of DevOps practices and the emergence of the DevOps Platform Era as a solution to the challenges faced by organizations at various maturity stages. Historically, managing DevOps toolchains required substantial investment without necessarily adding value beyond maintenance, often described as a "DevOps tax." The rapid evolution of DevOps has led to outdated tools and processes, prompting a shift towards integrated DevOps platforms that enhance collaboration, automation, and traceability. GitLab has been at the forefront of this transition, advocating for a single-application DevOps Platform that supports rather than burdens users, akin to the evolution seen in CRM platforms. The blog highlights the tangible benefits of such platforms, as evidenced by their 2021 DevSecOps Survey, which reports improved DevOps performance, quicker time to market, and reduced failure rates. This transition, supported by industry recognition from entities like Gartner, reflects a broader platform mindset shift, enabling organizations to streamline their processes and better adapt to changing requirements.

GitLab's experience with configuring Sidekiq, a background job processor for Ruby-on-Rails using Redis for job queues, highlights challenges faced in scaling large deployments like GitLab.com. Initially, GitLab used a one-queue-per-worker approach, which increased complexity as the number of job classes grew to over 400, straining Redis's single-threaded CPU usage. To address this, GitLab transitioned to a routing rules system that simplifies job queue management by directing jobs into fewer queues, reducing CPU saturation from 95% to 75%. This change involved experimenting with workload simulations and gradually shifting jobs to new routing rules, which helped optimize the catchall shard's performance. While this approach eased the current load on Redis, GitLab anticipates that Redis will eventually become a bottleneck again, prompting considerations for future architectural changes or introducing multiple Sidekiq fleets.

GitLab participated in the Google Summer of Code (GSoC) for the first time, hosting four student interns who worked on distinct projects with guidance from multiple mentors. This initiative, part of Google's long-standing program to introduce students to open source, involved over 200 organizations and offered GitLab an opportunity to engage with young talent. The interns, hailing from diverse backgrounds, worked on projects ranging from enabling Courseware as Code to improving backup and restore features, with mentorship praised for being supportive and enriching. Mentors noted the effectiveness of asynchronous communication and highlighted the importance of clearly defining project deliverables and required skills upfront. While the program was deemed successful and fulfilling, areas for improvement were identified, such as enhancing the community bonding period and refining project visibility. The overall experience was positively received, with hopes of participating in future GSoC programs.