November 2019 Summaries

GitLab security researchers identified a critical security vulnerability involving REST API calls that exposed the potential for a path traversal attack, which could result in unauthorized access and privilege escalation on gitlab.com. The vulnerability was demonstrated using GitLab's Customers Portal, where user-controlled parameters in API requests could be manipulated to traverse and access unintended API endpoints. This flaw allowed attackers to inject arbitrary attributes, potentially promoting regular accounts to admin status with minimal effort and cost. The problem was swiftly mitigated by enforcing numerical constraints on the vulnerable parameter and implementing additional security measures to prevent similar attacks. This incident underscores the importance of rigorous testing and secure coding practices in applications that rely on backend services via API calls, highlighting the role of security research in maintaining the integrity of GitLab's products and services.

The deadline for submitting proposals to the 2020 KubeCon/CloudNativeCon Europe is approaching, and GitLab's Technical Evangelism team is eager to assist participants in preparing their submissions. Enthusiastic about joining the Cloud Native community after attending KubeCon in San Diego, the author highlights the inspiring nature of the event and GitLab's commitment to encouraging community engagement. The team offers to review and develop any proposal ideas for the open call, with the submission deadline set for December 4, 2019, at 11:59 PM PDT. GitLab aims to amplify the voices of its passionate and engaged community, inviting participants to use their form or contact the Director of Technical Evangelism via Twitter for support in crafting their proposals for the event in Amsterdam.

Creationline, a GitLab reseller in Japan, stands out by actively contributing to both their customers and the GitLab community. Their engagement with GitLab began when they discovered the company's transparent and open-source philosophy, leading them to start distributing GitLab licenses in 2017. Jean-Baptiste Vasseur, an Agile Coach and DevOps Consultant, highlighted how the team has been involved in consulting, organizing meetups, and delivering CI/CD workshops, while also contributing to GitLab's codebase despite the challenges of balancing time. Hiroyuki Sato, a GitLab Evangelist, shared his journey from a user fixing a bug affecting Japanese language users to becoming a regular contributor and eventually joining Creationline to work more closely with GitLab, now involved in pre-sales, marketing, and customer support. Yuko Takano, Customer Success Manager, explained how the team uses GitLab internally across various business functions to better support their clients, showcasing their commitment to understanding and leveraging the product's features. The blog post encourages others to contribute to GitLab, offering resources and contact information for those interested.

This blog post, republished from the GitLab Unfiltered blog, presents the findings of a survey conducted to understand AWS Lambda tooling habits among GitLab users. The survey, which received 58 responses, explored various aspects such as the frameworks, testing tools, CI/CD tools, and monitoring habits used with AWS Lambda, with a focus on differentiating between hobby and enterprise users. The results indicated that the Serverless Framework is popular among respondents, with SMBs being notable users of both serverless and Terraform, while enterprises show significant usage of AWS CLI. The survey highlighted a lack of robust testing infrastructure for serverless technologies, as many projects either lack testing or only perform unit tests, with GitLab being a favored CI/CD tool among respondents. For monitoring, AWS CloudWatch was predominantly used, even though more advanced instrumentation was expected for production environments. The post concludes by highlighting GitLab's capabilities in integrating with AWS Lambda and offers project templates to help users get started.

The author reflects on their journey to improve productivity by shifting from an office environment to remote work, inspired by Cal Newport's concept of "Deep Work" and the "E-factor" from Peopleware by Tom DeMarco and Timothy Lister. They discovered that the ability to focus deeply is essential for producing valuable work, which is hindered by distractions typical in open-plan offices. By working remotely at GitLab, the author was able to control their environment, leading to an increase in uninterrupted work time from two hours to over four hours daily, significantly boosting their E-factor and productivity. They emphasize the importance of a dedicated workspace, a strong routine, and asynchronous work policies in facilitating deep work and reducing distractions, while also advocating for tracking time spent on tasks to demonstrate productivity improvements to managers, especially for those unable to work remotely.

Migrations, particularly from Jenkins CI to GitLab CI/CD, can be daunting due to the fear of unknown complexities, yet they are often necessary to avoid brittle builds and cumbersome plugin maintenance. Teams like Linagora have successfully transitioned by leveraging GitLab's comprehensive features, including Git repository management and built-in CI/CD, supported by Docker and Kubernetes for seamless integration. A practical interim solution involves running Jenkinsfiles within GitLab CI/CD using Docker while updating syntax, and Auto DevOps offers predefined configurations to simplify pipeline conversion from Groovy to YAML, including additional features like security and performance testing. The experience of companies such as adSoul highlights best practices such as starting with small projects, effectively using available tools like Docker and Auto DevOps, and maintaining clear communication throughout the migration process.

This text presents various tools and techniques to enhance productivity while using GitLab, particularly through customized navigation and automation. It highlights the use of keyworded bookmarks in browsers like Firefox and Chrome to quickly access specific GitLab pages, allowing for efficient navigation and search capabilities. The text explains how to create quick navigation links by using URL parameters and describes the process of adding custom search engines for GitLab documentation and handbook. Additionally, it introduces the TamperMonkey browser plugin for scripting webpage interactions, enabling the creation of custom quick actions such as applying labels or filtering tasks. The text also mentions using scripts to filter to-do lists and checkboxes, emphasizing the potential for personalized enhancements to streamline GitLab workflows. Lastly, it encourages readers to share their own tips for optimizing the GitLab experience.

GitLab, initially embraced by the HackerNews community and Y Combinator, reached a valuation of $2.75 billion during its Series E funding round and is now committed to supporting the startup community by offering its top-tier services at no cost to eligible startups, including Y Combinator companies. Recognizing the challenges startups face, such as hiring, partnerships, and financial management, GitLab provides free access to its self-managed Ultimate or cloud-hosted Gold tiers, which support the entire DevOps lifecycle and help streamline operations by reducing toolchain complexity and cycle time. The offer is initially extended to YC companies from the current or two most recent batches with less than $3M in funding, and it includes a year of free access and optional discounted support. Startups can apply via GitLab's Startups page, with further details available in their FAQ section or through direct contact.

After acquiring Gitter, the team set out to open-source its mobile Android and iOS apps, following the successful open-sourcing of the main web app in 2017. This process involved moving the projects from GitHub to GitLab, removing sensitive information using tools like truffleHog and BFG Repo-Cleaner, and setting up configurations to keep secrets out of source control. For Android, secrets were managed using a `secrets.properties` file, while for iOS, a `GitterSecrets-Dev.plist` was used. Both projects were eventually made publicly available, with the community encouraged to contribute improvements. Despite considering deprecating the apps, community support has kept them alive, with ongoing updates like dark themes and technical fixes being added.

The multicloud strategy, where organizations deploy applications across multiple cloud platforms like AWS, Azure, or Google Cloud, has gained popularity due to its flexibility and ability to avoid vendor lock-in. This approach allows companies to manage costs by leveraging global data centers and enhances operational resilience by minimizing downtime during outages or breaches. However, multicloud introduces complexity and increases the risk of cyber threats by expanding the attack surface. Effective multicloud management requires a comprehensive security strategy that includes a Zero Trust approach to access, consistent patching and upgrades, and centralized security monitoring across all platforms. Organizations are advised to adopt cloud-agnostic security tools, enforce uniform security policies, and consider confidential computing to protect data in use. As the cloud landscape evolves, companies must remain vigilant and adaptable, ensuring their strategies are robust against unforeseen changes and new technological trends to mitigate potential security breaches.

Deploying applications with GitLab has become streamlined, enabling developers to create a Kubernetes cluster on any cloud platform, connect it via GitLab's Kubernetes integration, and leverage Auto DevOps for a full deployment pipeline. However, the challenge arises when applications need to be run across multiple clusters in varied environments, including private data centers and air-gapped servers. Gravitational's open-source tool, Gravity, addresses this by allowing developers to create "cluster images," which are comprehensive, portable Kubernetes environments that can be deployed anywhere, maintaining consistency and compliance. Gravity uses standard CNCF-supported tools to bundle entire Kubernetes clusters with their applications and dependencies into .tar files, facilitating easy replication of environments across different infrastructures. GitLab serves as a central repository for application development, using its CI/CD capabilities for continuous delivery, while Gravity enables scaling across diverse environments, ensuring applications can be deployed consistently in both cloud and on-premises settings. Gravitational's approach simplifies deployment complexity, allowing for seamless application distribution, with GitLab and Gravity working in tandem to provide robust solutions for developers facing multi-cloud and on-premises deployment challenges.

Integrated toolchains are making a comeback in the software delivery market due to the rise of CI/CD and open-source tools, but their complexity can hinder innovation and compromise security, according to Forrester analyst Christopher Condo. Many development teams face the challenge of managing multiple toolchains, with some devoting significant resources to maintenance, which detracts from product development and increases the risk of security breaches. GitLab offers a solution with its out-of-the-box DevSecOps platform, integrating security into the early stages of the DevOps lifecycle to streamline processes, enhance communication, and improve visibility. By providing a unified application for the entire software delivery lifecycle, GitLab aims to accelerate delivery cycles while maintaining quality and bolstering security practices.

CI/CD processes, aimed at accelerating software release, often compromise security, posing significant challenges such as the lack of automated security testing tools and a broader attack surface that traditional methods can't protect. The security of the CI/CD pipeline itself is crucial, as it presents a tempting target for hackers due to its end-to-end software lifecycle. To address these issues, DevOps teams are encouraged to integrate security considerations throughout the software development lifecycle, embracing DevSecOps practices to safeguard their infrastructure effectively. Key strategies include automation, access management, enhancing user experience, and maintaining transparency to create a secure CI/CD environment. Automation helps match security practices with CI/CD's speed, access management ensures secure interactions, user-friendly tools reduce risky workarounds, and transparency fosters accountability. Using a single, well-integrated CI/CD tool, like GitLab, can further streamline security measures by embedding checks within the development workflow, reducing friction, and maintaining a single source of truth. Ultimately, achieving a balance between speed and security requires collaboration among development, operations, and security teams, fostering a culture of shared responsibility for software security.

At the Google Cloud Next '19 UK event, Google Cloud expanded its Anthos product line by introducing Cloud Run for Anthos running on-premise, with GitLab actively collaborating to support this launch by integrating CI/CD and GitLab Serverless capabilities to streamline the adoption of serverless solutions. This collaboration continues from their earlier announcement at Google Cloud Next '19 in San Francisco, aiming to maintain consistent user experience and workflows across Cloud Run deployments on Anthos. The partnership seeks to simplify the adoption of scalable, cloud-native solutions by leveraging infrastructure-centric managed services such as Google Kubernetes Engine and Cloud Run across hybrid environments. As GitLab serves as a comprehensive DevOps application, it facilitates the building, deploying, and managing of serverless applications without needing intricate Kubernetes configurations, using templates for deploying Knative services to Cloud Run. Currently, users can deploy to Cloud Run in three configurations: fully managed cloud service, Anthos on Google Cloud, and Anthos on-premise, with GitLab ensuring compatibility with various Knative versions and planning future support upgrades. Users are encouraged to provide feedback, with developments in internal testing being positive, as GitLab prepares to offer full support for Cloud Run for Anthos on-premise in upcoming releases.

Since the public launch of their bug bounty program in December 2018, the company has received 1,282 reports from external security researchers and paid out $515,899 in rewards. Initially, they adopted a model where part of the bounty was paid immediately upon triage of a report, but as of November 18, 2019, they increased awards for critical and high-severity vulnerabilities to incentivize the community further. The new payout structure offers $20,000 for critical vulnerabilities and $10,000 for high-severity ones, while the criteria for program scope, severity assessment, and engagement rules remain unchanged. This initiative acknowledges the significant contributions of the security research community, enhancing the security of the company's products. Additionally, a bug bounty contest running from October 1 to November 30 offers participants a chance to win GitLab swag.

A GitLab Runner issue that began with a static analysis failure led to a complex investigation uncovering a bug in the Docker client library, eventually resolved by upgrading the Go compiler. The problem involved silent test failures in GitLab's automated test infrastructure, causing productivity issues due to lack of output when errors occurred. The investigation revealed that changes in TCP keepalive settings and live CI trace feature flags were not the culprits, pointing instead to differences between Docker Machine and Docker executors. The issue was linked to Google Cloud Platform's 10-minute idle timeout affecting Docker Machine executors, which could be mitigated by adjusting sleep commands in the CI configuration. Attempts to monitor and decrypt network traffic between the GitLab Runner and Docker container were initially thwarted by HTTPS encryption and perfect forward secrecy, requiring modifications to disable specific cipher suites for effective debugging.

During his presentation at GitLab Commit London, Mario García shared his experience of overcoming various challenges while transitioning his Firebase application from development to production using Rust, Python, and GitLab CI. Initially written in Python, the application was migrated to Rust, a systems programming language developed by Mozilla aimed at enhancing memory usage without sacrificing performance. Despite his dedication to learning Rust, Mario encountered obstacles due to the lack of Rust alternatives for certain libraries, such as CairoSVG and Firebase, which were crucial for his project. To address this, he combined Rust with Python, using Python libraries to perform tasks not yet supported by Rust. He demonstrated how to configure the environment, integrate Rust and Python, and deploy the application using GitLab CI, highlighting the importance of comprehensive documentation, especially for non-native English speakers, to improve accessibility and prevent gatekeeping in the tech community.

GitLab's blog post highlights their ongoing efforts to enhance the Protect stage of their DevOps lifecycle application, aiming to foster collaboration between security professionals and developers. The Protect UX team is focused on offering features that help secure applications and cloud infrastructure by identifying and managing threats, vulnerabilities, and risks. Planned enhancements for 2020 include Runtime Application Self Protection and Threat Detection, among others, supported by extensive UX research. Despite challenges in recruiting participants for studies related to these newer security roles, GitLab emphasizes its commitment to evidence-based design, encouraging users with relevant experience to join their research program, GitLab First Look, to contribute valuable insights.

Teams aiming to enhance operations and development are increasingly adopting cloud native architectures, characterized by microservices and containerization, to boost efficiency and accelerate product delivery. Cloud native applications, built with microservices, offer scalability and resilience by allowing components to be managed and scaled independently, often through container orchestration tools like Kubernetes. GitLab supports these architectures with its Kubernetes integration, built-in container registry, and advanced CI/CD features, providing a cohesive platform for microservices and monorepo projects. GitLab's approach simplifies software development by offering native Kubernetes capabilities, feature flagging, and multicloud workflow portability, eliminating the need for intricate scripts or third-party tools, thus positioning it as a leader in the cloud native ecosystem.

GitLab is addressing hiring practices for specific technical support roles due to increased customer demands for tighter security controls over data hosted on their SaaS platform, GitLab.com. This discussion does not stem from any security incident but reflects the company's commitment to transparency and security compliance, particularly for roles requiring administrator access to sensitive customer data. While the proposal suggests restricting certain roles from being based in China and Russia, GitLab continues to hire internationally, including in these countries, and reassures that no current employees will be affected. The aim is to align with customer requirements for data residency and access, without affecting GitLab's self-managed products or contributions to its open-source code. The discussion, conducted publicly in line with GitLab's transparency values, highlights the challenges of balancing open corporate dialogue with diverse stakeholder opinions while maintaining a commitment to inclusivity and legal compliance.

GitLab's integration with Sourcegraph, set to be released with GitLab's 12.5 update on November 22, 2019, aims to enhance developer productivity by embedding advanced code navigation and cross-referencing capabilities directly into the GitLab platform. This collaboration eliminates the need for the Sourcegraph browser extension by integrating its features, such as 'go-to-definition' and 'find references', into GitLab's code views, file views, merge requests, and code diffs. The integration allows developers to efficiently navigate and gather more information about the code without losing context, thus improving code review quality and efficiency. Initially, the rollout will be tested within GitLab's own gitlab-org group, with plans to extend the functionality to all public projects on GitLab.com, while private projects will still require a configured private Sourcegraph instance. The collaboration highlights GitLab's commitment to transparency and iterative development, encouraging community feedback and participation.

GitLab.com, a large high-availability instance of GitLab, faced a critical incident when an expired TLS certificate for its database servers threatened to disrupt operations across its 271 production servers, serving four million users and 12 million projects. The incident arose because a test certificate, initially meant for a proof-of-concept installation, had been inadvertently used in production without proper transition planning. The globally distributed infrastructure team, consisting of 20 to 24 engineers, tackled the issue by disabling certificate validation temporarily, allowing time to implement a more robust, long-term solution without causing downtime. They devised a plan to restart all Consul services simultaneously using the Linux 'at' command, ensuring encrypted connections could re-establish without disruption. Despite the complexity and risk of the situation, the team successfully navigated the incident by leveraging both modern tools and traditional system administration techniques, ultimately maintaining service continuity and demonstrating the importance of thorough planning and a deep understanding of system operations.

GitLab employs a streamlined design handoff workflow using open-source tools like Sketch Measure and GitLab Pages to enhance collaboration between designers and developers, ultimately improving efficiency and communication. The process begins with creating design specs from Sketch using the Sketch Measure plugin, which outputs an HTML page with detailed design information, allowing developers to access all necessary visual and functional instructions without needing to use Sketch directly. These design specs are then committed to GitLab, where GitLab's continuous integration engine triggers GitLab Pages to publish a static website, providing an up-to-date URL for easy access and sharing. This workflow not only saves time and costs by eliminating manual tasks and Sketch license fees but also integrates seamlessly into the broader GitLab ecosystem, offering version control, transparency through commit messages, and the ability to revert to previous design versions. This approach is part of GitLab's move towards DesignOps, focusing on tooling, automation, process standardization, and collaboration to support high-performance design teams.

Shawn Sichak, a security engineer at GitLab since October 2018, discusses the challenges of balancing user convenience with security measures in his role. He emphasizes the importance of integrating automation into security processes to enhance visibility and scalability while minimizing friction for users. Sichak highlights the value of unique passwords, password managers, and two-factor authentication as basic yet often overlooked security practices. He champions transparency in security, which encourages community collaboration and strengthens defenses. Sichak also notes the growing trend of incorporating security into software development workflows, which has led to innovative tools and methods. He expresses interest in security for Industrial Control Systems and reflects on his career path from software engineering to security. Additionally, he shares personal preferences in a light-hearted manner, revealing his fondness for VIM, a desire for the superpower of always picking the fastest checkout line, and a preference for local pancake shops.

Remote work demands a unique combination of self-discipline and mental resilience to maintain productivity, with key strategies including creating a dedicated, comfortable workspace equipped with reliable tools and technology, such as a good webcam, noise-cancelling microphone, and a high-quality internet connection. Establishing clear boundaries between work and home life is crucial, which can be achieved by using different devices or software for work and personal tasks, and maintaining a professional mindset similar to that of a traditional office environment. To avoid distractions, it's important to communicate expectations to family and friends, keep personal media off work devices, and use the internet responsibly. Staying physically and mentally healthy is also essential, with recommendations to take breaks, exercise, and engage in outdoor activities to prevent cabin fever. Ultimately, remote work provides freedoms not found in conventional offices, but requires a structured approach to remain effective and productive.

As organizations increasingly adopt cloud-first strategies, optimizing cloud architectures through a multi-cloud approach has become a priority, with an estimated one-third of IT spending dedicated to cloud computing infrastructure. Multi-cloud strategy involves using multiple cloud providers to meet varying technical and business requirements, enhancing flexibility, resilience, and negotiation leverage while avoiding vendor lock-in. Despite the competition among cloud providers like AWS, GCP, and Azure, which offer numerous services to retain customers, multi-cloud enables workflow portability and the use of the most suitable tools for specific tasks. DevOps processes, particularly Continuous Integration/Continuous Deployment (CI/CD), play a critical role in supporting multi-cloud deployments, with tools like GitLab providing a cloud-agnostic platform that allows consistent productivity and governance across different clouds. The acquisition of GitHub by Microsoft highlights the strategic importance of developer tools in the cloud ecosystem, but GitLab's independence from specific cloud providers presents an alternative for businesses seeking to maintain flexibility in their cloud strategies.

adSoul, a data-driven online marketing startup from Germany, streamlined its processes by migrating from Jenkins to GitLab CI, primarily to enhance stability, reduce maintenance, and improve visibility. Philipp Westphalen, a fullstack developer at adSoul, shared the three-phase migration journey at GitLab Commit London. The phases included moving the repository, migrating the CI/CD pipeline, and refining the CI/CD processes, which involved overcoming challenges like issue migration and adapting scripts for GitLab API. The team emphasized a strategic, step-by-step approach, involving all team members to ensure a seamless transition. They successfully improved build times and project management by adopting Gradle, parallel job processing, and standard Docker images, while maintaining a user-friendly pipeline environment. The transition to GitLab offered adSoul a highly customizable CI/CD platform that mitigated interdependencies between branches, allowing for exploratory changes without impacting the main job, and significantly reduced maintenance efforts, which was crucial for their small team.

The text explores the concept of data organization maturity, contrasting the widespread focus on machine learning and artificial intelligence with the often underdeveloped state of most data teams, which primarily engage in basic analyses. It emphasizes that a mature data organization is fundamentally a mature analytics organization, progressing through three tiers of data analysis: reporting, insights, and predictions. The historical context highlights the evolution from data scarcity and centralized data gatekeeping to today’s democratization of data through modern tools and practices. The text argues for the importance of empowering organizations to self-serve data reporting, allowing data teams to focus on deriving insights and predictions, which provide more significant business value. It stresses the need for proper staffing, appropriate tools, and adopting modern technologies and processes to enable data teams to contribute effectively. Additionally, the use of open-source analytics and best practices from software engineering, such as DataOps and leveraging tools like dbt, can enhance data team efficiency and speed to value.

Several startups recently sought advice from Sid, the CEO and cofounder of GitLab, prompting him to share key insights broadly. He emphasized the critical importance of focusing on either growing usage or revenue, suggesting startups aim for a 10% week-over-week growth to avoid failure. Sid advised that getting users to fully switch to a startup's product, rather than just integrating parts of it into their existing workflows, enhances user feedback and product stickiness. He also highlighted the value of iteration, recommending that startups focus on building minimal viable solutions quickly, ideally within a week, to address big problems without being paralyzed by them. Additionally, Sid encouraged startups to embrace actions that don't scale, as these can provide crucial early growth opportunities by directly engaging with users, referencing examples from Airbnb to illustrate this strategy.

GitLab emphasizes transparency and fairness in its approach to evaluating and comparing DevOps tools, including its own position within the competitive landscape. By openly listing and comparing other DevOps tools on its website, GitLab aims to assist teams in making informed decisions while identifying areas for its own improvement. The GitLab Maturity Framework guides these comparisons by outlining the stages, categories, and features of the DevOps lifecycle, with GitLab excelling in areas such as source code management, code review, and continuous integration. Despite challenges in ensuring accurate and up-to-date comparisons due to varying vendor information and release schedules, GitLab maintains a structured process for information gathering and encourages contributions from both internal teams and external vendors to enhance the accuracy and richness of its tools landscape pages. The company strives to make comparison pages user-friendly by presenting both detailed feature-level comparisons and summary analyses for clarity. Ultimately, GitLab seeks to simplify the DevOps experience by offering a comprehensive platform that reduces the complexity of managing multiple tools.