August 2019 Summaries

In a recent survey of over 4,000 developers, security professionals, and operations team members, 50% of each group identified software testing as the primary cause of development delays. Mek Stittri, GitLab's director of quality engineering, highlights two main issues: insufficient testing and unreliable test results. GitLab is addressing these issues by encouraging developers to write unit and integration tests and by improving test requirement documentation to better align with business use cases. Stittri also emphasizes the importance of building a robust test framework concurrently with product development to ensure comprehensive coverage and reliability. He advocates for test automation that mimics manual testing strategies, where automation engineers focus on creating stable and efficient tests. Machine learning is seen as a potential future aid in test automation, with GitLab exploring ways to integrate bots for continuous, intelligent testing. The challenge is balancing increased testing with maintaining productivity, as Stittri's team works to optimize test runtimes while ensuring enterprise feature integrity.

GitLab's product team is focused on enhancing DevOps by advancing their monitoring products to provide greater observability into application performance and deployment health. This effort is part of their broader vision to integrate monitoring as a fundamental stage in the DevOps lifecycle, aiming to enable proactive problem-solving and uphold service level objectives (SLOs). The team's priorities include developing robust metrics, cluster monitoring, and incident management tools, with an emphasis on cloud-native technologies and integration with open-source tools like Prometheus. By dogfooding their new incident management system and encouraging GitLab administrators to utilize the same monitoring capabilities as developers, GitLab seeks to refine its tools through practical application and user feedback. The company acknowledges ongoing debates about the requirements for a comprehensive observability suite in the Kubernetes-driven cloud-native landscape and is committed to evolving its offerings based on user needs and industry trends.

GitLab Commit, an upcoming user conference taking place on October 9 in London, promises a dynamic lineup of speakers from top technology organizations worldwide, set to share insights on automation, exploration, and digital transformation. Hosted at The Brewery, an 18th-century venue, the event will feature keynotes such as Porsche's digital transformation journey, Capgemini UK's rapid development with Kubernetes, and Kiwi.com's cloud infrastructure innovations. Attendees will explore tracks on cloud-native technologies and DevOps in action, with practical sessions focusing on zero-cost infrastructure, automatic deployments, and leveraging GitLab for large-scale challenges. The conference aims to provide a user-centric experience, combining inspiring talks with a closing mini-golf party under the moonlight, and offers a registration discount until September 23.

The text explores the use of GitLab CI/CD's built-in Docker registry to enhance efficiency in continuous integration and deployment processes. By creating custom "build" Docker images that include all necessary dependencies, users can avoid the inefficiencies of repeatedly installing tools during each job run. This approach allows for a more streamlined pipeline, particularly when deploying applications like those using Google Firebase. The text emphasizes the benefits of using GitLab's registry to manage and reuse these custom images, thus reducing reliance on potentially unstable or outdated Docker Hub images. It highlights a practical implementation with Firebase, showcasing how to build and deploy using a specialized Docker image that integrates the Firebase CLI, ultimately speeding up the CI/CD pipeline and providing greater control over the build environment.

The story details a complex scaling issue encountered by GitLab.com, where customers reported intermittent errors during Git pulls, primarily due to SSH connections being unexpectedly terminated. The investigation revealed that the problem was linked to a systemic issue dubbed "the tyranny of the clock," where many scheduled jobs initiated Git fetches at predictable times, overwhelming connection limits. Initial diagnostic efforts, which included analyzing packet captures and HAProxy logs, identified that the SSH server was exceeding its MaxStartups limit during these peak times. Solutions involved incrementally increasing MaxStartups and implementing rate limiting and load balancing changes, such as switching to the leastconn strategy in HAProxy, which ultimately reduced error rates. The narrative underscores the importance of accurately measuring error rates early in the troubleshooting process and highlights the need for more nuanced connection management to handle traffic spikes, suggesting potential future improvements by addressing SSH authentication delays and optimizing server resources.

Delta Air Lines undertook the significant challenge of transitioning to cloud-native technologies by adopting a cloud-agnostic approach that avoids the pitfalls of siloed infrastructure and inflexible cloud vendor lock-in. This shift was driven by the need for increased competition, collaboration, and portability, which are crucial for fostering innovation and meeting business objectives. Jasmine James, IT Manager at Delta’s DevOps Center of Excellence, emphasized the importance of operationalizing Kubernetes on bare metal with the right tools before fully transitioning to cloud environments, a strategy aimed at enhancing their technological capabilities and innovation potential. Delta's experience serves as an example for other large organizations facing similar challenges in their cloud adoption journeys, and interested parties are invited to learn more about their approach at GitLab Commit Brooklyn.

GitLab's data team, initially comprising three members, has expanded to seven to meet the growing data demands of the company, which has seen rapid growth. The team emphasizes refining processes and workflows to enhance data-driven decision-making. Onboarding new hires, especially in a remote setup, involves familiarizing them with GitLab's data stack and novel concepts like separate storage and compute in Snowflake. Managing resource consumption efficiently is crucial, akin to conserving household water usage, to prevent operational costs from escalating with headcount. The team has developed a Snowflake spend monitoring package using dbt and Periscope, which they have made available to the public, aligning with their open-source values. This package allows for real-time monitoring of Snowflake costs, helping to manage expenses proactively. The setup includes configuring dbt projects and Periscope dashboards to automate cost tracking and reporting, which is shared daily via Slack, promoting resource management initiatives effectively.

Managing deployments in DevOps environments involves complex challenges, particularly when considering the impact of deployment on budgets and resources. Continuous Verification is essential for achieving efficiency in continuous deployment (CD) processes, but often budget and resource allocation checks are overlooked. These checks help avoid issues such as rollbacks and redeployments, which can negatively impact metrics like mean time to change and deployment time. Implementing these checks can be facilitated using tools like CloudHealth by VMware and GitLab, which allow for the integration of governance rules and scripts into CI/CD pipelines. The text also emphasizes the importance of understanding and implementing policy checks to ensure project budgets and resources are not exceeded. Despite the challenges in convincing DevOps admins to adopt these practices, adding a budgetary efficiency metric could incentivize better deployment efficiency. The authors, Bahubali Shetti and Tim Davis, bring a wealth of experience from their roles in VMware and the broader cloud solutions industry, advocating for improved public cloud operations and the adoption of cloud-native applications.

As 2019 comes to a close, the text explores how microservices can enhance team management and delivery speed, focusing on Agile development within GitLab's framework. Microservices are defined as independently deployable services that efficiently manage specific functions, as articulated by Martin Fowler. The discussion covers traditional and modern organizational structures, advocating for a shift from system-based to product area-based organization using GitLab Groups and Projects, which enhances visibility and collaboration. This method supports Agile practices by allowing parallel milestones and open contributions across different teams, while a hybrid approach combining product and system structures is suggested for organizations with cross-platform teams, ensuring flexibility and efficiency.

GitLab is transitioning to a single Rails codebase for its Community Edition (CE) and Enterprise Edition (EE) to simplify its development process and reduce complexity. Previously, separate repositories were used to distinguish proprietary code from free software, but this approach led to challenges in coordination, such as managing numerous merge requests and manual interventions during security releases. To address this, GitLab is consolidating its code into a single repository, moving development into the gitlab-ee repository, which will be renamed "gitlab," with a separate "gitlab-foss" repository serving as a public version without proprietary code. This approach retains the Git logs of both CE and EE, facilitates issue tracking, and maintains community contributions in a unified repository. The transition involves significant code refactoring, with nearly 1.5 million lines of code changed by 55 engineers across 600 merge requests. The new single codebase is expected to be implemented by the release of GitLab 12.3, with minimal disruption for users of Omnibus packages and Docker images.

The narrative provides insights into the author's internship experience at GitLab, emphasizing the importance of balancing asynchronous and synchronous communication, the role of mentorship, and the necessity of schedules in a remote work environment. The author highlights the value of having an in-person mentor, Emilie, who helped bridge the gap between these communication styles and instilled confidence through discipline. Additionally, the author discusses the learning potential of mistakes, emphasizing that they are integral to personal growth and understanding in the tech field. The experience underlined the dangers of burnout and the importance of taking breaks, as the work will remain even if one pauses due to health or personal reasons. Lastly, it reflects on the dynamic nature of data and the need to engage with it actively, comparing GitLab to a living entity that is constantly evolving and teaching the importance of adaptability and continuous improvement.

The blog post delves into GitLab's implementation of Zero Trust Networking (ZTN) as a means to enhance data protection by shifting access control from the organizational perimeter to individual assets and endpoints. GitLab's approach involves a data classification policy that assigns color-coded levels of sensitivity to data, ranging from RED for highly sensitive data to GREEN for public data, to ensure appropriate protection measures. The post highlights several challenges associated with ZTN, including the dynamic nature of data classification, the complexities of tracking data and metadata, and the application of data labels for compliance and access control. Additionally, it addresses the intricacies of managing data in a cloud-based infrastructure where control is shared with multiple SaaS providers, each with varying levels of administrative access. While GitLab benefits from not having a traditional corporate VPN, the company still faces challenges in enforcing access controls across diverse platforms. The blog concludes by acknowledging these unique challenges and hinting at forthcoming strategies to address them in future installments of the series.

A project manager's urgent request at the end of the workweek highlights the frustration many DevOps teams face with slow CI pipelines, a problem Ticketmaster experienced with Jenkins. Slow builds can stall development, leading to increased costs when organizations add more servers to handle peak usage, especially on cloud infrastructures where idle servers still incur expenses. GitLab's introduction of autoscaling runners aims to address this issue by dynamically creating and removing machines based on demand, optimizing resource usage, and reducing costs. Autoscaling allows teams to maintain the right balance of runners, ensuring efficient build processing without over-provisioning and enabling developers to focus on coding. The implementation of autoscaling runners by Substrakt Health, which led to a 90% reduction in EC2 costs, exemplifies the potential savings and efficiency gains. This approach supports the core DevOps goals of speed and efficiency by allowing teams to deploy better software without the infrastructure concerns that traditionally slow down the process.

Pair programming, an Agile approach to software development, involves two programmers working together at the same workstation, with one writing code (the driver) and the other reviewing it (the navigator). This method aims to accelerate Agile delivery by promoting collaboration, knowledge sharing, and overcoming obstacles more efficiently. However, some organizations view it as inefficient, questioning the need for two developers to focus on a single piece of code amidst pressing deadlines and technical debt. Successful pair programming relies on open communication and planning, considering project readiness, role distribution, potential roadblocks, and necessary technology tools. While it offers benefits such as increased morale, innovation, and learning opportunities, particularly for junior developers, it may also feel inefficient if overused or poorly executed. GitLab has embraced pair programming, adapting it to remote work environments and tracking its effectiveness in resolving issues more quickly. The practice encourages communication, knowledge sharing, and breaks down silos within engineering teams, but its success depends on finding the right balance and adapting to individual team dynamics.

The GitLab Unfiltered blog embodies a culture of transparency and inclusivity, allowing any GitLab team member to publish content, as long as it undergoes a peer review process first. This approach encourages contributions from across the team, fostering a diverse range of perspectives and insights.

Compliance as code is a method of integrating compliance requirements into the software development lifecycle, allowing for automated, low-friction compliance that saves time and reduces risk. By embedding compliance into workflows, teams can ensure that regulatory and company requirements are met without manual intervention, which is especially beneficial for large enterprises in highly regulated sectors like healthcare and finance. This approach fosters better alignment between developers and risk assessment teams, streamlines processes, and reduces the likelihood of fines and breaches by maintaining consistent compliance across an organization. However, challenges remain in keeping security up to speed with rapid changes in DevOps, and human oversight is still necessary to mitigate potential errors. Successful implementation involves defining compliance policies upfront and involving key stakeholders from management to infosec, while promoting a culture of security and collaboration. Companies are encouraged to adopt compliance programs for open source software to protect intellectual property and ensure adherence to licenses. GitLab exemplifies early adoption of a formalized compliance program, emphasizing the benefits of aggregate security controls for agile and efficient compliance management. Cloud platforms like Google Cloud, AWS, and Azure offer tools that automate core compliance actions, enhancing prevention, detection, and remediation efforts.

GitLab CEO Sid Sijbrandij discusses the challenges and benefits of running an all-remote startup in a conversation with Maren Kate on the From 5 to 50 podcast, highlighting the skepticism faced from venture capitalists due to concerns around team collaboration, product scalability, and financial metrics in remote settings. Despite initial skepticism, GitLab's commitment to a remote model, even after opening an office to ease Series A financing, underscores the evolving landscape of startup operations. Sijbrandij notes that while remote setups can hinder acquisition prospects and attract lower valuations, they offer significant advantages in hiring and scalability, forcing companies to adopt best practices early. He also points out that traditional co-location models are increasingly problematic due to talent retention issues in competitive hubs like San Francisco, and suggests that remote work can foster innovation by encouraging unconventional ideas. Both Sijbrandij and Kate acknowledge potential unconscious biases favoring in-office setups, with Sijbrandij emphasizing that co-location can stifle creativity by discouraging seemingly implausible yet groundbreaking ideas.

GitLab is a pioneer in all-remote work, fostering a diverse and empowered workforce of over 800 team members across more than 57 countries, making it one of the world's largest all-remote companies. The author, who recently joined GitLab to lead its remote initiatives, emphasizes the transformative potential of all-remote work, which allows individuals to work from locations that matter to them and contributes to community well-being by stemming the tide of depopulation in rural areas. Highlighting the advantages of remote work, such as reducing urban congestion and tapping into global talent pools in emerging tech hubs, the text argues for a shift in work culture, leveraging GitLab's platform to enable this transition. Through personal experiences, the author illustrates the enriched life afforded by remote work and expresses enthusiasm for extending these opportunities globally, inviting others to explore GitLab's approach to remote work and consider joining the journey towards a more liberated work environment.

At BSides Kansas City, discussions centered around integrating fuzzing into the DevOps process, highlighting its potential as a crucial part of the software security lifecycle despite current challenges in adoption due to complexity and friction. The article shares insights into automating fuzzing with American Fuzzy Lop (AFL) using GitLab CI/CD pipelines, providing a practical example of setting up a fuzzing environment with Docker and Ubuntu 16.04. It details the process of initiating and managing AFL within GitLab, including the use of Python scripts for execution control and artifact collection to streamline workflow integration. The author emphasizes the importance of making fuzzing accessible by simplifying its integration into organizational practices and expresses hope for future advancements in GitLab's fuzzing features. Additionally, potential improvements like automating crash issue creation and advanced triage are discussed, alongside the call for community involvement to enhance fuzzing accessibility and effectiveness.

Agile development can significantly enhance team performance and application delivery through best practices such as continuous integration, retrospectives, pairing, iterative development, and burndown charts. Continuous integration involves frequent code integration to detect errors early, while retrospectives provide a platform for teams to discuss successes and areas for improvement, fostering a culture of continuous learning. Pairing allows team members to collaborate closely, leading to reduced bugs and increased innovation. Iterative development encourages small, rapid changes that enhance user outcomes and speed up time to market. Burndown charts, particularly useful in a Scrum framework, help teams and stakeholders visualize sprint progress and manage expectations regarding feature delivery. Together, these practices promote efficiency, adaptability, and customer-centric product development.

DevOps practices have increasingly incorporated security measures, giving rise to the concept of DevSecOps, which emphasizes integrating security into the software development lifecycle (SDLC). Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) are essential methodologies under the Application Security Testing (AST) umbrella, designed to identify vulnerabilities in source code and running applications, respectively. SAST, a white-box testing approach, analyzes code during development to preemptively address vulnerabilities, while DAST, a black-box testing method, examines applications during runtime to simulate real-world attacks. Both SAST and DAST are crucial in forming a comprehensive security strategy, providing automated and continuous testing that aids developers in maintaining secure applications. Modern development environments often integrate these tools into Continuous Integration/Continuous Deployment (CI/CD) pipelines, although manual reviews and a security-first mindset remain indispensable.

GitLab's ongoing blog series on Zero Trust explores the complexities and challenges of implementing Zero Trust Networking (ZTN) within their organization, focusing on shifting access control from organizational perimeters to individual users, devices, and endpoints. The series highlights the difficulty in achieving a cohesive ZTN approach due to varying interpretations and implementations by different vendors, and emphasizes the need for real-time identification and authorization of users, devices, and geolocations to protect data effectively. GitLab's objectives for ZTN include flexible data protection, positive identification of team members and devices, and streamlined security processes that do not hinder productivity. Challenges arise from the lack of a corporate perimeter due to the remote nature of their workforce, the need for consistent security across multiple cloud platforms and services, and the necessity to scale solutions in line with the company's rapid growth while adhering to global regulatory and compliance requirements. Future posts in the series aim to delve deeper into the intricacies of data and infrastructure management within the context of ZTN.

GitLab initiated a significant shift in the DevOps industry by merging GitLab CI and GitLab Version Control into a single application, streamlining the process of code delivery and enhancing efficiency. Initially met with skepticism from GitLab's co-founders, the idea was championed by engineer Kamil Trzciński, who argued that a unified application would simplify operations and improve user experience. The decision to consolidate proved advantageous, allowing for innovations like auto-remediation, which automatically updates vulnerabilities in the codebase. This integration, which began in 2015, set a precedent that was later adopted by other major platforms such as Bitbucket and GitHub, reflecting a broader industry trend towards consolidating tools within the DevOps market. The unified approach not only aligns with GitLab's values of efficiency and innovation but also resonates with customer needs, demonstrating the market's validation of this strategy and encouraging further collaboration and contributions from the community.

The final blog post in a series on working remotely with children discusses the experiences and strategies of GitLab team members in balancing work and family life. It highlights the importance of flexibility in scheduling, as exemplified by parents like Heinrich Lee Yu and Annabel Dunstone Gray, who adapt their work hours to fit their children's routines. The post also stresses the need for discipline and efficiency, with insights from Eric Johnson and Grzegorz Bizon on managing time constraints. It explores the role of relationships in parenting, as shared by Chris Maurer and Micaël Bergeron, emphasizing the significance of mutual support and maintaining personal interests. The importance of setting clear expectations with partners to avoid stress and resentment is also discussed, drawing on experiences from Lyle Kozloff and Karlia Kue. Finally, the blog underscores the joy and fulfillment of spending quality time with children, as noted by Brittany Rohde, and acknowledges that remote work can facilitate better work-life balance despite the inherent challenges of parenting.

GitLab Commit, an inaugural conference held in Brooklyn, will feature a diverse lineup of speakers including industry experts and GitLab team members from organizations such as Delta Air Lines, Goldman Sachs, and T-Mobile. The event, scheduled for September 17th, aims to foster learning and networking through discussions on rapid and secure code delivery and innovation acceleration. Set in the Williamsburg neighborhood, the conference will transform venues within a two-block radius into themed spaces, providing a creatively immersive experience. Keynotes will cover topics like cloud-native transformations, CI/CD pipelines, and open-source culture, with sessions addressing cloud automation, infrastructure deployment, and chaos engineering. Attendees are encouraged to register by August 15th for discounted passes, while the event promises to inspire and empower participants to drive improvements within their teams and the broader tech community.

Jason Yavorska, the product manager for CI/CD at GitLab, discusses the exciting developments in GitLab's CI/CD pipeline features as they progress through the 12.x series of releases. Key enhancements include the introduction of a Directed Acyclic Graph (DAG) for out-of-order execution, enabling more flexible pipeline steps, and the expansion of pipelines for merge requests to work with forks. New features such as Visual Review Apps, parallel execution for merge trains, and automatic HTTPS certificate renewal for GitLab Pages are highlighted. The updates also focus on improving usability with features like collapsible job logs and multi-project pipelines becoming a core feature. Future releases aim to integrate Hashicorp Vault for better secret management, associate milestones with releases for enhanced data connectivity, and bring improved Helm v3 chart management. The GitLab team is committed to refining these features with community input, emphasizing iterative improvements to meet evolving customer needs.

Progressive Delivery is an evolution of continuous delivery that offers more precision in software deployment by incorporating new ideas and best practices to reduce the risks associated with large-scale releases. GitLab views Progressive Delivery as the next step in DevOps, emphasizing the use of Feature Flags to selectively roll out features without altering the source code. Feature Flags, also known as feature toggles, allow developers to deliver incomplete features incrementally and test them before full release by toggling them on or off. This approach minimizes the risk of new feature releases, as developers can roll back changes easily and manage user-specific features. However, Feature Flags can introduce complexity if not properly managed, leading to issues like stale flags. To address these challenges, organizations utilize feature management systems such as Launch Darkly or Optimizely, and GitLab is integrating Progressive Delivery features like Review Apps and Feature Flags into its DevOps platform to improve efficiency and visibility while reducing the need for complex integrations.

As remote work continues to gain popularity globally, tools like video conferencing have become essential for communication and collaboration, especially for all-remote companies like GitLab. While asynchronous communication is often preferred for its inclusivity and documentation benefits, GitLab suggests a video call when a topic requires more than three exchanges. Successful video calls hinge on using personal equipment like headphones to minimize background noise and turning on cameras to enhance interaction, fostering a more connected and inclusive environment. Participants are encouraged to actively contribute to discussions, even if it means interrupting, and meetings should be strictly timed to prevent scheduling disruptions. GitLab promotes these practices to ensure effective remote communication and offers insights into their approach to all-remote work.

Joining GitLab without a technical background is entirely feasible, as evidenced by the author's journey from unfamiliarity with the platform to becoming a contributing team member. The author outlines the importance of issues and merge requests in GitLab's workflow, explaining how issues serve as a starting point for initiatives and collaboration, while merge requests formalize changes. Emphasizing the significance of communication in a remote setup, they highlight GitLab as the primary medium for interaction, with Slack and email as secondary options. The onboarding process, though initially daunting, is structured to guide new employees through essential tasks, including adding themselves to the team page, which can be complex due to the use of .yml files. Video calls via Zoom are recommended for effective communication, with features like "Touch up my appearance" easing concerns about being camera-ready. The author encourages feedback on onboarding experiences to enhance resources for future newcomers.

In the third installment of a series on remote work with children, GitLab team members discuss strategies for managing workspaces at home amidst various family dynamics. The piece highlights insights from a GitLab unconference session led by Lyle Kozloff and Sean McGivern, where participants shared personal experiences ranging from using noise-cancelling equipment to designating specific household areas for work. It underscores the importance of creativity in space management, involving children in planning communication protocols, and setting clear boundaries to maintain productivity. Diverse approaches are shared, such as utilizing a loft as an office or employing signs to indicate if interruptions are allowed. Some team members, like Mike Greiling and Alessio Caiazza, find working from coffee shops or coworking spaces beneficial, emphasizing that remote work solutions vary and should cater to individual and family needs. The series promises additional guidance on topics like time management and relationships in its next installment.