April 2021 Summaries

Dependabot Preview, which has facilitated over seven million pull requests for more than 30,000 organizations, is being phased out in favor of GitHub-native Dependabot, following its integration into GitHub after Dependabot's acquisition in May 2019. As of now, Dependabot Preview no longer accepts new customers and will cease operations on August 3, 2021, requiring users to upgrade to GitHub Dependabot to continue receiving package update pull requests. This transition includes a new configuration process via a file similar to the previous version, with added improvements, and users can access update logs through their repository's Insights page. While most features from Dependabot Preview are now available in GitHub Dependabot, some, like live updates and PHP environment variable registries, are absent, though alternative solutions such as GitHub Actions are suggested. Auto-merge is also not supported, with recommendations to use third-party apps or GitHub Actions instead. The switch to GitHub Dependabot underscores the importance of keeping dependencies updated for software security, with future enhancements planned for GitHub Enterprise Server support and improved notifications. Users are encouraged to contact GitHub Support for any migration assistance and can explore further information through the provided documentation and roadmap.

GitHub is seeking feedback on its policy regarding security research, malware, and exploits to establish clearer terms for collaboration on its platform, focusing on the distinction between harmful content and code that supports security research. The proposed updates aim to clarify the use of terms like “exploit,” “malware,” and “delivery” to enhance understanding and expectations for the community, encouraging open communication and collaboration in security research. GitHub invites public comments on these clarifications in its site-policy repository, providing a 30-day period for community stakeholders to contribute their insights. Mike Hanley, GitHub’s Chief Security Officer, emphasizes the importance of an open and impartial platform for security research, with the goal of making GitHub a home for security researchers.

GitHub has successfully addressed the challenge of maintaining and repacking some of the largest and fastest-growing Git repositories by implementing a new strategy that allows for faster repacking and improved performance. Traditionally, GitHub's maintenance job involved repacking entire repositories into a single packfile, which was costly in terms of time and resources, especially for large repositories. To overcome this, GitHub developed solutions that include multi-pack indexes and multi-pack bitmaps, which allow for efficient object lookups across multiple packs and support reachability bitmaps beyond a single pack. This new approach involves a geometric repacking strategy that distributes objects across multiple packfiles, focusing on recently added objects, thereby optimizing repack times and reducing the frequency of full repository repacks. The changes lead to significant reductions in CPU time and repack duration, and the improvements are being contributed to the open-source Git project for future releases.

The GitHub Docs team utilizes GitHub Actions to automate project management tasks, streamlining the process of managing open-source documentation by automating repetitive tasks such as triaging issues, checking for spam, and managing pull requests. They leverage GitHub's REST and GraphQL APIs to access and organize project data efficiently, using prebuilt actions and custom scripts to enhance productivity. These workflows, such as assigning labels, transferring issues, and monitoring pull requests, allow the team to focus more on content improvement and less on manual management, thus enhancing their open-source contributions. The YAML format enables all team members to contribute to these workflows, democratizing the process beyond just engineers. The team encourages community involvement by welcoming feedback and contributions to further refine their processes.

GitHub Desktop 2.8 introduces new features to enhance the user experience, particularly for working with diffs and managing multiple copies of the same repository. Users can now expand diffs to gain more context around changes, either by clicking to expand specific sections or using a context menu to view the entire file. Additionally, the option to hide whitespace changes in diffs allows users to focus on significant code alterations. This feature was primarily developed by community contributor Steven Yeh, reflecting GitHub Desktop's strong community involvement. For developers managing multiple repository copies, local aliases can be created to differentiate them easily. These updates are influenced by user feedback from GitHub Desktop's vibrant community, which boasts over one million active users and 200 contributors, highlighting the platform's commitment to continual improvement based on user engagement.

GitHub employs feature flags as a strategic tool to enhance deployment efficiency and mitigate risks, allowing them to introduce new features and improvements while maintaining service continuity. By isolating potentially risky changes behind feature flags, they can enable or disable these features swiftly, ensuring minimal disruption in case of issues. This approach not only reduces deployment risks but also facilitates the incremental development of new features, enabling staff to work on small, manageable batches and avoid long-lived feature branches. The use of feature flags allows for thorough testing in various environments and supports different shipping strategies, such as enabling features for specific user groups or a percentage of users, which helps gather feedback and validate implementations before full-scale deployment. Despite the additional planning, coordination, and technical debt associated with this approach, GitHub finds that the advantages, including easier code reviews and faster, more confident shipping, significantly outweigh the costs.

GitHub is committed to fostering an environmentally sustainable platform for developers, inspired by Microsoft's climate change initiatives, with goals like achieving carbon neutrality since 2019, transitioning to 100% renewable energy by 2025, and becoming carbon negative by 2030. The company emphasizes the importance of a remote-first culture, ensuring the carbon footprint of remote and office-based work is accounted for and offset. GitHub is also pursuing global circularity and zero-waste operations by designing protocols to repurpose IT infrastructure and achieving zero-waste office certification. Water sustainability is another focus, with plans to replenish water used in operations by 2030 and incorporating water-efficient practices into office spaces. Through these efforts, GitHub aims to empower developers to make sustainability-conscious decisions in software development, advocating for community collaboration to embed green practices into everyday workflows.

GitHub Actions is a versatile continuous integration and continuous deployment (CI/CD) service that enhances developer productivity by automating software workflows, but its growth has also attracted abuse, particularly in the form of cryptomining. To address this, GitHub has implemented new measures focused on reputation assessment and manual approval of pull requests from first-time contributors to protect maintainers from being unfairly flagged due to malicious activities. The changes require manual approval from a repository collaborator with write access before any Actions workflows can run for first-time contributors, ensuring more control over potential abuse. These updates aim to maintain the platform's trustworthiness and safeguard the GitHub community from abuse, with plans for additional settings to improve flexibility based on user feedback.

GitHub has revamped its Customer Terms for the enterprise community to enhance user experience by significantly simplifying and shortening the agreement, reducing the page count by over 70%. The new terms are designed to be more accessible and user-friendly, consolidating product-specific terms into addenda and aligning the Data Protection Agreement with Microsoft's where possible. This update allows Microsoft customers to apply existing terms seamlessly to GitHub purchases, and GitHub Enterprise or Microsoft customers can now view all enterprise terms on a single legal webpage. Additionally, GitHub has introduced a streamlined Partner Agreement, offering partners flexible terms through a straightforward one-page enrollment process, aiming to build trust and facilitate a seamless onboarding experience.

This morning marked a historic moment as the Mars Helicopter, Ingenuity, successfully took flight in the thin Martian atmosphere, thanks to the efforts of NASA, the Jet Propulsion Lab (JPL), and a global community of open source developers. Nearly 12,000 contributors on GitHub unknowingly played a role in this achievement by developing software used by Ingenuity, including components from projects like SciPy, Linux, and F Prime. To acknowledge their contributions, a Mars 2020 Helicopter Mission badge has been added to the profiles of these developers on GitHub. This initiative is part of a broader effort to recognize open source contributions, which now includes an Achievements section on GitHub profiles, featuring badges like the Arctic Code Vault badge and one for supporting open source work through GitHub Sponsors. Nat Friedman highlights the collaborative success of this mission, emphasizing the significance of open source development in enabling groundbreaking advancements in space exploration.

The GitHub CLI tool, version 1.9.0, enhances developers' abilities to manage GitHub Actions directly from the terminal, introducing new commands like `gh run` and `gh workflow` for more effective workflow management. These commands allow users to gain insights into workflow runs and their files, providing an overview of workflow executions, and enabling detailed analysis of individual run steps and logs to diagnose failures. Additionally, the tool facilitates artifact management and allows users to rerun failed workflows without leaving the terminal. Workflow files, defined in YAML format, can be viewed and managed more easily using commands like `gh workflow view`, which help track recent runs and enable or disable workflows. The CLI also supports triggering workflows on command with `workflow_dispatch` events, streamlining workflow testing and development. This integration aims to improve the efficiency of managing GitHub Actions, encouraging feedback from users to further refine the tool.

GitHub Actions offer a robust platform for automating software development workflows, with a focus on secure and efficient secret management to control access to external resources. Secrets can be stored at different levels within GitHub, depending on their scope and intended use, and should follow principles of least privilege to ensure access is restricted to only necessary parties. A company like MonaCorp exemplifies best practices by using environment protection rules, CODEOWNERS, and branch protection rules to safeguard secrets, ensuring they are only used by authorized workflows on designated branches and environments. Integration with centralized secret stores like HashiCorp Vault or Azure Key Vault is also possible, allowing organizations to manage secrets without redundancy while maintaining tight access controls. Secrets can be automatically rotated in Vault, and GitHub's capabilities like masking and REST API facilitate secure secret updates, providing comprehensive tools for secure authentication management in workflows.

February's open-source project releases featured a diverse range of exciting updates and new versions, showcasing the relentless innovation within the community. Highlights include the substantial new version of VS Code, which now supports Apple Silicon and offers enhanced features like improved word navigation and personalized icon themes. Rust 1.51 introduced significant improvements, including better compile times on macOS and new APIs, while Gatsby 3.0 boasted faster web experiences with optimized loading and responsive images. Crystal 1.0 emerged as a Ruby-like programming language with C integration, and Ghost 4.0 enhanced its publishing platform with a new dashboard for managing content and subscriptions. Additionally, Sentry Python 1.0 was released to help developers optimize Python performance, and PixiJS 6.0 continued to facilitate smooth 2D rendering for WebGL content. Other notable releases included HTTP Prompt 2.0, posh-git 1.0, and Bitmapflow 1.0, each bringing unique enhancements to their respective fields.

The global internet has thrived on legal safe harbors that protect online platforms from liability for user-generated content, fostering innovation but now facing threats from increased regulation aimed at curbing objectionable content. These regulations often target general social media platforms but risk negatively impacting software development platforms, which have a lower risk profile for such content. Narrowing these safe harbors could limit access to software code, undermining developers' rights and the broader software ecosystem. Developers are encouraged to engage with policymakers to ensure regulations are informed by technical perspectives, advocating for solutions that respect human rights and do not disproportionately affect software collaboration. Various countries, including Australia, Brazil, the EU, and the U.S., are considering or have implemented reforms, and developers' input is crucial in shaping laws that balance safety with innovation.

In March, GitHub experienced three significant incidents affecting several services, including Actions, due to issues like exceeding Int32 limits, database migration errors, and an authentication provider outage. On March 1, a high error rate was caused by check suite IDs exceeding Int32, affecting job queuing and search functionality; a code fix was deployed to address this, and audits of external libraries were conducted to prevent recurrence. The incident on March 12 resulted from a database index change leading to full table scans and service timeouts, prompting plans for better tooling and index inventory management. On March 15, an authentication provider outage hindered hosted job processing, which was resolved once the service was restored, leading to considerations for extending authentication token lifetimes. GitHub emphasizes its commitment to service reliability and continuous infrastructure and tooling investment to enhance user experience, as detailed on their engineering blog.

The Supreme Court's decision in Google v. Oracle marked a significant milestone for developers, affirming that the ability to transfer code and skills across platforms is a critical interest that deserves protection under copyright law. The ruling determined that Google's use of Oracle's Java API constituted "fair use," highlighting the importance of developer interests in the context of copyright. Recognizing that developers invest substantial time and effort in learning platform APIs, the Court's decision prevents API owners from gaining excessive control over developers once they select a particular stack. By classifying APIs as further from the "core" of copyright, the decision acknowledges that the value of an API partly derives from the programmers' efforts to learn and utilize it, thus promoting innovation and creativity. Additionally, the Court emphasized that allowing API owners to restrict reimplementations could harm public interest by stifling creativity and innovation, as the costs and challenges of developing alternative APIs might lead to monopolistic control over interfaces. This landmark decision underscores the recognition of developer interests and is expected to enhance developers' freedom to innovate in the future.

The Global Maintainer Summit, hosted by GitHub on June 8-9, is a free virtual event designed for open source maintainers to discuss the unique challenges and rewards of project maintenance. This inaugural summit will feature short streaming talks and opportunities for networking through platforms like gathertown, allowing maintainers to share strategies for managing notifications, utilizing automation, and balancing work with personal time. Participants are encouraged to share experiences through a Call for Proposals, with an emphasis on honest discussions about problem-solving in open source projects. The summit is part of the broader Maintainer Week, a collaborative effort involving various organizations to highlight the vital role of maintainers in the open source ecosystem. The event aims to foster a community where best practices and mental models are shared, helping to alleviate individual burdens and enhance the open source community.

About a year ago, a migration was executed to replace an old rate limiter with a more resilient system to handle increased traffic, leading to the adoption of a Redis backend with client-side sharding. The previous Memcached-based system faced issues due to changes in architecture and data persistence, prompting the switch to Redis for its better persistence, sharding, and replication capabilities. The new design included using Redis for expiration instead of database storage and implementing storage logic in Lua for atomicity, while avoiding MySQL to prevent extra write traffic. The transition involved isolating persistence logic into a new RedisBackend class and using feature flags to gradually roll out the change. However, post-release, bugs were reported regarding the "wobbling" of the X-RateLimit-Reset header and inconsistent rejection responses due to Redis's replica lag in data expiration. These issues were addressed by managing "reset at" times in application code and ensuring responses were prepared using consistent data from Redis. Ultimately, the new rate limiter improved system reliability and reduced support load, although it highlighted potential improvements for handling request increments more effectively.

GitHub has introduced new authentication token formats designed to enhance security and improve the detection of compromised tokens. The updated tokens feature identifiable three-letter prefixes and a separator to distinguish them clearly, reducing false positives in secret scanning to 0.5%. A checksum has been added to virtually eliminate false positives, using a CRC32 algorithm encoded with Base62. The new token formats also maintain or increase token entropy, ensuring a high level of uniqueness without altering token length. GitHub encourages users to reset personal access and OAuth tokens to benefit from these security improvements. Additionally, service providers issuing tokens are advised to adopt these practices and participate in GitHub's secret scanning program to enhance their security measures.