August 2022 Summaries

Rate limiting on GitHub's API is a technique used to manage the performance and prevent system overload by controlling the number of requests a user can make. Unauthenticated users are limited to 60 requests per hour, while authenticated users can make up to 5000 requests per hour. Enterprise accounts have an increased limit of 15,000 requests per hour. Techniques to optimize the usage of these limits include increasing page size limits, using conditional requests with ETags, managing multiple GitHub accounts, and employing incremental ingestion strategies. Advanced methods such as utilizing GraphQL, token rotation, and GitHub App installations can further enhance data retrieval efficiency. Despite these strategies, users may still encounter rate limits and need to implement waiting strategies for resets. These techniques have been successfully implemented at Endor Labs to improve the speed of data fetching from GitHub, demonstrating significant performance enhancements while occasionally waiting for rate limits to refresh.

After leaving his role as VP Engineering at Citrix, the author joined Endor Labs as Managing Director and Head of the India R&D team, attracted by the startup's focus on software supply chain security and its alignment with industry trends towards "shifting left" in cybersecurity. This shift involves addressing security earlier in the software development process and providing developers with the tools they need to enhance productivity without compromising security. The author highlights the fragmented nature of current solutions and Endor Labs' innovative approach to helping developers produce secure code. With a vibrant developer community in India and an impressive founding team, including CEO Varun Badhwar and CTO Dimitri Stiliadis, the author sees an opportunity to engage with top engineers globally and create developer-centric products. His vision is supported by the potential to build security products that balance both productivity and security, leveraging open-source community interest and covering all areas of product development in India.