February 2022 Summaries

We have implemented a development workflow that utilizes GitHub Actions to automate tasks such as testing, code quality, and compliance, while also maintaining a strong emphasis on code reviews. Our pull request and merge workflow revolves around automating as much of the process as possible, with a focus on quality and velocity. We use GitHub Actions to run tools for testing and code quality, and have implemented custom actions to validate SpiceDB schemas using test data. Our code review process involves creating a draft PR that allows authors to push intermediate changes to get feedback from automated actions, and reviewers can check out the PR branch and run it in their local workspace before reviewing the code. We maintain a 24-hour turnaround time for reviews and encourage anyone (usually the reviewer) to merge an approved PR once complete. Our merge workflow aims to strike a balance between completeness of checks without requiring too much developer intervention, but we've encountered issues with stale PR approvals and unnecessary re-releases, leading us to seek recommendations for a merge queue solution to improve our process.

On February 16th, 2022, Authzed noticed consistently higher API latency with occasional spikes into the hundreds of milliseconds due to new metrics features in their billing section. The root cause was determined to be always present and made observable once fully exercised with the introduction of the new functionality. After a full deployment rollback, the service returned to its previous stability, but the root cause was still identified as being related to caching issues in SpiceDB, specifically the thundering herd problem caused by traffic patterns. The issue was resolved by optimizing SpiceDB to be more conscious of its impact on the Go scheduler and increasing the number of cores available to SpiceDB. Following the resolution, overall API latency improved 27%.

SpiceDB is Authzed's open source, Zanzibar-inspired permissions system that operates on a relationship-based approach, allowing for more complex permission resolution. It stores relationships between resources and subjects in its data store, which provides faster performance and consistency compared to other permissions systems. The system can be updated through various methods, including 2PC-like approaches, streaming commits, and asynchronous updates, each with their own advantages and trade-offs. Additionally, SpiceDB allows for more flexible schema designs, enabling applications to not store permissions-related relationships in their relational databases, which can simplify permissions management without compromising performance or consistency.

The SpiceDB team has released a new feature for wildcards in December and improved API performance transparency by landing major functionality in both Authzed.com and Authzed Enterprise. They also documented their experience with the new feature on their blog and provided product feedback to the GitHub team. The rest of the month focused on security updates, including fixing CVE-2022-21646 in SpiceDB v1.4.0 and introducing revision-aware schemas. Zed v0.3.1 shipped with support for logging request complexity and fixed Alpine Linux builds. Authzed.com users can now delete development permission systems and have access to an alpha HTTP Gateway. In Authzed Enterprise, OIDC is now available for authentication, and load testing can be performed on installations configured without TLS. The team also recorded a video modeling familiar apps, including Google Groups, and published it on their blog.

SpiceDB is a database that stores permissions information, which must be updated as an application evolves. Unlike general-purpose databases, SpiceDB allows for permission calculations to be written and checked without migrations, making certain types of migrations simpler or even unnecessary. This is achieved through its schema design, where relationships are written but permissions are computed over them, allowing changes to be made without affecting existing data. However, in cases where migrations are required, SpiceDB provides a set of strategies for safe and efficient migration, including ensuring that new relationships can always be added to an existing schema and removing relations only if there are no references to it. The database also enforces rules to prevent conflicts during migration, such as requiring a TOUCH operation when backfilling relationships.