Petsem: Sometimes Dead Is Better
Blog post from Fly.io
As AI-related credentials become increasingly attractive targets, Fly.io describes how it replaced HashiCorp Vault as the primary store for customer application secrets after its single-region, Raft-based setup caused slow reads and made app startup depend on one region. Its in-house system, Petsem, keeps a write master in one region and distributes read replicas across others, reducing p99 secret-read latency from about 10 seconds to 300 milliseconds while accepting that reads may briefly return stale values. Petsem also uses macaroon-based authorization: a worker must present both its own credentials and a separate token confirming it is authorized to access a particular app’s secrets, limiting the impact of a compromised worker and logging denied requests for alerts. The migration began in 2024 and was completed for customer secret reads in July 2025, though Vault remains in use elsewhere in Fly.io’s infrastructure. Petsem cannot protect secrets already exposed in a compromised host’s memory, but it can constrain access to secrets for apps running there; Fly KMS further reduces exposure by allowing apps to perform operations such as signing or encryption without receiving the underlying secret.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| Secrets Management | 33 | 2,876 | 539 | 148 | -7% |
| LLM | 1 | 3,674 | 823 | 199 | -44% |
| Observability | 1 | 2,390 | 523 | 156 | -23% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.