How we built ngrok's data platform

Ngrok's data platform is managed by a small team, primarily driven by a single data engineer, who shares insights into building and maintaining a data lake while addressing privacy concerns and leveraging open-source tools. The data stored includes customer information from globally distributed Postgres instances, usage metrics, subscription and payment details, and third-party data, with strong access controls ensuring privacy. The data engineering role at ngrok is integrated across the engineering organization, focusing on backend engineering tasks and allowing subject matter experts to handle data modeling. The architecture has evolved from a utilitarian, AWS-heavy setup to a more open-source-oriented platform using tools like Apache Flink, Kafka, Dagster, and dbt for efficient data processing, streaming, and analytics. The team has addressed challenges such as schema management and integration within a Go monorepo, while also developing solutions for fighting abuse by analyzing metadata signals. The article highlights the benefits of their data platform and invites interested engineers to explore or contribute to ngrok's data engineering efforts.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.