Zillow: Optimistic Concurrency with Write-Time Timestamps

Dan Podhola, a Principal Software Engineer at Zillow, presented at the ScyllaDB Summit on managing optimistic concurrency with write-time timestamps to address the challenge of processing out-of-order messages in their high-throughput backend systems. Zillow's system handles property and listing data, translating it into a common format for internal communication, while ensuring the most current information is always processed. The team encountered issues with out-of-order data due to concurrent message queues, which could lead to inaccuracies and require manual intervention. To resolve this, they implemented a method using ScyllaDB's "USING TIMESTAMP" feature, which allows them to ensure only the newest data is written without relying on traditional SQL locking or Lightweight Transactions. This strategy enhances scalability and performance, allowing Zillow to process over 6,500 records per second without impacting real-time workloads, ensuring efficient data handling across their systems.