Company
Date Published
Author
Carl Sverre
Word count
1708
Language
English
Hacker News points
2

Summary

SingleStore and Redpanda are used together to solve the operational complexity of global logistics by scaling worldwide parcel shipping to unprecedented volumes. This is achieved through a reference architecture that combines SingleStore's relational database with Redpanda's streaming platform, allowing for efficient processing and storage of large datasets. The system uses a scale-out simulation cluster to generate thousands of packages, emits events into Redpanda topics, and loads them into SingleStore using Pipelines in batches of up to 100 million records. A data model is used to efficiently process and store the data, including two columnstore tables for raw package and transition records, a rowstore table for undelivered packages, and a reference table for locations. The system also includes a stored procedure to keep track of transitions and states in sync with the latest data loaded from Redpanda topics. Data visualization is achieved through Grafana dashboards that showcase performance statistics and actual data, including maps showcasing real-time inventory and current package destinations. The system has been deployed on Google Cloud and achieved a raw computational performance of almost 400k deliveries per second, exceeding the goal of handling 100x the volume of global package delivery per year.