Company
Date Published
Author
Oryan Moshe
Word count
1897
Language
English
Hacker News points
None

Summary

The author of the blog post is upgrading their SingleStore cluster and replicating data from PostgreSQL to achieve high performance for analytical queries while keeping transactional data in PostgreSQL. They have a 1TB cluster, which is not justified, but they need the performance of SingleStore's columnstore for queries. The author uses a replication service to replicate rows from PostgreSQL to SingleStore's rowstore in real-time, allowing them to use the columnstore for analytics and speed up queries. However, updating rows is slow due to the way data is stored in a columnstore. The author tries different solutions, including using ActiveRecord callbacks and replicating the whole table from PostgreSQL once in a while, but these methods are not efficient. They eventually compress the file before loading it into SingleStore, which speeds up the process and makes storage less of an issue. The author creates a Rails module to replicate any query from PostgreSQL to SingleStore easily, including truncating old data and minimizing downtime during replication.