How We Migrated Our Massive Crashes Database
Blog post from Luciq
Migrating a high-volume database poses significant challenges, as illustrated by Luciq's experience with their crashes database, which serves over two billion devices and includes billions of records. The company faced difficulties with data retention and table partitioning, opting to explore migration tools like Large Hadron Migrator (LHM) and gh-ost to overcome these issues. LHM, although previously effective, proved unsuitable due to its reliance on triggers, leading to latency, lock contention, and deadlocks. In contrast, gh-ost, developed by GitHub, offered a triggerless solution using binary logs for asynchronous migration, reducing the need for locks and accommodating the high insertion rate of Luciq's crashes data. However, gh-ost required custom modifications to address issues like slow migration and lock contention, such as prioritizing copy tasks over event tasks and temporarily dropping indices to enhance performance. This experience underscores the complexity of managing large data migrations and the necessity of customizing tools to meet specific requirements.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.