Company
Date Published
Author
Batel Zohar and Noam Shemesh
Word count
866
Language
English
Hacker News points
None

Summary

JFrog Xray, a tool designed for DevSecOps teams to identify security vulnerabilities and ensure license compliance in software applications, faced challenges with its database synchronization process, initially taking up to 16 hours. To address this, the team implemented a combination of solutions, including distributing database tables, using online servers for component information, and adopting a hybrid approach with a central database. Investigation using the pprof tool revealed inefficiencies in database operations, leading to improvements such as batching inserts, queuing for specific durations, and aggregating data in Go channels. These changes resulted in a significant performance boost, reducing the sync time to about 2 hours. For the initial database sync, an optimistic data insertion approach was adopted, while daily syncs were optimized with parallel processing using workers and bulk loading data followed by index creation. These enhancements collectively improved JFrog Xray's performance, aligning with best practices for efficient database management.