30 days to improve our crawler performance by 50 percent | Algolia

The authors analyzed and optimized the internal workings of their high-performance application, which uses a parallel and distributed computing architecture to crawl websites. They identified several bottlenecks and areas for improvement in their NodeJS and Typescript back-end, RabbitMQ queues, and Kubernetes cluster. Key optimizations included using short-lived queues with TTLs, improving DNS resolution times, and reducing image sizes through multi-stage Docker builds. Additionally, they optimized the front-end bundle by enabling tree shaking and compression with Webpack. The authors were able to achieve significant performance improvements, including a 99% reduction in CPU usage, faster indexing times, and reduced costs. Their experience highlights the importance of ongoing optimization and monitoring in maintaining high-performance applications.