Company
Date Published
Author
Molly Struve
Word count
1239
Language
-
Hacker News points
None

Summary

Kenna Security, a company that helps Fortune 500 clients manage cybersecurity risks by analyzing data on assets and vulnerabilities, has significantly improved its Elasticsearch indexing capabilities to handle the substantial growth in data volume. Initially struggling with a cluster that was unstable and inefficient, Kenna's cluster now holds 4 billion documents and processes over 200 million daily. Key techniques that facilitated this transformation include adjusting the refresh interval to optimize resource allocation, implementing bulk processing to efficiently handle large batches of documents, and using routing to reduce the number of threads needed for indexing by grouping documents by shard. These strategies have enabled Kenna to maintain efficient indexing and search capabilities as their data demands increase. The improvements came with careful planning and adaptation, preparing Kenna for continued scaling, and ensuring their clients can effectively organize and access their data. Molly Struve, a Senior Site Reliability Engineer at Kenna, played a pivotal role in this successful scaling process over her three years with the company.