September 2019 Summaries

The text discusses the importance of AWS CloudTrail logs in enhancing security by tracking actions taken by users, roles, or AWS services, both through the AWS console and API operations. It highlights that AWS allows for easy and cost-effective flow log tracking compared to traditional on-premise infrastructures. The article emphasizes the need for companies using AWS to monitor user, admin, and application activities for security purposes and to meet internal or regulatory requirements. It introduces Coralogix as a solution for integrating CloudTrail logs to provide smarter analytics and details various alert types and conditions that can be set up using CloudTrail, such as monitoring root account usage, changes to security groups, and network configurations. The article encourages readers to adapt the discussed methods to their specific use cases and invites them to reach out for support or further information.

Coralogix enhanced its data pipeline by employing Kafka Streams and Kafka Connect to optimize performance and resilience, handling up to tens of billions of messages daily. Initially, their system struggled with non-linear scaling and infrastructure issues due to combining external I/O with in-memory processing. The introduction of Kafka Streams allowed Coralogix to decouple services from external I/O, making them CPU-bound and reducing resource usage by 80%. By using Kafka for data flow, they achieved a push-updated cache, minimizing delays and external side effects, and increased system stability against external source glitches. Further improvements included optimizing RocksDB with compression and implementing an in-memory LRU cache, which significantly reduced disk lookup latency. These changes have led to a more efficient and stable pipeline, demonstrating the power of streamlining data processes through advanced stream architecture.