June 2018 Summaries

Fluentd and Fluent Bit are open-source tools developed by Treasure Data to streamline logging processes in distributed environments, particularly for Kubernetes and Docker. Fluentd, established in 2011 and written in Ruby, is a log aggregator that unifies and processes data from multiple sources into JSON objects, emphasizing scalability and resilience with a wide array of plugins. Fluent Bit, introduced in 2015 and written in C, is designed for high-performance environments with limited resources, acting primarily as a log forwarder with a smaller footprint and plugin ecosystem. While Fluentd handles heavy throughput and complex aggregation tasks, Fluent Bit excels in lightweight data collection and delivery, often serving as an edge device agent. Together, they form an efficient logging pipeline where Fluent Bit collects and forwards data to Fluentd for processing and routing. The growing adoption of Kubernetes bolsters their use, with Fluent Bit's lightweight nature complementing Fluentd's robust capabilities, enhancing observability in modern IT infrastructures.

Log data is crucial to Security Information and Event Management (SIEM) systems, serving as the foundational material for understanding IT events, but requires extensive processing and management steps to be truly useful. The ELK Stack, a popular open-source platform for log analysis, plays a significant role in log management within many open-source SIEM solutions. However, while it excels at data collection, parsing, storage, and analysis, the ELK Stack on its own lacks essential SIEM features like built-in alerting, correlation rules, and incident management capabilities. To transform the ELK Stack into a comprehensive SIEM solution, organizations often need to integrate additional components, which can be technically challenging and resource-intensive. Logz.io addresses these gaps by offering Cloud SIEM, a fully-featured, ELK-based solution with prebuilt rules, dashboards, and integrations, simplifying the process for users who prefer not to manage these complexities themselves. For those opting to build their own SIEM, augmenting the ELK Stack with other tools is necessary, though it demands significant engineering effort and expertise.

The AWS Marketplace facilitates access to a wide range of SaaS and API products, streamlining procurement and billing processes for users who can integrate these services with their existing AWS accounts. Logz.io, now available on the AWS SaaS Marketplace, allows AWS users to monitor their cloud environments more flexibly by integrating with services like CloudWatch, S3, CloudTrail, and RDS. Users can ship data from S3 buckets or use AWS Lambda, employing Logz.io’s built-in Kibana dashboards for analysis, and take advantage of its alerting engine to manage alert fatigue. Logz.io also utilizes advanced machine learning to identify unnoticed events within AWS environments. Subscriptions are available in 1, 12, and 24-month increments, with pricing based on daily data shipment, and automatic renewal is the default setting. Users can modify contracts from the subscription page but cannot cancel or delete them once subscribed. A free 14-day trial is offered, allowing users to test the service without commitment.

The article reviews five commercial Security Information and Event Management (SIEM) systems—AlienVault USM, Micro Focus ArcSight ESM, IBM Security QRadar, Splunk Enterprise, and LogRhythm—by evaluating their intended audience, deployment models, features, and strengths and weaknesses. AlienVault USM is deemed suitable for small to medium organizations but lacks certain advanced features and requires technical expertise for configuration. ArcSight ESM is recognized as a comprehensive solution for large enterprises but is noted for its complexity and high cost. QRadar is praised for its robustness and scalability but criticized for being overwhelming and not user-friendly. Splunk Enterprise offers high flexibility and extensibility, allowing for customization through add-ons, but its base system lacks core functionalities. LogRhythm is highlighted for its extensive security features and reporting capabilities, though it is costly and requires expert configuration. The article concludes by suggesting that organizations choose a SIEM system based on their size and specific needs, with recommendations tailored to different organizational requirements and resources.

In a world increasingly reliant on big data, visualizing this data has become a crucial practice for monitoring environments and conducting root cause analysis. Grafana and Kibana are two popular open-source tools used for data visualization, each serving distinct purposes. Kibana is part of the ELK Stack and is primarily used for analyzing log messages stored in Elasticsearch, offering robust querying and visualization capabilities. Grafana, originally a fork of Kibana, excels in metrics visualization and integrates with various time-series databases such as Graphite and Prometheus. While both tools offer powerful visualization and dashboard functionalities, they differ in their core design—Kibana focuses on log analysis, whereas Grafana is designed for metrics. Grafana has built-in alerting features, whereas Kibana requires additional tools for alerts. Additionally, Grafana provides extensive customization options and supports multiple data sources through plugins, whereas Kibana is limited to Elasticsearch. Despite their differences, many organizations use both tools in tandem to complement their monitoring infrastructure, benefiting from each tool's strengths.

Maintaining optimal performance in Elasticsearch clusters requires strategic monitoring and configuration, as various factors such as data types and usage patterns can impact performance over time. Establishing monitors to alert on issues and understanding trade-offs based on business priorities are crucial first steps. Key considerations include managing Elasticsearch queues to ensure they remain nearly empty for efficiency, configuring HEAP memory correctly to avoid performance degradation, and adjusting the flush interval for better indexing throughput. Disk sizing and managing shard allocations must be approached with awareness of watermarks to prevent performance issues, while field data and query caches should be carefully budgeted to avoid excessive HEAP usage. The complexity of optimizing Elasticsearch performance, influenced by diverse logging environments, suggests using benchmarking tools like Elasticsearch’s Rally to measure the impact of changes. Regularly comparing the cluster's configuration with desired outcomes and business logic helps in making informed adjustments, ensuring that the cluster's performance aligns with organizational needs.

In 2018, Docker marked its fifth year in production amid a rapidly evolving landscape for container technology, where it played a pivotal role in popularizing containers and enabling their integration into DevOps practices through lightweight, portable, and self-sufficient containers. Despite facing challenges from emerging container orchestration platforms like Kubernetes, which has gained dominance, Docker adapted by integrating Kubernetes into its Docker Enterprise 2.0, indicating a strategic shift to align with industry trends. As DockerCon 2018 approached, the company experienced significant organizational changes, including the departure of co-founder and CTO Solomon Hykes, yet continued to thrive with substantial investments and a robust user community. Although questions about Docker's strategic direction and the impact of serverless computing remain, studies suggest that container adoption is on the rise, providing Docker with a strong foothold in the market and a cautiously optimistic outlook for the future.