January 2017 Summaries

Elastic's blog post discusses the enhancements made in Logstash 5.0 and 5.2 to improve monitoring and manageability of Logstash filters in production environments. The post highlights the integration of Elastic Stack's x-pack plugin, which allows administrators to monitor Logstash metrics directly from Elasticsearch, offering vital insights like ingestion rates and filter performance without relying on external systems. A significant focus is on identifying slow filters using the built-in APIs and the slowlog feature, which logs events that exceed a specified processing time. The article also explains the utility of the hot threads API for diagnosing CPU-intensive operations within Logstash, emphasizing the importance of optimizing regular expressions in the grok filter to enhance performance. Elastic promotes the use of its self-service tools for better insight into Logstash's operation and hints at future updates aimed at further simplifying pipeline management.

Elasticsearch 5.2.0 introduces several enhancements, including numeric and date range fields, which allow indexing of date and numeric ranges for more complex queries, and the cluster-allocation-explain API, which simplifies troubleshooting shard allocation issues. The release also introduces keyword normalizers, enabling case normalization and similar functions for keyword fields, and a new approach to terms aggregations that allows partitioning to return all terms, addressing high cardinality field challenges. These updates aim to enhance Elasticsearch’s functionality, allowing for more nuanced data handling and problem-solving capabilities. The release is available for deployment on Elastic Cloud, and users are encouraged to try it and provide feedback.

The blog post, part of a six-part series, discusses how to scale the integration of ArcSight with the Elastic Stack, specifically focusing on enhancing the architecture for improved performance and retention. It details the use of a hot-warm architecture for Elasticsearch to manage high indexing throughput and long retention policies. The post highlights the incorporation of a message queue, such as Kafka, between ArcSight and Elasticsearch to provide architectural isolation and scalability for Logstash instances. Kafka also serves as a buffer for incoming data during peak periods to prevent data loss, while Logstash ensures end-to-end delivery, albeit with some risk of data loss if a Logstash instance fails. The post further explains how to set up the Elastic Stack, including Elasticsearch, Kibana, and Logstash, and integrate Kafka to pull data for indexing into Elasticsearch. Additionally, it mentions using Docker for simplifying installation and configuration and emphasizes the importance of maintaining updated credentials across the setup to ensure seamless operation.

In a blog post by Ali Beyad, the cluster-allocation-explain API in Elasticsearch is described as a tool to diagnose shard allocation issues, which can lead to a RED cluster state when shards are unassigned or misallocated. Introduced experimentally in version 5.0 and refined by version 5.2, the explain API addresses questions about why shards are either unassigned or assigned to specific nodes. Shard allocation is crucial for maintaining data availability and performance, as Elasticsearch divides indices into shards distributed across nodes for scalability and fault tolerance. Various factors like node failures or misconfigurations can lead to allocation problems, and the explain API assists administrators by providing detailed explanations of allocation decisions, helping to resolve issues by adjusting configuration settings. The blog post demonstrates the API's utility through examples, showing its effectiveness in diagnosing both unassigned primary and replica shards, as well as understanding why assigned shards remain on certain nodes, thus highlighting its value in ensuring cluster stability and efficiency.

Elastic{ON} 17 is an upcoming event featuring new sessions that cover various aspects of the Elastic Stack, targeting enhancements in security, container management, benchmarking, and monitoring. Notable presentations include a guide on integrating ArcSight with the Elastic Stack to boost SIEM capabilities, insights from Barclays on their cybersecurity initiatives utilizing Elasticsearch, and practical advice on deploying and monitoring the Elastic Stack in Docker environments. Additionally, the event will feature demonstrations on using Rally for Elasticsearch benchmarking and configuring alert systems with X-Pack to ensure optimal cluster performance. The post also advises readers to register before February 1st to enjoy a discounted ticket price.

Jay Greenberg's blog post discusses utilizing Kibana and a custom tile server to visually analyze NHL play-by-play data from the 2016-2017 season. The NHL provides JSON-formatted data for each game's plays, including details such as play type, players involved, and coordinates on the ice. By importing this data into Elasticsearch and using Kibana dashboards, users can generate visualizations like timelines and pie charts to analyze plays and team performance. The post details the creation of a custom map tile server using GeoServer to correctly visualize play locations on a hockey rink. This involves converting an arena image to a GeoTIFF, setting up GeoServer, and configuring Kibana to use this custom map. The blog encourages further exploration, such as analyzing goalie vulnerability or penalty trends, and invites readers to experiment with the data themselves.

Artemis is an intelligent assistant developed by Endgame to enhance cyber defense operations by automating routine tasks and facilitating more efficient data analysis for security professionals. Unlike traditional bots that are prevalent in other industries, Artemis addresses the unique challenges faced in information security, where the shortage of skilled professionals and the need for contextualized data analysis are critical. By providing a conversational interface, Artemis allows analysts to interact with endpoint data using natural language, thereby simplifying complex queries and enabling rapid contextualization of security alerts. The assistant incorporates expert recommendations, streamlines workflows, and reduces the data burden by focusing on relevant information, ultimately empowering analysts to concentrate on advanced investigatory tasks. Artemis integrates with Endgame's open API, offering both junior and experienced users the flexibility to leverage its capabilities programmatically while maintaining control over their investigative processes.

Elasticsearch 5.x introduces a "Hot-Warm" architecture for optimizing large time-based data analytics by using a tiered system with three types of nodes: Master, Hot, and Warm. Master nodes focus solely on cluster management, enhancing stability by being free from data handling, while Hot nodes manage all indexing and store recent indices on SSDs due to their high I/O demands. Warm nodes, in contrast, are designed for storing less frequently accessed, read-only indices on larger spinning disks. Efficient data allocation is achieved by tagging nodes with attributes like "hot" or "warm" and using settings such as "index.routing.allocation.require.box_type" within Elasticsearch or index management tools like Logstash and Beats. Automation of index movement between nodes is facilitated by Elasticsearch's Curator tool, which also supports operations like force merging to optimize storage. Additionally, the introduction of Rollover and Shrink APIs provides a streamlined method for managing time-based indices by reducing the number of shards, further enhancing performance and efficiency.

In response to a recent attack on open-source databases where data was copied, deleted, and held for ransom without using traditional ransomware, Elasticsearch emphasized the importance of securing instances that are accessible over the Internet. The attack highlighted the necessity of implementing proper security configurations to prevent data loss or breaches. Elasticsearch recommends that users secure their deployments by using TLS encryption, user authentication, and role-based access control, which are now included for free in the default distribution of Elastic Stack 6.8/7.1 or later. It advises against exposing unsecured instances directly to the Internet and suggests using firewalls, VPNs, or reverse proxies to restrict access if Internet-facing deployment is unavoidable. Elastic Cloud SaaS offerings come with built-in security measures, including X-Pack security with randomly assigned passwords, redundant firewalls, and encrypted communication, ensuring customer data protection. For those less familiar with these security practices, Elasticsearch encourages utilizing their cloud service where security configurations are automatically managed.

Kibana versions 5.1.2 and 4.6.4 were released, featuring various bug fixes and improvements. Version 5.1.2 addressed issues such as errors triggered by using a secondary datetime field, reinstated the collapse button in the Timepicker, and improved spy tab performance on the Discover page. Version 4.6.4 incorporated many changes already present in version 5.0, including upgrading Node.js to version 6.9.0 for better memory management and resolving a segfault issue, along with improved support for Internet Explorer through shorter URLs and rendering performance enhancements in the Discover app. Additionally, this release upgraded the Hapi server to version 14.2.0 to accommodate the new Node.js version and made adjustments for better handling of percentile ranks aggregation and metric visualizations.

Elasticsearch announced the release of version 5.1.2, the latest stable update, and version 2.4.4, the newest release in the legacy 2.x series, both available on Elastic Cloud. The update addresses several key issues, including a security vulnerability in X-Pack 5.1.1, where document- and field-level security was improperly applied to certain requests. This release also includes performance improvements for mapping updates, ensuring only changed types are reprocessed, and resolves issues with master nodes leaving clusters under heavy indexing load by utilizing consistent IndexReader statistics. Additionally, the Netty recycler has been disabled to prevent heap exhaustion and out-of-memory errors, enhancing system stability under high throughput conditions. Users are encouraged to upgrade and provide feedback on their experience.

The blog post, part of a series on integrating the Elastic Stack with ArcSight SIEM, explores how to monitor security data in Elasticsearch using X-Pack's alerting features. It offers a step-by-step guide to setting up alerts for successful SSH logins from external IPs using Common Event Format (CEF) data, processed with Logstash and visualized in Kibana. The approach involves creating "watches" to detect anomalies by setting an alert execution schedule, defining input data, and transforming the data into a readable structure. The post provides a detailed example of a watch configuration that triggers alerts when specific conditions are met, such as a successful login from an external IP, and discusses the use of Elasticsearch’s native scripting language for processing. As part of a broader series, this post builds on earlier discussions and sets the stage for further exploration into scaling architecture and detecting complex patterns with machine learning in subsequent entries.

Elasticsearch 5.2 introduces new Numeric and Date Range field types, enhancing its capabilities to index and search over continuous ranges, which was previously a challenge with discrete numeric and date fields. These new fields, including integer_range, float_range, long_range, double_range, and date_range, are implemented using advancements in Apache Lucene, specifically the Bkd-Tree Data Structure, which allows for efficient indexing and searching of discrete numeric points. The range queries employ a domain-specific language (DSL) with optional relational parameters like INTERSECTS, CONTAINS, and WITHIN, enabling more complex and flexible search queries. This development is particularly beneficial for diverse use cases, such as scheduling conflicts, media broadcasting timelines, and medical diagnostics, by providing a streamlined method for handling continuous data ranges. The update is a significant step towards supporting multi-dimensional indexing and searching, with potential future enhancements aimed at expanding their capabilities to handle up to 8-dimensional ranges.