November 2016 Summaries

Elasticsearch 5.0.2 has been released as a stable update, incorporating bug fixes and improvements from its previous versions based on Lucene 6.2.1, and is available for deployment on Elastic Cloud. The update addresses several critical issues, including a bug in X-Pack that previously hindered rolling upgrades from version 5.0.0 to 5.0.1, frequent failures under load due to caching issues in X-Pack Security, and cluster instability that could lead to unallocated primary shards. Enhancements include the introduction of a HostFailureListener in the Transport Client to manage node connectivity and the ability for the Tribe node to use custom ports. Users are encouraged to upgrade to this version, provide feedback on social media, and report any issues through GitHub.

Kibana 5.0.2 has been released, introducing a security fix and several bug fixes to enhance functionality and security. A significant update ensures that operations in the Advanced Settings panel and the short URL service are performed by the currently authenticated user rather than the "Kibana Server" user, resolving a security concern where read-only users could alter global settings without accessing protected data. Bug fixes include improved error messaging when Kibana's version does not match Elasticsearch's, resolving visualization errors in Visualize or Dashboard, and correctly handling spaces in file paths with the kibana-plugin tool. The release notes provide further details, and the updated version is available for download, with upgrade procedures outlined in the documentation.

Elasticsearch performance can be significantly impacted by the implementation of the TRIM command for SSD storage, which informs the drive of data blocks that are no longer in use. This process helps reduce write amplification and improve garbage collection efficiency, ultimately enhancing the application's speed. Observations from Elastic's performance benchmarks revealed that enabling regular TRIM operations can stabilize performance, as evidenced by improved benchmark results following a shift from weekly to daily TRIM scheduling. However, in production environments like Elastic Cloud Enterprise, users are advised to carefully consider the timing of TRIM execution to avoid potential disruptions, suggesting it be scheduled during low-traffic periods or disabled entirely if the system is under constant load. Additionally, configurations may vary based on operating systems and setups, such as those involving LVM or dm-crypt, requiring specific adjustments to fully leverage TRIM's benefits.

Elasticsearch announced the release of bug fix updates for versions 2.4.2 and 1.7.6, marking the final release in the 1.x series, available on Elastic Cloud. Key changes in version 2.4.2 include a fix to prevent unintended deletions when using wildcards to delete indices, a downgrade from an exception to a deprecation warning for the position_increment_gap issue, and improvements in Tribe nodes' processing to focus only on relevant cluster state parts to reduce unnecessary processing and memory use. Users are encouraged to download the update, provide feedback on social media or forums, and report issues on GitHub.

Apache Lucene has undergone significant changes in how document values (doc values) are indexed and accessed, aiming to enhance performance and ensure users only pay for what they use. These updates, which will feature in the upcoming Lucene 7.0 release, move from a random-access API to a more restrictive iterator API, allowing for better compression and optimization, particularly benefiting sparse cases. Improvements include a new codec design to remove abstraction layers, implement sparse cases directly, and introduce a faster advanceExact API for specific document targeting. These changes have resulted in improved search performance and reduced index sizes, as demonstrated by new benchmarks using the New York City taxi ride data corpus. The benchmarks, which test both sparse and dense documents, indicate significant performance enhancements, despite initial setbacks from transitioning APIs. Additionally, index-time sorting, although slower during indexing, offers notable search speedups, illustrating a beneficial trade-off for many users. This evolution in Lucene underscores the importance of automated benchmarks in detecting performance regressions and guiding optimizations.

Elasticsearch 5.0 introduced the ingest node feature, which allows for pre-processing documents before indexing through an intercepting node that applies transformations defined in a pipeline. This blog post guides users on creating a custom ingest processor, specifically one that extracts URLs from a text field and stores them in an array, which can be used for various purposes like pre-fetching data or displaying URLs in applications. The process involves using tools like Gradle and Cookiecutter to set up and test the plugin, while highlighting the importance of unit testing and security considerations in Elasticsearch's Java environment. The blog also emphasizes the flexibility and potential of developing custom processors, encouraging users to explore possibilities such as geo-processing, language detection, and machine learning integration, all while maintaining performance by offloading tasks to dedicated nodes.

In a discussion about his experience at Elastic{ON} Tour conferences and the integration of Elasticsearch into Yodle's Centermark product, Mark Drago highlights the journey and evolution of the platform's technical architecture. Initially relying on Hadoop for pre-aggregating metrics, Yodle faced challenges in providing flexibility to their customers, leading them to adopt Elasticsearch for its speed and efficiency in delivering business insights. Drago appreciates the collaborative environment of the conferences, where he gains insights into new features and engages with peers to enhance their product offerings. He candidly shares the trials and triumphs of building Centermark, a marketing automation platform for franchises, and notes the sense of atonement in publicly discussing the project's early shortcomings.

Researchers at Endgame have developed a method to detect domain names generated by Domain Generation Algorithms (DGAs) using deep learning, specifically Long Short-Term Memory networks (LSTMs), which outperform existing state-of-the-art techniques. DGAs are used by adversaries to create pseudorandom domain names for connecting malware to command and control servers, which makes blacklisting or sinkholing ineffective. The deep learning approach eliminates the need for manual feature engineering, allowing the model to adapt swiftly to changes by automatically learning feature representations. Unlike traditional methods, this technique does not rely on contextual information like NXDomains or domain reputation, yet it achieves higher accuracy in identifying DGA-generated domains. The model was trained using data from Alexa's top 1 million sites for benign domains and custom DGA algorithms for malicious data, demonstrating an impressive AUC of 0.9977 and a 90% detection rate with a 1/10,000 false positive rate on a diverse dataset. The research highlights the potential for LSTMs to enhance cybersecurity measures against evolving malware threats.

Logstash's Dissect filter is introduced as an efficient alternative to the Grok filter for extracting structured fields from unstructured data, especially within Logstash 5.X environments. Unlike Grok, which relies heavily on regular expressions and can slow down processing due to excessive backtracking, Dissect uses a delimiter-based approach to extract fields, resulting in improved performance and predictability. Dissect is designed to handle known patterns effectively by splitting text with specified delimiters, offering various field syntaxes such as Normal, Skip, Append, and Indirect to cater to different extraction needs. Performance comparisons indicate that Dissect outpaces Grok + CSV combinations in throughput, although it may not be as flexible as Grok in handling irregular text. The introduction of Dissect aims to enhance Logstash's ability to manage high-throughput environments by providing a faster and more reliable method for field extraction, potentially simplifying configurations and increasing pipeline efficiency.

The article by Russ Cam discusses the evolution and future of handling attachments in Elasticsearch, specifically for .NET developers using the NEST high-level client. Historically, the mapper-attachments plugin allowed for the indexing and searching of text-based document attachments, such as Word documents and PDFs, by extracting metadata and content. However, with the release of Elasticsearch 5.0, this plugin became deprecated in favor of the ingest-attachment processor plugin, which offers a more efficient and streamlined approach to processing attachments by performing extraction before indexing. The article provides a detailed walkthrough of transitioning from the mapper-attachments plugin to using the ingest-attachment processor, including installation, configuration, and mapping steps, as well as demonstrating how to index and search attachments within Elasticsearch. This new method, which is part of the ingest node suite, improves the predictability and performance of attachment handling, ensuring that extracted metadata is readily available in the search results and stored in the source field.

The text discusses the implementation and benefits of using ingest nodes in Elasticsearch 5.0 as part of the Elastic Stack deployment, specifically for transforming data before indexing. While traditionally Logstash has been used for data transformation, ingest nodes offer an efficient alternative for simpler tasks by allowing direct connections from Beats like Filebeat and Winlogbeat to Elasticsearch, thus potentially reducing the need for message queues. The text provides a detailed guide on configuring Elasticsearch, Filebeat, and Kibana to ingest and visualize web access logs from an Apache server. It highlights that ingest nodes streamline processes and simplify architecture when complex Logstash filters are unnecessary, leveraging the efficiency of operations performed within Elasticsearch. The integration with Kibana and its Timelion feature further enhances data visualization capabilities, making it a comprehensive solution for data processing and analysis.

On November 15, 2016, Kibana released versions 5.0.1 and 4.6.3, addressing a security vulnerability in the short URL feature that allowed potential redirects from the Kibana domain to other websites. This vulnerability was identified as ESA-2016-09, thanks to the efforts of the GE Digital Security Team. Alongside the security fix, Kibana 5.0.1 included several bug fixes: an updated error message for when sessionStorage is disabled, improved basePath configuration handling when trailing slashes are removed, proper rendering of the Sharing UI in dark theme dashboards, and corrected tile map bounding box filters to prevent errors in Elasticsearch aggregation responses. Users are encouraged to review the release notes and upgrading documentation and to download the latest versions from the Kibana website.

Elasticsearch 5.0.1, based on Lucene 6.2.1, has been released with essential bug fixes, focusing on improving stability and security, especially for X-Pack Security users. Key issues addressed include the persistence of user security contexts across requests, which previously could lead to failures in basic operations, and a bug allowing duplicate documents with auto-generated IDs due to an optimization in version 5.0.0. The release also resolves problems with certain queries not parsing "now" correctly, an upgrade to Lucene 6.2.1 to fix a bug in custom analyzers, and issues with snapshots and the Java REST client. Users are encouraged to upgrade to benefit from these fixes and provide feedback through the available channels.

The text provides an in-depth explanation of shard allocation and management in Elasticsearch, detailing the processes involved in allocating shards to nodes within a cluster. It covers various scenarios, including index creation, shard relocation, full cluster restarts, and single-node loss. Key concepts discussed include the role of allocation deciders, which evaluate and approve shard placement decisions, the importance of balancing data across nodes, and the mechanisms used to ensure data integrity during shard movement. The document also highlights Elasticsearch's use of Lucene for data storage, the transaction log for data recovery, and the use of features like sync_flush and sync_id to optimize shard synchronization and recovery. Through these scenarios, the text illustrates how Elasticsearch maintains performance and data consistency, even during complex operations such as rebalancing, node loss, and cluster restarts.

RedOwl, utilizing the Elastic Stack as a crucial component of its platform, participated in the Elastic{ON} Tour in New York to explore insider threat detection and share insights on leveraging Elastic's capabilities. The event was an opportunity for RedOwl's team, including Adam Reeve and Russell Snyder, to engage with Elastic engineers and attendees, discussing the integration of Prelert's analytics functionality and the scalability of their solution for large enterprises. During their presentation, they highlighted the unique aspects of RedOwl's platform, which synthesizes intelligence from diverse unstructured data streams and explained their transition from Hadoop to Elasticsearch, emphasizing its role in their data processing architecture alongside technologies like Apache Nifi and RabbitMQ. Adam Reeve, with extensive experience in scalable service architectures, leads RedOwl's efforts in ensuring the platform's security and performance.

Rashid Khan's article explores the use of the .if() function in Timelion, a tool integrated into Kibana for time series data visualization and manipulation. Designed to enhance the flexibility of data analysis, the .if() function operates similarly to the .condition() function, enabling dynamic point-wise comparisons within time series data using operators such as less than, equal to, or greater than. The article provides practical examples demonstrating how to modify data points based on specified conditions, such as transforming a series into binary form or comparing it against a moving average to highlight trends visually. Khan illustrates how to use the function to apply different outputs, like drawing bars or shading areas, to effectively visualize when data exceeds or falls below certain thresholds. Overall, the article aims to educate users on leveraging Timelion's capabilities for enhanced data insights, encouraging readers to explore further resources for a deeper understanding.