September 2016 Summaries

Adrien Grand discusses the suitability of Elasticsearch as a column store, particularly highlighting its strengths in storage efficiency and query performance due to its columnar data handling. While it does not serve as a general-purpose replacement for traditional column-stores, Elasticsearch leverages its indexing capabilities, particularly through doc values, to efficiently manage and query large datasets. Doc values, introduced in Lucene and later integrated into Elasticsearch, offer a low-memory, compressed, column-oriented view of data, made efficient by indexing at the document ID level. This allows Elasticsearch to excel in analytics on smaller subsets of data and supports features like sorting and aggregation, despite not being entirely space-efficient for sparse fields. The continuous development of Elasticsearch, with a focus on enhancing compression techniques and handling sparse data more efficiently, reflects its evolution from relying on memory-intensive fielddata to using doc values as the default columnar storage mechanism.

On September 28, 2016, a patch release for Reporting version 2.4.2 was announced by Joe Fleming, introducing a significant update to the user interface aimed at clarifying report access permissions. Previously, users could download and view reports created by others, but this capability was not clearly presented. The update now ensures that all reports from all users are visibly accessible, eliminating past confusion about permissions. Additionally, the release addresses bugs related to navigation errors in Settings and the improper use of the &sync timeout with Watcher, while also updating the timeout setting to reporting.generate.socketTimeout and deprecating the old configuration.

Elasticsearch 2.4.1, a bugfix release based on Lucene 5.5.2, has been announced, addressing several significant issues and is now available on Elastic Cloud. Users, particularly those on Windows or using X-Pack, are encouraged to upgrade due to critical fixes that include correcting a bug in the Windows startup script, which previously used an inefficient garbage collector for the multithreaded application. Other notable fixes involve resolving an exception with the Flush API that mistakenly triggered shard recoveries, correcting a bug in Watcher that caused loss of custom cluster state data, and allowing watches to be deleted or updated during execution, which was previously hindered by the new reporting feature. Full details of these changes are available in the release notes, and feedback is welcomed through Elastic's Twitter or forums, with issues reportable on GitHub.

Grok, a filter plugin for Logstash, is pivotal in parsing and structuring log data, enabling tools like Elasticsearch and Kibana to extract insightful information from raw text. Although there are over 200 Grok patterns to handle various data formats, performance issues frequently arise, particularly when a match fails, leading to reduced processing speed. This inefficiency stems from the reliance on regular expressions and the need for Grok to match patterns within lines of text rather than from start to finish. To mitigate this, adding anchors to Grok expressions can significantly enhance performance by reducing unnecessary checks and anchoring the regular expression to specific positions in the string. Monitoring tools and settings, such as timeout configurations and Logstash's Metrics API, provide valuable insights into Grok's performance, allowing users to detect and address inefficiencies in real-time. Through strategic pattern design and performance measurement, users can optimize Grok's role within the data processing pipeline, ensuring efficient and accurate log parsing.

Elasticsearch's version 5.0 introduces Ingest Nodes as a new feature to perform data transformation and enrichment within the Elastic Stack. These nodes utilize processors, which are configured into pipelines to handle tasks such as renaming fields or parsing logs. The Ingest Node includes 20 built-in processors, like grok and date, and supports plugins such as Ingest Attachment, Geoip, and User Agent for additional functionality. Users can configure pipelines through the _ingest API, and the Simulate API allows testing configurations before processing real data. An example demonstrates how an Ingest pipeline can parse web logs in the Combined Log Format to extract and transform data into structured fields, which enhances search and visualization capabilities within Elasticsearch. The blog concludes by mentioning a forthcoming guide on using Filebeat, Elasticsearch, and Kibana for web log ingestion and visualization.

Elasticsearch and Kibana Docker images have been released in beta form, tracking the latest versions of Elasticsearch and Kibana 5.0, and are pre-installed with the x-pack extension. Hosted on Elastic's Docker Registry, these images can be easily deployed using Docker Compose, provided that the necessary prerequisites are met, and instructions are available on GitHub. Users can launch an Elasticsearch and Kibana stack, view logs, access the Kibana interface, and manage data through simple commands. The beta images allow data persistence across sessions but are not recommended for production use until the full release. Suggestions and contributions are encouraged via issues and pull requests on GitHub.

Elasticsearch 5.0.0-beta1, based on Lucene 6.2.0, introduces significant enhancements and bug fixes aimed at improving performance and usability, though it is intended only for testing purposes and not for production deployment. Key advancements include an 80% improvement in indexing performance, the adoption of Lucene's LatLonPoint for geo-point fields, and the introduction of Painless as the new default scripting language to enhance security and speed. The release is part of a series leading up to the full 5.0.0 version, and it includes a Migration Helper plugin to assist users transitioning from earlier versions, offering tools like Cluster Checkup, Reindex Helper, and Deprecation Logging. Other notable changes are the switch to Log4j2 for logging, the default enabling of deprecation logging, support for atomic index and alias updates, and the setting of default heap sizes to 2GB, with users encouraged to test and provide feedback on this beta release.

Kibana 5.0.0-beta1 introduces a host of new features and improvements, including the integration of Timelion as a core component for advanced time series analysis, enhanced scripting options with the new Painless language, and customizable visualization legend positions. The update also includes a revamped Sharing UI for more efficient data sharing and the introduction of PDF report generation via X-Pack Reporting. Additionally, users can save and return to their workspaces in X-Pack Graph, while a fix for URL length errors in Internet Explorer aims to improve user experience. Several other enhancements, such as logarithmic scaling for bar charts and Boolean field formatters, have been implemented alongside dozens of bug fixes. The release encourages user feedback through GitHub, forums, and social media platforms while reminding users not to deploy the beta version in production environments.

Painless is a new scripting language introduced in Elasticsearch 5.0, designed for security and performance, developed due to the challenges of securing existing languages for remote execution. Created by Jack Conradson and team, Painless features syntax similar to Groovy and supports dynamic and static types, lambda expressions, and built-in regular expressions, with performance comparable to Java. It employs method and field-level whitelisting for stringent execution control and can be used anywhere in Elasticsearch where scripts run by specifying the 'lang' parameter as 'painless.' As the default language in Elasticsearch 5.0, Painless also allows for efficient scoring scripts and has been enhanced by contributions from Robert Muir, Uwe Schindler, and Nik Everett. Conradson, having previously worked on the Expressions language at A9.com, continues to improve scripting at Elasticsearch.

Colin Goodheart-Smithe discusses enhancements made to Elasticsearch's request cache feature, specifically focusing on its performance improvements for analytics use cases. Initially introduced in version 1.4.0, the request cache was disabled by default until version 5.0 due to its limited utility for computationally heavy queries with varying time windows. By refactoring the search request representation and implementing query rewriting, Elasticsearch can now effectively use the request cache, significantly accelerating search performance on repeated queries over static indices. The post highlights the process of rewriting queries to optimize cache usage, particularly for time series data, leading to dramatic reductions in query execution time, as illustrated with a UK house sales dataset. These improvements demonstrate the potential for further enhancements to query and aggregation caching strategies, promising even more substantial performance gains in future iterations.

Elastic has announced its acquisition of Prelert, a company known for its unsupervised machine learning engine capable of automatically identifying insights in large datasets, which aligns with Elastic's goal of improving data search and analysis capabilities. Prelert's technology has been successful in detecting anomalies in various contexts, such as transactions and user behavior, and its integration into the Elastic Stack promises to enhance the platform's ability to address a broader range of questions across IT Operations, Security, and Business domains. This collaboration aims to create a more integrated product by incorporating Prelert as a feature within the Elastic ecosystem, which will transition from its current separate offerings to a more seamless integration, improving user experience. The transition will see the end of Prelert's Splunk App, but existing commitments to customers will be honored, with support provided for those migrating to the Elastic Stack. The partnership is expected to deliver a robust platform that leverages machine learning to extend beyond traditional search functionalities, offering significant improvements in data analysis and user interaction.

Metricbeat, a lightweight shipper for host and service metrics introduced in the Beats 5.0 release, replaces Topbeat and expands its capabilities by incorporating control group (cgroup) metrics from the Linux kernel. Cgroups, used for resource allocation and usage metering in processes, enable Metricbeat to collect detailed metrics from containerized environments without relying on container tool APIs, though this approach limits access to container-specific metadata. By enabling cgroup metrics in Metricbeat's configuration file, users can gather data on CPU, memory, and block I/O usage, with forthcoming network metrics, offering a process-centric view that can equate to a container-centric perspective when running single processes per container. This feature, initially released as experimental, may undergo enhancements based on user feedback, and while it provides critical resource usage data, users can engage with the Beats engineering team for further questions or feature requests through Elastic forums.

In the blog post, Christoph Büscher, Luca Cavanna, and Isabel Drost-Fromm discuss a significant refactoring process undertaken in Elasticsearch 5.0 to improve the way search requests are handled internally. Previously, search requests were parsed at the shard level, resulting in multiple parses and inefficiencies, as well as difficulties in error handling and testing. The refactoring introduced an intermediate representation of queries, allowing them to be parsed once at the coordinating node and serialized for shard-level execution, thus improving efficiency and enabling optimizations like "Instant Aggregations." This change required extensive testing and incremental development to ensure stability and maintainability, ultimately leading to cleaner code architecture and paving the way for new features. The authors reflect on the challenges and benefits of this architectural shift, emphasizing the importance of frequent testing and small, manageable updates throughout the process.

In collaboration with MITRE, the article explores how adversaries use persistence techniques, such as COM Object Hijacking, to maintain access to compromised systems and evade detection. The MITRE ATT&CK framework categorizes these techniques, aiding defenders in identifying and mitigating threats. COM Hijacking, a method where adversaries manipulate software components to execute malicious code, has been a long-standing tactic but remains effective due to its complexity and potential for false positives. The article emphasizes the importance of a proactive hunting mindset and using frameworks like ATT&CK to systematically detect anomalies beyond traditional indicators of compromise. Automated tools, such as those developed by Endgame, can efficiently scan for these techniques, allowing defenders to focus on identifying sophisticated threats in a structured and timely manner.

In the first post of a three-part series on Instant Aggregations, Simon Willnauer and Shay Banon of Elastic explore the challenges and potential solutions for improving query performance in Kibana through caching. The story begins in 2013 with discussions about the slow initial dashboard load times due to Elasticsearch recomputing queries from scratch each time, despite similar searches being executed. The narrative unfolds with brainstorming sessions and technical analyses, leading to the realization that utilizing Apache Lucene's point-in-time index views could allow for better caching. The team proposed leveraging Lucene's top-level reader to create a request cache, ultimately aiming for efficient query rewriting based on time properties. Despite initial enthusiasm and a prototype developed by Mike Mccandless, the team faced the reality that substantial refactoring of the codebase was necessary, requiring significant engineering effort. However, the potential benefits of improving search efficiency and reducing workload for Kibana dashboards justified the investment, resulting in a 12-month development process followed by an additional six months to release an alpha version of the feature. The post concludes by suggesting further insights in subsequent articles.

Geena Rollins, a senior principal software engineer at Symantec, underscores the significant benefits of engaging with Elastic's dedicated support, highlighting her journey from being a new user to an Elasticsearch expert at a company with 10,000 employees. Despite her extensive background in search technology and a PhD in computer science, Geena finds the support and training from Elastic invaluable for tackling complex scenarios and avoiding potential pitfalls. Elastic's support, exemplified through her collaboration with support engineer Greg Nieman, has facilitated Symantec's transition from Solr to Elasticsearch, optimizing their search functions and addressing challenges like slow queries and search rejections. This partnership has not only enhanced Geena's expertise but also empowered her to consult effectively across various Norton Engineering product groups, underscoring the ongoing value of technical mentorship and educational support in utilizing Elasticsearch within Symantec's operations.

Dockbeat is an open-source project developed by Ingensi, a unit of Cyres Group in France, to monitor Docker containers by collecting and indexing their statistics into Elasticsearch via the Beats platform. Designed to enhance big data solutions using tools like Hadoop and Elastic Stack, Dockbeat capitalizes on Docker's API to track real-time resource usage statistics such as CPU, memory, network, and disk IO. It integrates with Elastic Stack for efficient log management, allowing for visualization in Kibana dashboards, and supports outputs to Elasticsearch, Logstash, and Kafka. Since its inception, Dockbeat has grown to version 1.0.0 and is available on GitHub, inviting community contributions for documentation, bug reports, and feature development.

Elasticsearch 5.0 introduces a significant change by removing the string type, which caused confusion due to its dual functionality for both keyword and full-text searches. To address this, the string type has been split into two distinct types: text for full-text searches and keyword for keyword searches, simplifying the process and eliminating the complexities associated with the previous model. Additionally, by default, string fields will now be mapped as both text and keyword, allowing for more efficient searching and aggregations without requiring reindexing. This update aims to enhance user experience by reducing complications and improving default mappings, while backward compatibility is maintained for existing indices, and automatic conversion is available for new indices to ease the transition. However, users are encouraged to update their templates and mappings in preparation for the eventual removal of backward compatibility in Elasticsearch 6.0.

Elasticsearch Curator 4.1 introduces several new features designed to enhance the user experience and functionality of the tool. Key updates include the implementation of Configuration Validation, which prevents misconfiguration by ensuring all options are within acceptable parameters, thereby avoiding errors like unintended index selection. Users can now filter indices by alias or count, allowing them to manage their data more effectively by selecting indices based on their inclusion in aliases or preserving a specified number of recent indices or snapshots. Additionally, Curator 4.1 allows the use of environment variables in configuration and action files, providing more flexibility, although this feature is still experimental and should be used with caution. These updates collectively aim to improve data management and configuration efficiency while maintaining safety through recommended practices like the use of the --dry-run flag for testing.

Kibana 4.6.1 and Reporting 2.4.1 have been released, addressing significant issues from previous versions. Kibana 4.6.1 resolves a regression causing fatal errors during term-ordered aggregations, while Reporting 2.4.1 fixes a critical security vulnerability (ESA-2016-05) that could lead to unauthorized report generation through a CSRF exploit. Users are strongly advised to upgrade promptly, with Elastic Cloud users receiving automatic updates. Upgrade instructions are provided, and support is available through forums, GitHub, and Twitter.