January 2018 Summaries

In January 2018, the General Availability (GA) release of the .NET clients for Elasticsearch 6.x was announced, emphasizing full compatibility with Elasticsearch 6.0 and 6.1, while also preparing for future updates. The release involved significant changes, including serialization updates, the internalization of Newtonsoft.Json, and adjustments to accommodate the removal of multiple types in a single index. Major features introduced include a new serialization path for user-defined types, allowing greater customization without affecting core client functionality, and the ability to use any version of Newtonsoft.Json without conflict. The update also addressed breaking changes such as the need for separate indexes for different .NET types and introduced enhancements to client code generation and exception handling. The community's contributions were acknowledged as integral to the development of this release, and users were encouraged to explore and provide feedback on the new version.

Elasticsearch released version 6.1.3 and 5.6.7, both of which are bugfix updates aimed at improving stability and performance, with version 6.1.3 being the latest stable release available for deployment on Elastic Cloud. These updates are based on Lucene 7.1.0 and 6.6.1, respectively, and address several critical issues, including a bug where transient cluster settings were not applied after setting additional persistent settings, and a problem with replica recovery entering an endless flushing loop. Several improvements were also made to snapshot/restore edge cases. Users are encouraged to download the new versions, test them, and provide feedback via Twitter or the Elastic forum, with any issues being reportable on GitHub.

In response to Google's decision to discontinue Google Site Search (GSS) by April 1, 2018, many businesses are transitioning to Elastic Site Search, a solution developed by Elastic after acquiring Swiftype. Elastic Site Search, which is built on Elasticsearch, is praised for its ease of use, robust features, and support from experienced search engineers, making it a suitable alternative for former GSS users. When choosing a replacement for GSS, it is important to consider factors such as content ingestion, ease of implementation, scalability, search analytics, and customer support. Elastic Site Search offers a straightforward migration process from GSS with a 14-day free trial to test its capabilities, allowing users to index data via a web crawler or API, customize search results, and implement a highly customizable search interface. Trusted by major companies like AT&T and Lyft, Elastic Site Search provides a reliable and advanced site search solution, supported by a team of specialists to assist with the migration process.

Logstash users often encounter limitations with existing filter plugins for data transformations, prompting them to either write custom plugins or use inline Ruby code in the pipeline configuration. While inline Ruby filters are simple to implement, they can lead to code duplication and increased complexity, making maintenance challenging. To address these issues, the ruby filter plugin version 3.1.0 introduces file-based scripting, which allows users to organize complex Ruby logic into separate files, enhancing reusability and maintainability. This method also supports parameterization and testing within the script files, providing a more robust solution compared to inline code. Despite its benefits, file-based scripting does not support third-party library integration or centralized pipeline management via the UI, but it is a significant improvement for Logstash users seeking to manage custom Ruby code effectively.

Data enrichment with Logstash enhances security analytics by providing additional insights into data, helping identify potential threats such as botnet IPs or visits to malware URLs. Three common enrichment methods are Elasticsearch, DNS, and translate filters—each suited for different data feeds. The Elasticsearch filter performs lookups against indexes for threat data integration, while the DNS filter resolves domains to IPs or vice versa, often using services like Spamhaus for blacklist checks. The translate filter uses dictionaries to map values, such as URLs against a malware list. For larger workloads, the memcache plugin enables fast, non-blocking lookups, ideal for high-volume threat data or asset lookups, with a scalable enrichment layer that doesn't maintain state on individual nodes. This approach allows for rapid identification of threat-related activities, supporting incident response and further investigation in security use cases.

Rich Collier's article explores the process of using custom Elasticsearch aggregations to manage derivative calculations in machine learning jobs, particularly focusing on detecting sharp rates of change in data, such as a "brown-out" recovery. The article emphasizes setting up a job configuration using the ML API in Kibana and creating a datafeed that defines the data source, which includes aggregating data through a date histogram and performing a sum aggregation followed by a derivative calculation. This process results in the creation of a new field called "orders_deriv" that is used by the machine learning job to detect anomalies, such as a significant recovery in order volume. Although the results can be viewed in Elasticsearch's UI, the article notes a limitation in visualizing derivatives over time due to constraints in the current Single Metric Viewer, which cannot dynamically reverse-engineer complex queries for raw data plotting. The author suggests that future updates may address this limitation while also recommending existing resources for machine learning users to enhance their workflows.

Logstash and Elasticsearch ingest nodes are both valuable components of the Elastic Stack, serving different roles in data processing and architecture. Logstash is a versatile tool with a wide array of plugins that can handle various input and output formats, making it suitable for complex architectures where data needs to be pulled from or pushed to different sources, including databases and message queues. Conversely, ingest nodes, introduced in Elasticsearch 5.0, streamline data processing by allowing data to be sent directly to Elasticsearch, simplifying architecture but with some overlap in functionality with Logstash. While ingest nodes offer simplicity and a reduced hardware footprint for smaller use cases, Logstash provides greater flexibility and capabilities such as persistent queues and integration with message queues, which are crucial for managing back-pressure and ensuring data integrity. The choice between the two depends on the specific architectural needs, data processing requirements, and whether integration with external systems is necessary, with the possibility of using both in tandem to optimize different parts of a data processing pipeline.

Elasticsearch's attribute-based access control (ABAC) system enhances document security by using user-specific attributes to determine access permissions, leveraging the terms_set query and templated role query features in X-Pack. Unlike traditional role-based access control (RBAC), which assigns access through predefined roles, ABAC evaluates user attributes such as team membership or certifications against document attributes like sensitivity level. This method offers a more dynamic and scalable approach, allowing for complex access rules based on multiple conditions, such as security levels, program memberships, and compliance with training requirements. The integration of these features in Elasticsearch enables more granular control over document access, ensuring that users can only access data for which they meet all specified criteria.

Elasticsearch's version 6.1 introduces an on-demand forecasting feature as part of its X-Pack machine learning capabilities, enabling users to predict future data trends based on historical data. This feature allows for capacity planning, such as estimating server disk space needs or predicting customer call volumes, by analyzing past performance indicators. Users can access forecasting through the Single Metric Viewer, where they can initiate forecasts and view results represented by trend lines and confidence bands. The accuracy of forecasts depends on various factors, including the amount of historical data available—ideally at least three weeks—and the maturity of the machine learning model. Forecasts come with unique identifiers and are automatically deleted after 14 days when run via the UI, although API usage permits customization of data retention durations. Elastic's machine learning tools require a Platinum subscription, but a free trial of X-Pack is available for new users.

In response to the widespread concern over the Spectre and Meltdown vulnerabilities, the text outlines an investigative approach using hardware performance counters to detect and mitigate these hardware attacks. These vulnerabilities exploit speculative execution and cache side-channels to disclose privileged memory, posing significant security challenges. The research focuses on using CPU performance counters, such as Transactional Synchronization Extensions (TSX) and Last-Level-Cache (LLC) UOPs counting, to identify abnormal patterns indicative of attacks, proposing strategies for detecting cache timing attacks and speculative branch execution leaks. The research highlights the potential of performance counters as cross-platform, low-overhead tools for monitoring and defending against these exploits. Despite the promising results, the text emphasizes that the research is ongoing, aiming to engage the broader research community in developing comprehensive defense mechanisms for this evolving class of vulnerabilities.