May 2017 Summaries

Elastic Cloud Enterprise (ECE) 1.0 has officially become generally available, offering a comprehensive solution for managing Elasticsearch clusters and Kibana instances. It enables organizations to provision, monitor, and orchestrate these resources from a single console, addressing common challenges such as scaling, multitenancy, and version management. ECE tackles issues like different access needs, service level agreements, and security compliance, which can complicate cluster management. By streamlining operations like creating new clusters, scaling, hosting multiple versions, and enabling Kibana, ECE helps production teams focus on business advancement without being bogged down by technical complexities. This product, which uses the same code base as Elastic's cloud service, is designed for organizations expecting growth with Elastic, providing centralized management and support for various environments, making it a valuable tool for both existing users and those looking to adopt Elasticsearch in their operations.

Cluster Alerts in X-Pack Monitoring for Elasticsearch was introduced to provide users with real-time alerts for common cluster issues by leveraging X-Pack alerting via Watcher. This feature periodically queries monitoring data to identify and alert users about critical issues, such as mismatched versions of Elasticsearch nodes or cluster status changes. Initially, four types of Watches were included, with plans to expand alerts to cover X-Pack license expiration, resource utilization, and node activity. Future improvements aim to offer customizable thresholds and integrations with various notification platforms like Slack and PagerDuty. Users can experiment with these capabilities using a trial license, ensuring they remain informed about their Elastic Stack's health even while on vacation.

The blog post discusses how the Elastic Stack can be utilized to detect and analyze the WannaCry ransomware, which exploited a vulnerability in SMBv1 to infect over 400,000 computers globally. It describes setting up a simulated environment using Windows and Linux virtual machines to replicate WannaCry's behavior and outlines how tools like Packetbeat and Winlogbeat, in conjunction with Sysmon, can identify malware signatures and suspicious activities. The post highlights the use of Elasticsearch and Kibana to monitor network traffic, detect potential downloads, and identify execution and spread of the ransomware within an organization's network. It also addresses the role of the infamous kill switch in halting the malware's execution and suggests that the Elastic Stack's alerting capabilities could be vital in providing early warnings of infection. Furthermore, the article emphasizes the importance of refining detection methods and suggests leveraging machine learning capabilities within the Elastic Stack for more effective monitoring in larger infrastructures.

The article introduces a new feature in Filebeat, which allows users to read Redis slow logs by adding a specialized prospector type that connects to Redis and retrieves slow logs, inspired by the community Beat, Rsbeat. This feature, which is experimental and set to be included in version 6.0.0, enables Filebeat to generate an event for each Redis slow log entry, capturing details like command, duration, and arguments, and clearing the buffer for subsequent polls. The update is part of the "Brewing in Beats" series, which provides weekly insights into Beats' latest developments, including changes to Filebeat, Metricbeat, Packetbeat, and infrastructure and documentation adjustments.

The blog post provides a comprehensive guide on writing, debugging, and testing watches within the X-Pack alerting framework, emphasizing efficient workflows and practical tips. It introduces the concept of a watch as a unit of work that monitors data sources, triggers alerts based on specified conditions, and executes actions when these conditions are met. The post details the use of the Execute Watch API to manually test watches, examining their execution paths and outcomes without waiting for scheduled runs. It also delves into the watch history, which logs detailed execution data for monitoring and debugging purposes, and discusses how alternative inputs and simulated actions can assist in robust watch testing. Additionally, the post highlights the croneval tool for evaluating cron expressions and briefly mentions the new Watcher UI available in Elastic Stack 5.4 for enhanced user experience. The piece concludes with an invitation to explore further advanced topics and engage with the X-Pack discussion forum for additional support.

The blog post delves into the intricacies of Elasticsearch's query and filter execution processes, challenging the common misconception that filters always precede queries. It explains that execution involves two phases: an initial approximation phase using operations like `nextDoc` and `advance`, followed by a verification phase using `matches`. The post highlights that the order of execution is determined by factors such as cost and match cost, rather than a fixed sequence, as illustrated through examples like term queries, disjunctions, conjunctions, and phrase queries. The document underscores the importance of metadata from the inverted index, such as term and document frequencies, in optimizing execution order. It clarifies that while the order of queries and filters in the query DSL does not impact execution, the system automatically reorders them based on their respective costs, and ultimately, execution is interleaved to ensure efficiency.

The text discusses the upcoming changes in Elasticsearch concerning the use of "types" within indices, focusing on the move towards eliminating types due to user misconceptions and technical pitfalls such as sparsity and scoring issues. Elastic, the company behind Elasticsearch, has observed numerous support cases where users misinterpreted types as equivalent to relational database tables, leading to complications. To address this, Elastic plans to phase out types, replacing them with a special field for parent-child relationships, aiming for a smooth multi-phase transition. This shift will culminate in the complete removal of types in version 7.0, with interim features introduced in versions 5.x and 6.0 to facilitate adaptation. The changes are expected to be seamless for most logging and security analytics users, while those using Elasticsearch as a search or document database are advised to review and adjust their type usage. Elastic encourages users to engage on their forums for support during this transition.

Elastic has launched its online store, making it possible for fans and community members to purchase branded merchandise such as stickers, t-shirts, and custom-designed socks inspired by the open-source project Kibana. Previously, Elastic merchandise was only available at meetups, conferences, or Elastic{ON} events. To celebrate the opening, the store is offering a limited-time 20% discount on Elastic socks, which are uniquely designed by Sock Club. Core Elastic stickers are available for free, with customers only needing to cover shipping costs. The company encourages fans to stay updated on new products and promotions by following Elastic on social media platforms like Twitter, Facebook, and LinkedIn.

In May 2017, Henrik Nordvik detailed efforts to resolve a native memory leak in Elasticsearch version 5.1.1, particularly affecting small clusters with minimal activity. The issue was identified when memory usage by Elasticsearch continually increased until the system rebooted due to memory exhaustion, indicating a leak. Although traditional tools like Yourkit and MAT didn't reveal the problem within the heap, enabling Native Memory Tracking (NMT) pinpointed the leak outside the JVM's tracking capabilities. To address this, jemalloc, a memory allocation library, was used to identify native memory allocations, revealing that java.util.zip.Inflater and java.util.zip.Deflater were major contributors due to improper memory release. These findings prompted fixes in both Lucene and Elasticsearch, resolving the issue in version 5.2.1. The investigation highlighted the importance of using advanced memory profiling techniques and tools beyond traditional JVM methods to detect and address native memory leaks in complex systems.

The blog post elaborates on using machine learning and Elasticsearch, specifically the X-Pack features, for security analytics by detecting anomalies in log data that may indicate cyber threats. It clarifies that machine learning serves as an algorithmic assistant rather than a magical solution, aiding security teams to automate the analysis of log data for identifying patterns and anomalies under expert guidance. The post introduces the concept of "machine learning recipes," which provide structured configurations for detecting specific security threats, such as DNS tunneling, through automated anomaly detection. These recipes guide users in setting up machine learning jobs by detailing the theory, description, and steps involved in modeling and analyzing results. With X-Pack's integration into the Elastic Stack, machine learning results can be utilized to trigger real-time alerts, enhancing threat monitoring. Additionally, the post highlights that these recipes allow security analysts to leverage machine learning without programming expertise, expanding their capacity to detect and respond to cyber threats effectively.

In May 2017, significant updates and changes were announced for the Elastic Puppet ecosystem, focusing on the deprecation of Puppet 3 and the introduction of new features across various modules. With Puppet 3 reaching its end-of-life in December 2016, support for this version will be phased out with the 6.x release of the Elastic Stack, allowing a period for users to upgrade. The modules have been relocated to the "elastic" namespace on Puppet Forge, with updates and releases to follow there. An official Puppet Kibana module was released at version 5.0, offering enhanced management capabilities, while the Elasticsearch module introduced support for indices and pipelines, allowing for better management of settings and processing workflows. Logstash version 5 brought improvements in service management, with the module now supporting only Logstash 5.x, simplifying service handling and enhancing stability. Additionally, comprehensive plugin support and a re-written test suite ensure reliability across supported systems and Puppet versions.

The WCry or WanaCrypt0r ransomware, which rapidly spread across Europe and Asia in May 2017, affected nearly 100 countries and disrupted numerous services, including 45 hospitals in the UK. The ransomware utilized the ETERNALBLUE vulnerability to propagate and execute its damaging activities, such as encrypting files on infected systems. It demanded ransom payments in Bitcoin for file decryption, though it was not guaranteed that paying would ensure data recovery. A notable feature was a "killswitch" function that, when activated, halted the malware's operation, significantly slowing its spread. This was discovered and utilized by researchers, notably MalwareTech, who set up a sinkhole that mitigated the outbreak's impact. The malware's execution involved several technical steps, including exploiting vulnerabilities, setting up encryption keys, and creating persistence mechanisms, with the primary goal of extorting victims for financial gain. Security experts emphasized the importance of patching systems and maintaining backups to prevent such attacks.

Elasticsearch 6.0.0-alpha1, a pre-release version based on Lucene 7-SNAPSHOT, introduces several changes and features aimed at simplifying the upgrade process from previous versions and enhancing performance. This alpha release is intended for testing purposes only and is not compatible with the final 6.0.0 version, nor should it be used in production environments. Notable improvements include support for sparse doc values, index sorting for faster search results, and sequence numbers for more efficient shard recovery. The upgrade process is streamlined through rolling restarts and cross-cluster search capabilities, negating the need for full cluster restarts and reindexing of older indices. Elasticsearch 6.0.0 will also phase out mapping types, allowing only one type per index, which simplifies the structure and enhances API interactions. Users are encouraged to test this alpha version, report issues, and provide feedback to contribute to its development.

The article explores the process of using Elasticsearch's Ingest Node to parse a CSV file containing New York City's subway station data, transforming it into a structured JSON format to be indexed and visualized using Kibana. The example uses an open dataset of subway entrances and exits, illustrating how Ingest Node can replace Logstash for simpler data ingestion tasks without the need for additional software. It details setting up an Elastic Cloud cluster, manipulating CSV data, and using a Grok processor to parse and break down unstructured lines into structured fields. The process includes creating an index template with specific field mappings to facilitate document filtering and geo-location queries. After indexing, the data is visualized in Kibana, where users can create a Tile Map to display station locations and utilize various visualizations to answer questions about station amenities, such as the presence of elevators. The article emphasizes the ease of starting from scratch with Elasticsearch and Kibana to turn a text file into insightful visualizations.

Elastic has introduced machine learning features for the Elastic Stack, integrated through X-Pack, to enhance the capabilities of Elasticsearch by adding time series anomaly detection using unsupervised machine learning. This integration aims to empower users to extract deeper insights from their time series data, such as identifying unusual behaviors or processes in services, and is particularly useful for analyzing log files, performance metrics, and transaction data. The machine learning functionality is designed to handle large volumes of data by detecting anomalies in real-time and is optimized to run natively on an Elasticsearch cluster, allowing millions of events to be processed efficiently. Users can create machine learning jobs directly in Elasticsearch, with the results seamlessly integrated into Kibana for easy visualization, thus offering significant performance and operational advantages by keeping data within the cluster. Although the features are currently in beta as part of X-Pack version 5.4, Elastic encourages users to try them and provide feedback to further refine and expand their machine learning capabilities.

Elasticsearch 5.4.0, based on Lucene 6.5.0, introduces significant improvements and optimizations in search functionality and query execution. The release features batched reduction of search results to manage memory usage more effectively on coordinating nodes, allowing the shard soft limit to be set as unlimited by default. It also enhances the accuracy of proximity queries for multi-word synonyms through improvements in Lucene's handling of graph token streams and introduces optimized query execution for range and nested queries. Additional updates include adjustments to Netty's default receive predictor size, date-range query enhancements in the percolator, and migration of sensitive settings to a secure keystore. The release aims to provide a more efficient, reliable, and user-friendly search experience, inviting users to test and provide feedback through various platforms.

The Elastic Stack's 5.4 release introduced CentOS 7 as the standardized base for its official Docker images, replacing the previously used Alpine Linux. This change aims to create a more cohesive and improved user experience across Elasticsearch, Logstash, Kibana, and Beats by providing a common operating system foundation. While Alpine Linux is appreciated for its minimalistic and security-focused nature, its small image size was not fully realized due to the heavy dependencies of Elastic's products, particularly those relying on the JVM. The transition to CentOS 7, with its larger but more feature-complete base, offers benefits such as greater stability and reduced network transfer size due to layer reuse. Additionally, CentOS's use of GNU libc, a more established library than Alpine's musl libc, addresses several technical challenges and enhances system reliability. Elastic encourages users to engage in forums and report issues to improve their experience with Docker images in the Elastic Stack.

The text provides a comprehensive guide on deploying Elasticsearch and Kibana on Azure using an Azure Resource Manager (ARM) template, available through the Azure Marketplace. It explains how this template simplifies the process of provisioning resources, allowing users to integrate Elasticsearch deployments into their build automation pipelines. Key features of the ARM template include the ability to configure parameters such as snapshot and restore functionality, Transport Layer Security (TLS), load balancing, and SSL offloading with Application Gateway. The document details deployment scenarios using Azure CLI and PowerShell, including configuring Service Principals for automated deployments, setting up TLS and SSL certificates, and integrating additional Azure resources like storage accounts and virtual networks. It also covers scenarios such as deploying multiple clusters within a virtual network, connecting Azure App Services to Elasticsearch for internal communication, and performing data snapshots and restores using Azure storage. The guide concludes by highlighting the ARM template's flexibility and ease of integration with existing Azure resources, enabling robust data analysis using Elasticsearch and Kibana.

The blog post discusses a method for visualizing geo-referenced data using a custom map in Kibana, demonstrated with data from Elastic{ON} 2017. The process involves using tools like Inkscape, QGIS, and GeoServer to create a vector-based map that can effectively handle different zoom levels. The authors describe converting an SVG or PDF of the Elastic{ON} floor plan into shapefiles, which are then imported into GeoServer to create a layered tile map. This tile map is used to display wireless network usage and sentiment data collected from the event, but the approach can be adapted for other geo-enriched metrics, such as plotting server metrics over data center floor plans. The article highlights the importance of using vector representations to maintain image quality at various zoom levels and provides guidance on configuring Kibana to display the custom map.

Elasticsearch, widely recognized for its search and analytics capabilities, can also be adeptly utilized for text classification, offering a more streamlined alternative to traditional Natural Language Processing (NLP) tools. The platform simplifies the text classification process by using language analyzers for tasks like tokenization and stemming, and through its More Like This (MLT) query, it can efficiently identify and categorize similar documents. Unlike conventional methods that rely on supervised machine learning models or complex NLP libraries, Elasticsearch's approach enables dynamic model updates at index time without application downtime, making it ideal for agile environments where data changes frequently. While it may not match the precision of specialized algorithms like SVM or Naive Bayes, its integration simplicity and the ability to rapidly process and classify data directly from the existing Elasticsearch infrastructure present practical advantages, especially for enterprises dealing with evolving datasets. The author, Saskia Vola, with a background in computational linguistics, highlights how Elasticsearch can serve as a powerful tool for text mining in real-world applications, such as e-commerce data classification, offering both efficiency and ease of use.