May 2016 Summaries

The guide provides a step-by-step approach to configuring Elasticsearch with Shield and integrating it with Windows Active Directory for domain authentication, aimed at Elasticsearch administrators. It begins by outlining prerequisites, such as having an Elasticsearch cluster on Linux and DNS configuration, before detailing the process of configuring the Elasticsearch Yum repository. The guide then explains how to set up an Active Directory user and security group, install necessary plugins, and configure Shield authentication with Active Directory by modifying configuration files on each cluster node. It concludes with testing the setup by performing a query and suggests further resources for customization and advanced configurations, emphasizing the flexibility and multiple options available for tailoring the setup to specific architecture requirements.

Kibana 5.0.0-alpha3, released by Court Ewing, marks the third alpha release in two months, introducing significant updates despite initial doubts about its feasibility. Designed to work exclusively with Elasticsearch 5.0.0-alpha3, this version is not intended for production use, as indices created will not be compatible with the final GA release, and upgrades from this version are unsupported. The release focuses on stability, featuring 30 pull requests in 28 days, and enhances the X-Pack UI with new user and role management features, as well as improved report processing capabilities. Additional updates include a field exists filter button, configurable status page for anonymous access, and several bug fixes such as improved histogram functionality and consistent back button behavior. The team plans to release one more alpha before transitioning to beta, where the focus will be on stability and upgradability, and they encourage user feedback through forums and bug reports on GitHub.

Elasticsearch 5.0.0-alpha3, based on Lucene 6.0.0, introduces a host of features and improvements for testing purposes as it approaches feature completeness. Notable updates include Painless scripting enhancements, indexing performance boosts, and the resolution of dots in field names to treat them as objects. The release also highlights tools like the Elasticsearch Migration Helper, which assists in transitioning from Elasticsearch 2.3.x to 5.0, and introduces the Cluster Allocation Explain API for better shard management. While this alpha version is not suitable for production, it allows users to explore changes and report issues, contributing to the development of the final release.

The Elastic Stack 5.0.0-alpha3 release introduces several new features and improvements across its components, including Elasticsearch, Kibana, Logstash, Beats, and ES-Hadoop, though it remains in the alpha stage and is not yet suitable for production use. Key enhancements in Elasticsearch focus on reducing ambiguity through refactoring, boosting indexing performance by minimizing lock contention, and improving dynamic scripting speed with Painless scripting. Kibana sees numerous bug fixes, stability improvements, and new PDF reporting features in X-Pack, while Logstash introduces a plugin generator tool and a unified configuration file, logstash.yml, for easier management. Beats expands its capabilities with Metricbeat, which extends metrics collection beyond Topbeat’s scope to include various servers, and simplifies configuration files for user-friendliness. Additionally, ES-Hadoop upgrades its Storm support to version 1.0.x, and the Migration Helper tool is available to assist users migrating from version 2.3.x to the upcoming 5.0 release.

In anticipation of the 5.0 release, which will remove several deprecated query types from version 2.x, the article explains how to effectively use the bool query to handle complex boolean operations in Elasticsearch. It details the transition from traditional queries like filtered, and, or, and not, to the bool query's fields: must, must_not, should, and filter, which correspond to AND, NOT, OR, and filtering operations respectively. The bool query allows for the nesting of multiple boolean expressions to accommodate complex logical conditions, demonstrated through examples involving student preferences for fruit snacks. The should parameter differs from a standard boolean OR as it allows setting a minimum_should_match value to specify how many clauses should match for a document to be returned. Additionally, the filter field replaces deprecated filtered queries, enabling refined document searches without affecting relevance scores. The article emphasizes the importance of understanding these changes and testing extensively in preparation for the upgrade to version 5.0.

Terradue, an ESA spin-off, is at the forefront of developing platforms and applications that aid researchers in extracting valuable information from the vast amounts of data generated daily by Earth observation satellites. By leveraging Elasticsearch technology, Terradue has created a query engine that efficiently selects optimal satellite acquisitions for InSAR applications, a technique used to map ground deformation via radar imagery. This capability was notably applied in assessing the aftermath of the 2015 Nepal earthquake. The process involves indexing extensive metadata linked to satellite imagery, encompassing acquisition properties, orbit state vectors, and tie points, allowing for rapid computation of satellite data pairings necessary for precise ground displacement measurements. Terradue's use of Elasticsearch facilitates real-time data processing, achieving significant speed and efficiency in querying and filtering satellite data, supporting geohazard research and broader Earth science applications. As Terradue celebrates its 10th anniversary, it continues to expand its cloud platform solutions, underscoring its commitment to advancing Earth sciences through scalable and innovative data processing technologies.

Windows event logs, particularly logon and logon failure events, offer crucial insights into the security of Windows-based infrastructures, and can be effectively monitored and visualized using Winlogbeat and the Elastic stack. Winlogbeat, a lightweight shipper, facilitates the collection and forwarding of event log data to Elasticsearch or Logstash, and its version 5.0 enhances data filtering and aggregation by shipping raw event data. By configuring Winlogbeat to track security events and applying specific filters in Kibana, users can identify abnormal logon patterns that might indicate compromised credentials or unauthorized access attempts. Monitoring failed logons can help detect unauthorised access attempts and infrastructure issues, with Windows using event ID 4625 for such occurrences. Moreover, visualizing the geographic origin of remote logons, especially when using Remote Desktop connections, can uncover anomalies like unexpected access from unfamiliar locations, necessitating further investigation. The integration of GeoIP filters in Logstash enriches event data with location information, allowing for map-based visualizations in Kibana, and requires specific configurations to enhance the Elasticsearch index template used for Winlogbeat data.

Elastic Graph, part of the commercial X-Pack plugins for the Elastic Stack, offers powerful analytical tools for examining complex data relationships, such as those found in the Panama Papers. By integrating with the Kibana app and a new Elasticsearch API, this technology simplifies the process of visualizing and exploring data connections, making it accessible for both forensic investigations and summarizing mass user behaviors. The blog post discusses how Elastic Graph can trace intricate networks of offshore financial arrangements and also highlights its capacity to mine collective intelligence from user behavior data, such as click logs, to generate actionable insights. The tool emphasizes identifying significant associations rather than popular but common links, using methodologies akin to TF-IDF for relevance ranking, a distinguishing feature compared to traditional graph databases. The article concludes by outlining the two primary usage modes of Graph: detailed forensic analysis, where every document holds potential importance, and broader wisdom-of-crowds analysis, which focuses on identifying significant patterns amidst widespread data noise.

Elasticsearch 5.0 introduces the ingest node feature, designed to provide a lightweight solution for pre-processing and enriching documents before indexing. This feature is integrated into all nodes by default, allowing users to create ingestion pipelines that transform data, such as converting string data types to integers and adjusting case sensitivity for keywords. The blog post, using the NEST client for .NET, illustrates how to set up a pipeline to handle tweets, converting retweets from strings to integers and uppercasing language codes. It also addresses error handling by incorporating on_failure processors to manage conversion errors, demonstrating how to automatically set retweets to zero if conversion fails. The post highlights the importance of considering dedicated ingest nodes for heavy ingestion tasks and suggests increasing timeouts for large bulk requests to prevent exceptions. Overall, the ingest node feature offers flexibility and efficiency in document processing within Elasticsearch, and the author encourages users to explore the official documentation for a deeper understanding.

Kibana patch versions 4.5.1 and 4.1.7 have been released, featuring an updated version of Node.js that addresses a high severity vulnerability in OpenSSL identified as CVE-2016-2107. Users who have configured Kibana to utilize TLS/SSL are strongly advised to upgrade promptly to ensure security. The latest versions are available for download on the downloads page.

Elasticsearch 2.3.3, based on Lucene 5.5.0, has been released as a bug fix update, urging all users of version 2.3.x to upgrade due to several important fixes and improvements. Notable changes include the resolution of a JVM crash issue when reading completion stats on a closed index reader, improved CORS handling for same origin requests with HTTP compression, and the refinement of wildcard usage for field highlighting to prevent exceptions with geo fields. Additionally, geo-shape CONTAINS relationships are now functioning correctly, and thread local leaks in certain web containers have been addressed. The Marvel component no longer logs exceptions for temporarily empty node stats, and a deprecated Shield REST endpoint has been updated. Users are encouraged to download the updated version, test it, and provide feedback through social media or the GitHub issues page.

Centralizing logs using open-source tools like rsyslog, Logstash, and Elasticsearch on Ubuntu 14.04 can significantly streamline log management and analysis. This process involves setting up a centralized rsyslog server to collect logs from multiple systems, which are then forwarded to Logstash for parsing and enrichment before being stored in an Elasticsearch server. The guide details each step, starting with configuring the rsyslog servers to send and receive log data, formatting logs into JSON, and configuring Logstash to handle these logs. The final step ensures Elasticsearch is correctly receiving data, enabling users to audit, monitor, and analyze server and application performance efficiently. With logs stored in Elasticsearch, users are encouraged to explore Kibana for data visualization, enhancing the utility of their log data through graphical representations.

Logstash is undergoing several updates and improvements as it approaches its 5.0 release. JSON logging is being reintroduced after resolving previous issues, and a new Java Event API has been integrated, allowing for a significant update of over 80 plugins to ensure compatibility. Efforts are also being made to improve release packages by aligning directory layouts with Elasticsearch, adopting upstart and systemd for managing services, and centralizing JVM options. The team is exploring the use of the Kryo library for event serialization and addressing a "resource leak" issue related to dynamic configuration reloading in collaboration with the JRuby team. Enhancements to RabbitMQ include reverting the SSL option to a boolean and adding certificate validation options. Other updates include removing feature flags for environment variable support, refining the list of plugins packaged with version 5.0, and setting Java 8 as a strict requirement for Logstash.

Apache Kafka and the Elastic Stack are key components in log and event processing, with Kafka often serving as a transport layer to manage large data volumes before they are processed by Elasticsearch for rapid search and analytics. The text outlines scenarios where Kafka proves beneficial, such as handling data spikes and ensuring data flow continuity during Elasticsearch downtime, by using a shipper and indexer architecture with Logstash. It discusses when not to use Kafka, highlighting the costs and management requirements involved, and suggests alternatives like Filebeat for situations with relaxed search latency needs. The text also provides design considerations for using Kafka with Logstash, including the importance of topics and partitions for data isolation and scalability, the role of consumer groups in enhancing fault tolerance, and the significance of efficient serialization formats to maintain performance. The use of multiple Logstash instances can increase processing capacity and ensure fault tolerance, with Kafka's ability to manage serialized message formats enhancing its adaptability in diverse data environments.

Jack Lawton, a Technical Consultant at Aiimi, explores the capabilities of the Elastic Graph, a plugin for Elasticsearch and Kibana, in analyzing Twitter data related to Donald Trump's political campaign. By using Logstash to collect tweets containing the term "Trump," Lawton demonstrates how the Kibana Graph plugin can visualize relationships between frequently used hashtags, revealing insights into social media trends. He highlights the advantage of categorizing data by different fields, such as time zones, to identify regional hashtag trends. Lawton also addresses challenges like data repetition, using features like the diversity field setting to refine analyses. Additionally, he explains how Graph helps filter out common terms to focus on significant links, showcasing its utility in distinguishing meaningful connections. The blog underscores Graph's integration with the Elasticsearch API, allowing for custom applications, and invites readers to view a webinar for further insights into Graph's application to various datasets.

Elastic is actively participating in and supporting a wide array of events and meetups across the globe, with a focus on DevOps, coding, and technology from May 9 to May 20, 2016. In Europe, events include Baltic DevOps, Codemotion Amsterdam, DevOps Days Kiel, InfoShare in Poland, and Chemnitzer Linuxtage in Germany. North America features the Community Leadership Summit, Oscon in Austin, and the Nebraska Code Conference. Asia is hosting the Database Technology Conference in China, and various meetups are planned in cities like San Francisco, Minneapolis, Austin, Toronto, Los Angeles, Copenhagen, Munich, Bielsko-Biała, London, Beijing, and Mexico City. Elastic encourages community involvement, offering support and promotional items to those interested in hosting meetups or giving talks on their technologies like Beats, Elasticsearch, Logstash, or Kibana.

Elasticsearch 5.0.0-alpha2, based on Lucene 6.0.0, is a pre-release version designed for testing and feedback rather than production deployment, with indices created in this version being incompatible with the final 5.0.0 GA release. This release, building on the previous alpha1 version, introduces several new features and improvements such as faster indexing and range queries due to the use of dimensional point fields, the replacement of the percolate API with a percolate query for enhanced search functionality, and the implementation of stored scripts/templates to simplify index recovery. It also introduces safety measures, including checks for JVM heap size and virtual memory access, alongside a new circuit breaker to manage the size of all in-flight requests, preventing system overloads. Migration tools like the Elasticsearch Migration Helper are available to assist users in transitioning from Elasticsearch 2.3.x to 5.0, offering functionalities such as cluster checkups, reindexing of old indices, and deprecation logging.

On May 3, 2016, Kibana 5.0.0-alpha2 was released, marking a significant milestone for the software and its development community. This alpha version, specifically compatible with Elasticsearch 5.0.0-alpha2, introduced several new features and bug fixes, including the renaming of Sense to Console, which now supports the ES 5.0 APIs, and improvements like configurable header whitelists for requests, millisecond support for interval durations, and persistent UUID generation for cluster instance distinction. Additionally, various bugs were addressed, such as fixing extraneous bounds on tilemaps, honoring custom labels on metrics, and enhancing dark theme support. The announcement hinted at the upcoming Kibana 5.0.0-beta1 release, inviting users to test the alpha version and provide feedback through forums, IRC, or social media channels.

Elastic Stack 5.0.0 alpha 2 introduces several updates across its components, with notable enhancements in Elasticsearch, Kibana, Logstash, Beats, and X-Pack. Elasticsearch now includes the percolate query and native IPv6 support, while Kibana has revamped its interface by renaming "Sense" to "Console" and updated its capabilities. Logstash can now utilize MaxMind's GeoIP2 database for improved GeoIP filtering and supports Elasticsearch's ingest pipelines for better data processing. Beats has improved its Redis output and standardized its directory layout to align with Elasticsearch, also offering sample Kibana dashboards. X-Pack streamlines its settings with a new prefix and introduces built-in users and a superuser role for enhanced security management. Additionally, ES-Hadoop v5.0.0-alpha2 has been released, further expanding the capabilities of the Elastic Stack.