March 2016 Summaries

Elasticsearch 2.3.0 and 2.2.2 have been released, featuring significant updates and new features designed to enhance user experience and prepare for future versions. Elasticsearch 2.3.0 introduces the highly requested reindex API, update-by-query API, and task management API, facilitating easier data reindexing, document updating, and task monitoring or cancellation, respectively. These additions are aimed at simplifying complex operations like reindexing terabytes of data and managing long-running tasks. The update also includes deprecation logging to help users identify outdated features that will be removed in Elasticsearch 5.0. Alongside these Elasticsearch releases, new versions of Kibana, Logstash, and Beats have been launched, all available on Elastic Cloud. Users are encouraged to explore these enhancements and provide feedback through various channels.

Kibana 4.5.0 has been released, featuring new updates that align with the latest advancements in Elasticsearch, requiring at least version 2.3.0. The release, which is relatively smaller as the team focuses on preparing Kibana 5.0, includes enhancements such as the ability to assign labels to metric aggregations, automatic clearing of errors after five minutes, customizable display of saved objects, and the listing of installed plugins via the command line interface. Bug fixes address issues like pre-flight request errors, improper handling of legend values with special characters, and usability improvements in areas such as area chart highlighting, color palette usage, and timeouts in Elasticsearch communication. This version is available on Elastic Cloud and for download, promising users a more stable and efficient experience.

In "Where in the World is Elastic?", Luisa Antonio and Megan Wieling provide an overview of upcoming Elastic events and meetups around the world over a two-week period, highlighting the various locations and dates of these gatherings across North America, Europe, Africa, Asia, and South America. The events include Code PaLOUsa in Louisville, KY, and Lone Star PHP in Dallas, TX, alongside numerous Elastic meetups in cities like Portland, Mountain View, Minneapolis, and international locations such as Amsterdam, Lisbon, and Seoul. The authors also invite readers to host meetups or speak about Elastic-related topics, offering support and promotional items as incentives.

Elasticsearch introduces two new features in its updates: _reindex and _update_by_query, designed to enhance data management capabilities. _reindex allows users to copy documents from one index to another, enabling document enrichment or index recreation to modify locked settings at creation. It facilitates both straightforward document copying and more complex operations like selective copying based on specific tags or adding new tags during the process. On the other hand, _update_by_query updates fields within documents in the same index and can accommodate online mapping changes, useful for tasks like adding new searchable fields. Both features, capable of handling millions of documents, can be monitored for progress and canceled if required, although cancelation doesn't roll back changes. While these features are powerful, they don't optimize resource efficiency and are better suited for transforming existing data rather than updating data post-ingestion.

Elastic is hosting a variety of events and meetups across different regions, including Europe, North America, and Australia, to engage with the community and promote its technologies. Notable events this week include BreizhCamp in France from March 23-25 and several Elastic user group meetups in cities such as San Francisco, Seattle, Los Angeles, Austin, Boston, London, and Sydney. The company encourages community involvement by offering support for those interested in hosting meetups or giving talks about their products like Beats, Elasticsearch, Logstash, or Kibana and is willing to provide promotional materials.

Elastic Cloud, the official hosted Elasticsearch service by the creators of Elasticsearch, Kibana, Beats, and Logstash, offers a convenient solution for companies and individuals seeking to avoid the complexities of self-hosting the Elastic Stack. It ensures a seamless experience by providing the latest version updates, world-class technical support, security features, alerting, and monitoring capabilities. Attendees of Elastic{ON} 16 in San Francisco had the opportunity to explore Elastic Cloud's architecture and history through presentations and live demos by Elastic engineers, highlighting its ability to manage real-time search and analytics efficiently. Elastic Cloud caters to diverse requirements, offering scalable resources from 1GB of memory to 4TB of storage, and can be trialed with a free 14-day offering, making it an attractive option for those looking to simplify their Elasticsearch needs.

As cyberattacks grow increasingly sophisticated, the need for effective detection and response systems is paramount, with the Bro Network Security Monitor and Elastic Stack offering a robust solution. Bro, an open-source network security monitor, inspects real-time and historical network traffic, logging detailed data that can be managed and analyzed with the Elastic Stack, which comprises Elasticsearch, Logstash, and Kibana. These tools work together to collect, normalize, store, and visualize log data, enhancing it with threat intelligence feeds to identify and mitigate threats swiftly. Through detailed configurations, Logstash can enrich log data with geolocation information and integrate threat intelligence, while Kibana provides visualizations to contextualize the data for business insights. This setup not only addresses the detection deficit highlighted in cybersecurity reports but also empowers organizations to detect and respond to threats more efficiently, reducing the risk of data breaches.

Elastic{ON} 16 proved to be a rewarding experience for Doug Turnbull, a search relevance consultant at OpenSource Connections, as it offered high-quality talks and fruitful discussions that sparked new ideas and collaborations. Among the highlights were presentations on BM25 text scoring, Eventbrite's use of Elasticsearch for recommender systems, and a demonstration of graph search by Mark Harwood. The event facilitated productive conversations with engineers from major companies such as Apple, Facebook, and Roku, providing a platform for troubleshooting and sharing insights about the Elastic Stack. As a speaker, Doug appreciated the interactive and supportive environment, which included thorough preparation by Elastic to enhance his presentation skills. His talk, "The Ghost in the Search Machine," delved into innovative uses of Elasticsearch's inverted index, and he invites further engagement at upcoming Elastic meetups.

Elasticsearch 2.2.1, built on Lucene 5.4.1, has been released as a bug fix update and is available on the Elastic Cloud platform. This update addresses several key issues, including the Tribe node's failure to pass relevant settings to client nodes, Marvel's inability to export data to remote monitoring clusters when Shield was enabled, the _aliases endpoint ignoring configured filters, and the GCE plugin failing due to insufficient privileges. Users experiencing these issues are encouraged to upgrade and provide feedback via Twitter or the forum, with any problems being reportable on the GitHub issues page.

Elastic is hosting a series of events and meetups across the globe, focusing on its open-source search and analytics engine. Notably, the FOSSASIA 2016 event in Singapore from March 18-20 will feature a workshop on Elasticsearch by Medcl Zeng and three LARC Engineers. In North America, the Great Wide Open conference in Atlanta on March 16-17 will include a talk by Shaunak Kashyap on "Elasticsearch for SQL Users." Additional meetups are scheduled across Asia, Australia, Europe, North America, and South America, covering various topics related to Elastic's technology. The Elastic team encourages community engagement and offers support for those interested in hosting meetups or giving talks on their products such as Beats, Elasticsearch, Logstash, or Kibana.

In March 2016, Kibana released versions 4.4.2, 4.3.3, and 4.1.6, primarily to address security vulnerabilities related to the OpenSSL CacheBleed issue and fix memory leak problems when connecting to Elasticsearch over SSL. These releases included several bug fixes and improvements, such as enhancements to the plugin installer for .tgz file types, adjustments to handle HTML unsafe characters in field names, and updates to distro package metadata. Notable changes across versions included a node version upgrade to v4.3.2 and various user interface and functionality improvements, ensuring a more stable and secure experience for Kibana users.

The article by Robin Moffatt explores how to leverage the Elastic Stack, comprising Elasticsearch, Logstash, and Kibana, to visualize Oracle database performance data, specifically focusing on Active Session History (ASH) data. The Elastic Stack offers an open-source, flexible solution for diagnosing Oracle database performance issues by allowing users to stream, store, and analyze data in a customizable way. By using Logstash's JDBC input plugin to pull data from Oracle's V$ACTIVE_SESSION_HISTORY table, the data is streamed into Elasticsearch for storage and analysis with Kibana. The article details the setup process, which includes configuring Logstash to poll Oracle data continually and sending it to Elasticsearch while resolving any data type issues that arise. Kibana is then used to create visualizations and dashboards, enabling interactive analysis of Oracle performance data. The Elastic Stack's ability to integrate seamlessly and scale horizontally makes it a powerful tool for monitoring and diagnosing database performance, as demonstrated through various examples and visualizations.

Elastic is actively participating in a range of events and meetups across Europe and North America, as detailed in the March 7, 2016 update. Notable events include Big Data Paris, where Elastic will be present at Stand 705 to answer questions and distribute promotional items, and Javaland 2016 in Brühl, Germany, where a featured talk titled "JVM Deep Dive" by Daniel Mitterdorfer is scheduled. The week also features several meetups in cities such as Vienna, Basel, Amsterdam, and Novi Sad in Europe, along with Atlanta and Seattle in North America. Elastic encourages community engagement by offering support and promotional items to those hosting meetups or delivering talks related to their products, which include Beats, Elasticsearch, Logstash, and Kibana.

Gabriel Moskovicz's article explores the complexities of phrase queries in Elasticsearch, emphasizing the importance of understanding both the indexing and querying processes to effectively retrieve desired data. It highlights the role of stopwords—common words often excluded from searches—and their impact on query results. The article demonstrates how Elasticsearch analyzes text by tokenizing and normalizing terms, which affects how phrases are matched. Through examples, Moskovicz illustrates how the position and order of words in a phrase can lead to unexpected query results if not properly understood. Ultimately, the article underscores the need for a comprehensive approach to indexing and querying, advocating for a smart analysis process that aligns with user needs to achieve accurate search outcomes.

1&1 Internet SE's Operations team has significantly enhanced its process transparency and error categorization using the Elastic Stack, comprising Elasticsearch, Logstash, and Kibana, alongside ETL tools like Pentaho Kettle. The team manages approximately 50,000 BPMN process instances weekly, generating about 50GB of logs daily, which the Elastic Stack allows them to analyze in real-time. This system provides vital insights into customer orders and supports the maintenance of process SLAs, like timely order provisioning. To comprehensively understand process errors, additional data from sources such as issue trackers is integrated, leading to a more efficient error detection and resolution process, improving the addressing rate from 60% to over 90%. The Kibana dashboard facilitates this improvement by offering a centralized view of errors across processes, linking to various tools for deeper analysis. Looking ahead, the team plans to integrate Apache Spark for real-time monitoring and KPI computation, enhancing order clustering and campaign estimation.

The text discusses the evolution of the Logstash File Input plugin, detailing its functionality and improvements over time, particularly in versions up to 2.2.0. Initially designed to handle continuously updated files, the file input also attempted to accommodate static file uploads, utilizing the sincedb for tracking read progress. Challenges such as file handle exhaustion and identity mapping led to the introduction of new configuration options like `ignore_older` and `close_older`, and the development of a tracking object approach to manage file states. The plugin allows for multiple file inputs but faces limitations with a 20,000 identity cap. The text compares Logstash File Input to Filebeat, noting Filebeat's advantages in handling log file data. It also outlines future enhancements needed for the file input, emphasizing the need for an improved sincedb and the creation of a new `read_file` plugin to better handle the read use case, with features like prioritizing file activation and supporting gzip decompression.

Upgrading to Logstash 2.2 involves significant performance improvements and changes to the threading model, which now unifies filter and output worker threads into a single 'pipeline' worker thread type. This new architecture can lead to increased use of file handles, and requires tuning of the -w setting to optimize performance, potentially involving more threads than before but with improved efficiency due to less context switching. The introduction of a new batch size setting -b affects the default flush size for the Elasticsearch output, with the flush_size option now only setting the maximum flush size. However, an issue with Elasticsearch output workers not sharing the same backend connection pool can lead to an excessive number of connections, necessitating a temporary workaround by reducing the number of Elasticsearch output workers until a patch is released. Users are encouraged to provide feedback on these changes through various channels including Twitter, forums, and GitHub.