April 2015 Summaries

On April 29, 2015, Jay Modi announced the release of bugfix updates for Shield, specifically versions 1.2.1 and 1.1.1, which address several issues, particularly ensuring the proper functioning of Disk-based Shard Allocation on clusters secured by Shield. Users are encouraged to download the new installation or follow the provided upgrade instructions to maintain the health and security of their clusters.

This weekly roundup from April 2015 highlights updates and events related to the ELK Stack, including Elasticsearch, Logstash, and Kibana. It features the release of Elasticsearch versions 1.5.2 and 1.4.5, which addressed a directory traversal vulnerability, and the introduction of search capabilities across multiple models in the Ruby on Rails integration library. The roundup also includes insights into Verizon's use of Elasticsearch for handling 20 billion documents a day, community contributions like Nathan LeClaire's guide on automating Docker logging with the ELK stack, and Robin Moffat's analysis of open data using Kibana 4. It outlines various presentations and meetups worldwide, such as those in Algiers, Melbourne, Vienna, Paris, and several cities in the United States, offering insights into topics like securing Elasticsearch, scaling Logstash, and real-time search analytics. Additionally, training sessions led by core developers on Elasticsearch are scheduled in cities across the globe, from Copenhagen to Bangalore, providing opportunities for hands-on learning.

Elasticsearch has released versions 1.5.2 and 1.4.5 to address a critical directory traversal vulnerability, which allows attackers to access files on servers with installed site plugins. This vulnerability, identified as CVE-2015-3337, does not affect the core installation but is triggered by site plugins like Marvel and others, with the fix requiring users to upgrade or disable site plugins. Noteworthy improvements in these releases include enhanced handling of indexed scripts and templates, geo-shape precision fixes, better shadow replica resilience, and important back-ported changes to version 1.4.5 for faster shard recovery and improved translog handling. Users are encouraged to upgrade and share feedback on their experiences via Twitter or GitHub.

Logstash 1.5.0 RC3 has been released, addressing community feedback on performance regressions and other critical issues noted since RC2. The key improvements include enhancing throughput performance, particularly in configurations with conditionals in filters and outputs, by optimizing code generation. Additionally, bugs related to shutdown issues and Java options have been fixed, allowing users more flexibility with JVM settings. The update also addresses problems with the Grok filter, specifically the creation of empty fields due to previous performance adjustments, and resolves permission and corruption issues involving the sinceDB file position during Logstash shutdown. Users are encouraged to test this release candidate and provide feedback through GitHub or Twitter as the final release of 1.5.0 approaches.

The weekly roundup "This Week in Elastic" provides updates on the ELK Stack, including developments in Elasticsearch, Logstash, and Kibana, and shares useful resources like blog posts, tutorials, and articles. The post highlights events such as meetups and conferences worldwide, where enthusiasts and professionals can learn more about the ELK Stack, including Elasticsearch's storage requirements, scaling techniques, and integration with various technologies like Drupal and Silex. Additionally, it features insights into upcoming releases, tips for deploying ELK clusters, and offers a look at training sessions provided by core developers in various global locations. The article also emphasizes the community's efforts to bring together Elasticsearch users through numerous meetups and provides information about job opportunities within the Elastic company.

This blog post, co-authored by Christian Dahlqvist and Peter Kim, explores the storage requirements for Elasticsearch through a series of tests analyzing different configurations. The authors discuss how deployment sizes can vary significantly based on use cases, and they emphasize the importance of testing with representative data to estimate disk needs accurately. Key factors that influence disk space include whether fields are analyzed or not, the use of the "_all" field, and the enablement of doc values, each affecting how data is indexed and stored. The post highlights that the storage footprint can range widely depending on these settings, with structured data generally allowing for better compression than semi-structured data. The authors stress the importance of configuring Elasticsearch mapping intentionally, especially for larger deployments, as this can impact storage needs significantly. They also note that while Elasticsearch employs compression, it is designed to minimize query latency. The blog concludes by acknowledging common misconceptions about disk space in ELK-based solutions and encourages further exploration of hardware requirements beyond just storage.

This week in Elastic offers a comprehensive update on the ELK Stack, focusing on Elasticsearch, Logstash, and Kibana, along with related events and training opportunities worldwide. The newsletter highlights the release of Elasticsearch 1.5.1, which includes important bug fixes, and features guest posts and tutorials on scaling Elasticsearch and its integration within the Apache Hadoop ecosystem. The document also mentions numerous meetups and conferences across the globe, where enthusiasts and developers can explore the latest innovations and applications of Elasticsearch, including its use in civic hacking and government analytics. Additionally, it promotes upcoming training sessions in various cities, providing hands-on workshops and core Elasticsearch courses facilitated by the core developers, and invites community members to participate in events and share their experiences with the ELK Stack.

On April 9, 2015, the release of Elasticsearch 1.5.1 was announced, providing a stable version based on Lucene 4.10.4, which includes important bug fixes and improvements. A key update addresses the speed of allocating a shard to a new node during recovery by enabling merges on the target shard, thereby preventing the accumulation of small segments and enhancing recovery speed. Additional fixes involve improving the handling of repositories with numerous snapshots and forcing a refresh when deletes fill up the version map. Several experimental inner hits feature issues were also resolved. Notably, rivers have been deprecated in this version, and users are encouraged to download, test, and provide feedback on the release.

This weekly roundup offers insights into the ELK Stack ecosystem, highlighting developments in Elasticsearch, Logstash, and Kibana, including video tutorials, blog posts, and articles. Key updates include the release of the third installment of Kibana 4 tutorials, a detailed blog on Apache Lucene codecs, and the upcoming book "Taming Search" about search relevance. The roundup also features various global meetups and conferences, such as the Sydney ELK Meetup, Hadoop Summit in Brussels, and .NET Fringe conference in Portland, with presentations and workshops focusing on Elasticsearch's integration with other technologies. Additionally, there are opportunities for training sessions in several cities worldwide, and the post encourages community engagement by inviting readers to feature their own ELK Stack events and presentations.

Apache Lucene utilizes codecs as a core abstraction to manage the reading and writing of index files, which enables efficient data storage and retrieval without exposing the intricate details of data structure management. A codec is an instance of the org.apache.lucene.codecs.Codec class, identified by a unique string name and providing specific format classes for different parts of Lucene's index, such as stored fields and term vectors, each with its own reading and writing APIs. This abstraction not only facilitates experimentation and innovation in index file formats but also supports backwards compatibility by retaining older codecs for reading previous index versions. While codecs are crucial for performance and functionality, most users rely on the default codec, though developers can customize and test new codecs within Lucene's pluggable infrastructure. Experimental formats, while not backward-compatible or suitable for production use, often introduce features that eventually integrate into the default formats, highlighting Lucene's commitment to continual enhancement and adaptability.

The blog post provides an update on the developments in Elasticsearch and Apache Lucene, highlighting various enhancements and fixes across both platforms. Key updates for Elasticsearch include improvements to core networking, logging configurations, aggregation functionalities, snapshot/restore processes, and index expansion, as well as the removal of unsafe options and the addition of doc values support to boolean fields. Meanwhile, Apache Lucene sees advancements with the release branch of version 5.1.0, improvements in span queries, a new NRT suggester, and a faster query-time join based on global ordinals, alongside efforts to make queries immutable and refine the AutoPrefixPostingsFormat for efficient range and prefix queries. The post encourages readers to follow the blog for more information and updates on the ELK ecosystem, including Elasticsearch, Logstash, and Kibana.

This weekly roundup from Elastic covers updates and news about the ELK Stack, including new releases and feature deprecations. Kibana 4.0.2 has been released with enhancements and bug fixes, while the Rivers feature in Elasticsearch is being deprecated. The article highlights various tutorials, tips, and community stories, such as Inbot's use of Elasticsearch for CRM systems and The Guardian's Elasticsearch upgrade. It also lists numerous global events and meetups focused on ELK Stack topics, such as conferences in Australia, Belgium, Canada, and the United States, with sessions ranging from real-time analytics to geospatial search. Additionally, it mentions available Elasticsearch training courses across different cities worldwide, providing opportunities for users to deepen their knowledge and skills in the ELK ecosystem.

In a recent update from Elastic, it was announced that the Rivers feature in Elasticsearch is being deprecated, with full details and replacement recommendations available in Shay Banon's blog post. This week's developments in Elasticsearch and Apache Lucene include various updates and fixes, such as enhancements to the cat API, bug fixes in mappings and aggregations, and adjustments to scripting settings. Several plugins for Elasticsearch were released, including updates to language, river, and transport plugins. Apache Lucene is preparing for a 5.1.0 release with improvements like the extension of PrefixQuery and the introduction of a near-real-time document-based suggester. Additionally, Elastic's blog will continue to provide updates on the ELK ecosystem, offering insights into new features and use cases.