July 2015 Summaries

Kibana introduced Pinned Filters in version 4.1, allowing users to create filters in one part of the platform and carry them across different applications within Kibana, enhancing data exploration and analysis. This feature, inspired by a user suggestion, enables seamless transition of filters through a simple 1-click operation, where users can pin filters by selecting them and clicking a pin icon, making it easy to identify and manage pinned filters. By using the Actions menu, users can pin multiple filters efficiently, and these pinned filters persist when users switch between applications like Discover and Visualize, allowing them to explore data more effectively by maintaining context across different views.

Packetbeat is a tool designed to capture and analyze network data, sending the processed information to Elasticsearch for visualization in Kibana, and it is particularly suited for monitoring protocols such as HTTP, PostgreSQL, MySQL, Redis, and Thrift. Unlike Logstash, which focuses on log data, Packetbeat specializes in network traffic, although both tools can feed data into Elasticsearch. Installation and configuration of Packetbeat involve downloading the appropriate package, adjusting the packetbeat.yml file to set parameters such as server name, tags, and network interfaces, and specifying the protocols and ports to monitor. Packetbeat can also handle sensitive data by using parameters like hide_keywords and real_ip_header, and supports multiple output options, with Elasticsearch being the most common. The tool allows for visualization through Kibana, though it requires users to create custom dashboards as it does not include any by default.

On July 29, 2015, Jay Modi announced the release of Watcher 1.0.1, a bugfix update that addresses compatibility issues with Elasticsearch versions 1.6.1 and 1.7.2, released on the same day. This update introduces an enhancement allowing dynamic index names to specify a time zone for computing index names, with UTC as the default, whereas previously it was fixed to UTC. Users are encouraged to upgrade by following instructions in the Watcher documentation and are invited to provide feedback or ask questions in the Watcher category of the forums.

Elasticsearch has released versions 1.7.1 and 1.6.2, which include critical bug fixes aimed at addressing a rare but significant issue that could result in data loss when multiple simultaneous node failures or restarts occur, potentially deleting all copies of a shard from the cluster. This bug, identified as #12487, was initially introduced in version 1.5.0. Besides the shard deletion issue, the updates also resolve problems related to CIDR mask conversion for IPv4 addresses and allow Shield users to utilize the more-like-this API. Users are encouraged to upgrade to these versions and provide feedback through social media or the GitHub issues page.

Released in its 1.0.0-beta2 version, Packetbeat now supports the MongoDB wire protocol, enabling users to monitor MongoDB server performance by capturing network traffic and correlating requests with responses, which are then documented in Elasticsearch. This integration allows for the visualization of metrics such as MongoDB query performance and error detection using Kibana dashboards. The open-source nature of Packetbeat encourages community contributions, exemplified by Alban Mouton's addition of MongoDB support shortly after Packetbeat joined Elastic. Packetbeat's ability to decode the MongoDB protocol, even within clusters, offers advanced troubleshooting capabilities, although it currently only supports the newer version of the MongoDB protocol post-2.6. By utilizing Kibana dashboards, users can explore visualizations that highlight errors, response times, and query counts, providing insights into performance trends and system operations. Packetbeat's flexibility is enhanced by its configuration options, allowing for payload indexing to avoid capturing sensitive data inadvertently, thus catering to users' specific monitoring needs.

A common issue faced by users running Elasticsearch at scale is the Java heap pressure caused by fielddata, a data structure that inverts the inverted index to facilitate sorting and aggregations but consumes significant memory. Fielddata is loaded on demand, leading to increased memory usage and potential instability as segments are added. Although the Fielddata Circuit Breaker can block requests that exceed available memory, it does not clear existing fielddata. To address this, Elasticsearch recommends using doc values, which store fielddata on disk at index time, reducing heap memory usage and improving performance. Doc values are not compatible with analyzed strings, but multifields can be used to manage both analyzed and un-analyzed strings. Elasticsearch 2.0 aims to set doc values as the default for all fields, except analyzed strings, to mitigate this problem.

Shield 1.3.1 and 1.2.3 have been released as bug fix updates to address a serialization issue in Shield 1.3.0 and 1.2.2, which affected message authentication and hindered rolling upgrades when transitioning from Shield 1.2.1. The bug caused compatibility issues, resulting in "tampered signed text" errors during upgrades involving nodes with different versions. Users who have not upgraded to the problematic versions can proceed with a straightforward upgrade to the latest versions, while those using message authentication in Shield 1.2.2 or 1.3.0 must perform a cluster restart upgrade. This involves uninstalling the existing plugin and installing the new one on each node, following the detailed instructions provided. Feedback on these updates is encouraged through the Shield category in the forums.

"Where in the World is Elastic?" highlights upcoming events and meetups for Elastic, with a focus on North America and Europe from July 20-26, 2015. Elastic's presence at OSCON in Portland, Oregon, includes a booth and a talk by Leslie Hawthorn on avoiding self-sabotage in success, while in Europe, Honza Kral will discuss advanced Elasticsearch topics at EuroPython in Bilbao, Spain. The schedule includes various meetups across North America, Europe, and the rest of the world, such as in Seattle, New York, Dallas, London, Hamburg, Tel Aviv, and Japan. The team encourages participation and offers support for those interested in hosting or speaking about Elastic products like Beats, Elasticsearch, Logstash, or Kibana.

This edition of "This Week in Elastic" highlights the latest developments in the ELK Stack ecosystem, including recent releases and enhancements, such as improved query execution in Elasticsearch 2.0 and a new logstash-input-http plugin. The post also details various meetups, conferences, and workshops worldwide where Elastic's products are being discussed, offering attendees insights into Elasticsearch, Logstash, and Kibana through presentations, live demos, and real-world use cases. Additionally, the community is invited to engage with Elastic's offerings through training sessions and developer hangouts, while opportunities for employment at Elastic are encouraged irrespective of geographical location.

Elasticsearch 1.7.0 and 1.6.1 have been released, featuring important security fixes and improvements to cluster stability and recovery. The releases include fixes for a remote code execution vulnerability and a directory traversal vulnerability, with recommendations for securing systems for those who cannot upgrade. Elasticsearch 1.7.0 introduces delayed shard allocation, allowing clusters to wait for a node to rejoin before reallocating shards, thereby reducing unnecessary load. Additionally, it prioritizes index recovery based on settings like index priority, creation date, and name, ensuring more recently created indices are recovered first after events like a full cluster restart. This release marks the final feature update for the 1.x series, with future developments focusing on Elasticsearch 2.0 or later. Users are encouraged to download the new version, provide feedback, and report any issues.

Elasticsearch 2.0 introduces significant internal changes to improve query execution by merging queries and filters into a single object that can score documents or skip scoring, thus optimizing performance and consistency. This update, largely influenced by changes in Lucene 5.0 to 5.2, deprecates the filtered query in favor of a new filter clause and unifies conjunctions into the bool query, allowing for robust optimizations through two-phase execution. The two-phase iteration pattern enables better handling of complex queries, such as geo-distance and phrase queries, by separating approximation and verification phases to enhance efficiency. Additionally, Elasticsearch 2.0 implements smarter, automatic filter caching to avoid over-caching, prioritizing the reuse of filters and introducing sparse bit sets to make filter caching more memory-efficient. Multi-term queries can now be rewritten into bool queries when matching few terms, improving intersection efficiency, while span queries benefit from the two-phase iteration, with new span-within and span-containing queries enhancing their capabilities. These advancements collectively enable Elasticsearch to autonomously optimize query execution without requiring manual adjustments from users.

"Where in the World is Elastic?" highlights Elastic's upcoming events and meetups scheduled for July 2015, with a focus on various global locations including North America, Europe, and Asia Pacific. The events feature opportunities for participants to engage with Elastic's team, particularly at DevOps Days in Melbourne, Australia, where discussions around the ELK stack are anticipated. The listed meetups span cities such as Denver, Seattle, San Francisco, Salt Lake City, Zurich, Berlin, Osaka, and Kyoto, indicating a robust schedule aimed at fostering community engagement and knowledge sharing. The Elastic Team also extends an invitation for individuals interested in hosting meetups or speaking about Elastic-related technologies like Beats, Elasticsearch, Logstash, or Kibana, offering support and promotional materials.

In the blog post "Out of this world aggregations," Colin Goodheart-Smithe introduces Pipeline Aggregations in Elasticsearch 2.0, a feature that extends the existing Aggregations framework to allow additional computations on aggregation results. These new aggregations enable users to perform calculations like determining the maximum average monthly price or tracking daily user sign-ups. The post demonstrates the use of various types of Pipeline Aggregations, such as Derivative, Max Bucket, Min Bucket, and Bucket Script Aggregations, using spacecraft trajectory data from NASA’s Helioweb site for Voyager 1 and Voyager 2. By plotting the radial distance and speed of Voyager 1, the post illustrates how these tools can reveal insights, such as changes in radial speed during close encounters with celestial bodies. The article highlights how pipeline aggregations can be chained to calculate additional metrics, like radial acceleration, and invites feedback on the current implementations while suggesting potential future enhancements.

This Week in Elastic provides a comprehensive update on the latest developments, events, and resources related to the ELK Stack, including Elasticsearch, Logstash, and Kibana. Highlights include the release of Curator 3.2.0, which supports synced flushes, and a deep dive into store compression improvements in Elasticsearch and Lucene. Noteworthy blog posts and tutorials are shared, such as Automattic's successful transition from MySQL to Elasticsearch, resulting in significant performance gains. The newsletter also outlines numerous global meetups, conferences, and training opportunities, demonstrating the widespread community engagement and interest in the ELK Stack. Additionally, the Elastic team is actively participating in various international events like DevOps Days in Melbourne and EuroPython in Bilbao, showcasing their commitment to fostering community interaction and knowledge sharing.

The article "Where in the World is Elastic?" provides an overview of various upcoming events and meetups involving the Elastic team across different regions. In North America, Elastic will be present at DevOpsDays in Minneapolis and CodeStock in Nashville, where Shaunak Kashyap will discuss Elasticsearch's capabilities. In Europe, Elastic will participate in the Java Forum in Stuttgart, Germany, sharing demos and insights at their booth. Africa will host a talk in Johannesburg on "StaticElastic," focusing on Elasticsearch and Backbone SPA. Additionally, a series of meetups are scheduled in cities like Minneapolis, Silicon Valley, Los Angeles, Amsterdam, and Tel Aviv. The Elastic team encourages community engagement by offering support to those interested in hosting meetups or talks related to their technology stack.

In 2011, Lucene's document store lacked compression, leading to increased storage sizes, but the release of Lucene 4.0 in 2012 introduced a codec API that facilitated experimentation with file formats and ensured backward compatibility. This allowed for significant changes to the index format, including automatic document store compression using LZ4 in Lucene 4.1, which efficiently compresses short documents by grouping them into 16KB blocks. Lucene 5.0 further improved compression by allowing the use of DEFLATE, which provides better compression but at the cost of stored field performance, making it particularly beneficial for users with large data volumes. Additionally, these updates enable better management of hot and cold data, allowing older indices to be stored on cheaper machines with enhanced compression to save disk space. Furthermore, Lucene 5 introduced improved merging of stored fields, overcoming previous CPU-bound issues during merges by tracking and managing incomplete compressed blocks. These advancements are set to be available in Elasticsearch 2.0, highlighting the ongoing evolution of store compression in Lucene and Elasticsearch.

This Week in Elastic, dated July 1, 2015, provides a comprehensive overview of updates and events related to the ELK Stack, which includes Elasticsearch, Logstash, and Kibana. The roundup highlights new features and improvements, such as the release of Watcher 1.0 and Kibana 4.1.1, and the upcoming Elasticsearch 2.0. It includes insights into Elasticsearch logging through Andrei Stefan's article and mentions a tutorial by Jettro Coenradie on Watcher. The post shares information about various meetups and conferences around the world, where members of the Elastic community can learn and share their experiences, and it also promotes upcoming Elasticsearch training sessions in multiple global locations. Additionally, the post invites readers to subscribe to Elastic's weekly developer newsletter for more tutorials and live developer hangouts.