August 2016 Summaries

Kibana 4.6.0 introduces several significant enhancements, including the much-requested Reporting plugin, which allows users to generate PDF reports from Kibana visualizations and dashboards. This release also ensures compatibility with the latest Elasticsearch version 2.4.0, facilitating seamless Elastic Stack upgrades. Notable improvements in Linux packages make installing, upgrading, and using Kibana smoother, especially for those upgrading from version 4.5.4, with packages now offering better cleanup, version-based repositories, and improved logging. Additional features include options to configure headers for Elasticsearch requests, a designated data space for plugins, and enhanced status page information. Various bug fixes, such as improvements to error notifications and visualization editor controls, further refine the user experience, while deprecated features like sorting terms aggs by ascending _count are being phased out to enhance accuracy in future releases.

Elasticsearch 2.4.0, based on Lucene 5.5.2, has been released, introducing key enhancements such as a significant increase in indexing throughput due to a change in the locking strategy and addressing the previously problematic use of dots in field names. While the 2.0 update had prohibited dots in field names due to potential conflicts, Elasticsearch 5.0 aims to reintroduce this feature with improved internal mapping that better supports such structures. Users of version 1.x can now upgrade to 2.4.0 by disabling the restriction on dots, allowing for smoother transition to 5.0, although care must be taken to avoid field conflicts that could hinder future upgrades. Additionally, Elasticsearch offers solutions such as reindexing in 2.x or using the reindex-from-remote feature to facilitate direct upgrades from 1.x to 5.0, underscoring the importance of reindexing to ensure compatibility with the latest version. The release is available for deployment on Elastic Cloud, and users are encouraged to provide feedback via social media or GitHub.

Reporting 2.4.0 is a newly released product that enables users to generate PDFs of Kibana searches, visualizations, and dashboards, offering a convenient way to capture and share data insights. Initially set for a later release with version 5.0, it has been made available earlier as a plugin for Kibana 4.6 due to anticipation from users. The Reporting plugin integrates seamlessly into Kibana, allowing report creation from open items and supports automation through HTTP requests or integration with Watcher for triggering reports based on events or schedules. It operates asynchronously, utilizing Elasticsearch as its data source and enabling distributed processing across multiple Kibana instances to efficiently manage reporting jobs. Reporting is now accessible to all Elastic Cloud users, requiring only an upgrade to version 2.4.0, with additional resources like a downloadable page and a webinar for further learning.

Elasticsearch, initially released in 2010, has evolved from a flexible and versatile tool into a robust search and analytics engine, now focusing on stability, security, and performance over flexibility. By 2016, it had been downloaded by millions and was a core technology for many users and companies. The evolution to version 5.0 saw a shift towards a more controlled and secure environment, with Elasticsearch recommended to run as a standalone server communicating via a client, rather than being embedded, to enhance security and reliability. This version introduced stricter configuration guidelines, runtime safeguards, and a formalized plugin API to reduce complexity and improve stability, while the introduction of a Java HTTP client aimed to streamline communication and eventually replace the Java transport client. These changes, although limiting some of the previous flexibility, aim to create a more dependable product, with a focus on clear error messaging, early problem detection, and protection against unsafe operations.

Elastic's Infrastructure team has implemented a serverless approach to running Elasticsearch Curator on AWS Lambda to reduce management overhead and increase efficiency. Instead of using servers, they employ serverless computing to handle tasks related to storing and analyzing system metrics, logs, and performance data using Elastic Cloud and Elasticsearch clusters. The process is facilitated by Lambkin, an open-source tool that simplifies creating, publishing, and scheduling Lambda functions. The setup involves configuring AWS credentials, creating a skeleton Lambda function, and using Python to develop the function, which can be managed and scheduled to run regularly using a YAML configuration file. This method allows the team to maintain a reliable, scalable system without expanding their server infrastructure, and the entire setup can be easily modified or checked into version control for future updates.

Issac Kelly, an engineer and designer, describes his experience building an Arduino-based home weather station using the ESP8266 microcontroller, Elasticsearch, and Kibana. By utilizing off-the-shelf components from Adafruit and open-source libraries, Kelly created a DIY project to measure temperature and pressure, driven by his curiosity to verify his friend's claim that he spent too much time indoors. The project involved setting up a wireless sensor network that communicates data to Elasticsearch, where it is stored and visualized using Kibana. Kelly shares insights into the hardware and software setup, including the use of the Arduino IDE, necessary libraries, and configuring Elasticsearch for data indexing. The initiative not only provided Kelly with the opportunity to experiment with WiFi-enabled microcontrollers but also resulted in a humorous live demonstration with a popsicle and hair dryer, showcasing the potential of IoT devices for personal projects and further experiments.

Monitoring the search queries sent to an Elasticsearch cluster is crucial for understanding user interaction and maintaining cluster health. By using tools like Packetbeat and integrating them with the Elastic Stack, including a dedicated monitoring cluster and a Kibana instance, administrators can track search traffic over HTTP and gain insights into query performance and usage patterns. This setup involves configuring Logstash to filter specific search-related data and storing it in a separate monitoring cluster to ensure accessible insights during production issues. The process allows for identifying frequently used queries, slow queries, and search patterns, which can inform resource planning and prevent outages. Although Packetbeat cannot inspect encrypted payloads, it still provides valuable data that can be visualized in Kibana to enhance understanding of cluster performance and optimize Elasticsearch usage.

Elasticsearch's introduction of a Java REST client in version 5.0 aims to provide a more lightweight and loosely coupled alternative to the existing transport client, with fewer dependencies and a more stable API. Performance benchmarks were conducted to compare the Java REST client and the transport client, focusing on throughput and latency during bulk indexing and search operations. The benchmarks used a "noop" Elasticsearch plugin to emphasize client performance, revealing that the Java REST client performs slightly below the transport client under lab conditions, particularly in bulk indexing. However, real-life tests show similar performances between the two clients, with the REST client achieving a sustainable throughput of around 1,200 operations per second compared to 1,700 for the transport client. Despite these initial differences, the Java REST client demonstrates promising performance characteristics for practical applications, with ongoing improvements anticipated to enhance its capabilities further.

Working with USAA's Cyber Threat Operations Center, Elastic provided dedicated support engineers to guide the integration of the Elastic Stack into USAA's systems, transforming it into a vital component of their threat management strategy. Throughout the first year, Elastic assisted USAA in resolving initial deployment issues and gradually tackling more complex challenges, such as memory management and efficient data handling. The collaboration, led by USAA's primary architect Neelsen Cyrus, resulted in the successful analysis of billions of security and server events per day, and USAA has since become a proponent of the Elastic Stack within the broader community. The partnership highlights the value of active participation and skill-building in implementing robust software solutions, with USAA sharing their experiences and achievements at events like Elastic{ON} 16.

Logstash utilizes message queues like Redis to manage event surges and prevent slowdowns in systems such as Elasticsearch. Redis is favored for its speed and efficiency, having been used since Logstash version 1.0.4. The performance of the Logstash Redis input plugin can be optimized by adjusting the number of threads, which significantly affects throughput. Increasing the batch_count beyond the default of 125 can reduce performance due to changes in the Logstash pipeline architecture. The Redis input plugin now uses batch mode by default, with Lua scripts facilitating batch retrievals, but larger batch sizes can negatively impact performance due to transaction waits. Tuning the Redis input plugin, pipeline batch size, and worker threads is crucial to ensure the input can keep pace with the filter/output stage. Using multiple Logstash instances with a single Redis server can improve throughput, though connection contention should be managed. Future enhancements include integrating a JRuby wrapper for the Jedis library to support Redis Cluster mode.

Starting with Elasticsearch 5.0, the ip field will support indexing IPv6 addresses, a feature that has been long requested due to the previous inefficiencies in indexing such addresses. Earlier methods either indexed them as sortable strings, which hindered range performance, or used multiple levels of precision, increasing index size and slowing indexing speed. The introduction of Lucene 6's multi-dimensional points allows efficient indexing of numerics up to 128 bits by dynamically computing ranges based on data, reducing index bloat and improving indexing speed. This advancement enables IPv6 support on all new indexes post-upgrade, with IPv4 addresses being internally represented as IPv4-mapped IPv6 addresses, thus maintaining efficient disk usage through improved compression.

StockTwits, a social network for the finance community, successfully revamped its message-sharing system by replacing its initial MySQL and Redis-based architecture with Elasticsearch, significantly improving request speed and user satisfaction while maintaining scalability for future traffic growth. The original system struggled with scalability issues due to complex caching and database queries, prompting a shift to Elasticsearch, which allowed for more efficient querying and indexing of messages and user relationships. By implementing a solution involving two indexes—"messages" and "friendships"—StockTwits was able to streamline its architecture, reduce code complexity, and introduce new features such as fast filter queries and full-text search capabilities. The team overcame challenges related to index growth by partitioning the message index into monthly chunks and using aliases for querying, which resulted in sub-millisecond query performance. The changes not only simplified the system's design but also positioned StockTwits to easily scale by adding nodes and enhancing its application features, supported by Elasticsearch's efficient caching and query handling.

The release of Elastic Stack 5.0.0-alpha5 introduces several enhancements across its suite of tools, including Elasticsearch, Kibana, Logstash, Beats, and ES-Hadoop, though it remains in the alpha stage and is not recommended for production environments. Key improvements in Elasticsearch involve better index creation, aggregation enhancements, and optimized handling of numeric data, while Kibana sees updates to tile map visualizations and the timepicker's autorefresh feature. Logstash's performance has been boosted with faster Beats input, and Beats itself supports flexible filtering based on conditions. ES-Hadoop now supports Spark 2.0, emphasizing the ongoing integration of new technologies. Despite being in the alpha phase, users are encouraged to test the updates and provide feedback to aid in the successful development of version 5.0, with the X-Pack continually updated alongside each alpha release.

On August 8, 2016, Elasticsearch announced the release of its 5.0.0-alpha5 version, which is based on Lucene 6.1.0 and serves as a testing ground for upcoming features in the 5.0.0 General Availability (GA) version. This release, intended solely for testing, introduces various improvements and fixes, though it is not suitable for production deployment. Notable additions include enhancements to the Java REST client, improved index creation processes, and aggregation improvements. However, a significant bug was identified in the Netty4 implementation affecting REST requests over 1024 bytes, with several workarounds suggested. Additionally, the release includes tools like the Elasticsearch Migration Helper to assist users transitioning from version 2.3.x to 5.0. Noteworthy security and performance enhancements, such as simplified certificate generation and improved data storage efficiencies, are also featured. Users are encouraged to test this alpha version and provide feedback through designated channels.

In his blog post, Chris Earle outlines strategies for verifying data integrity when using Elasticsearch alongside other databases, such as PostgreSQL, emphasizing the challenges posed by the lack of transaction support across multiple systems. He discusses various methods for verifying data, ranging from simple existence checks to verifying entire documents, and highlights the importance of structuring data effectively. Earle delves into techniques like using Elasticsearch's _mget API for batch processing, leveraging histograms to identify missing data, and utilizing the _scroll API for comprehensive document verification. He also touches on the potential of sidestepping verification by employing versioning and trusting data ingestion processes, alongside enhancements like X-Pack Security for user access control. Through these methods, the blog advocates for creative approaches to data verification challenges by rethinking queries and utilizing Elasticsearch's features, encouraging readers to engage with the community for further discussion and support.

Elasticsearch 2.3.5, based on Lucene 5.5.0, has been released, focusing exclusively on bug fixes for Marvel, Watcher, and Shield without altering the core Elasticsearch. The update is immediately available for deployment on Elastic Cloud, the Elasticsearch-as-a-service platform. Notably, the update ensures that Shield functions correctly with Tribe Nodes. Users are encouraged to download the new version, provide feedback on Twitter or the forum, and report any issues on the GitHub issues page. Detailed changes are documented in the release notes for Elasticsearch and related components.

Kibana has released versions 4.5.4 and 4.1.11, addressing two significant security vulnerabilities, enhancing tile map zoom capabilities, and fixing several bugs to improve user experience. The security updates include resolving an XSS vulnerability in markdown visualizations and ensuring Kibana logging does not store cookies or authorization headers, thereby preventing potential session hijacking. Other improvements include increasing the maximum zoom for tile maps, resolving issues with the search box autocomplete in Google Chrome, and fixing errors related to quick clicks, disappearing labels in split charts, and permission errors during package manager upgrades. Users are encouraged to upgrade to these versions to benefit from enhanced security and functionality.

The Elastic Stack, particularly Elasticsearch and Kibana, is effectively utilized for analyzing and visualizing earthquake data sourced from the Northern California Earthquake Data Center. The dataset includes both natural and man-made quakes, such as nuclear blasts and quarry explosions. Using a simple Logstash configuration, the data is imported into Elasticsearch, with custom mappings and templates to enhance usability. Kibana dashboards offer rich visualizations, including heat maps, histograms, and Timelion plots, which allow users to explore data dynamically. The dashboards are designed with various features like markdown introductions and reset links to guide users through the data, while geo_polygon queries and WMS-compliant map imports provide advanced geographic insights. The blog post highlights the versatility of Kibana visualizations and their ability to integrate data from external sources like the World Bank API, offering a comprehensive tool for both statistical analysis and historical event recording. The Elastic Stack's capabilities are exemplified through practical use cases, such as detecting earthquakes via real-time Twitter feeds, showcasing its potential for applications beyond traditional data analysis.