May 2015 Summaries

Jettro Coenradie's article explores the use of Elasticsearch's percolator feature in combination with geo points to build a basic classification system for geographic data, exemplified through a sample application involving Dutch zip codes and provinces. Elasticsearch's geo support is highlighted, including its ability to store and manipulate geolocations with features like geo shapes and geo polygon filters, which are essential for identifying and categorizing locations within specified areas. The sample application, built with Spring Boot and AngularJS, demonstrates the data lifecycle stages, from obtaining and transforming data to storing and utilizing it for geographic queries. The percolator feature is particularly emphasized for its role in inverse querying, where queries are stored and documents are matched to them, allowing the application to determine the province of a city based on its coordinates. The article concludes by suggesting potential enhancements, such as tracking user interactions with the application to improve data accuracy and user experience.

This edition of "This Week in Elastic" covers a wide array of updates and events related to the ELK Stack, including the introduction of Packetbeat to the Elastic family. Highlights include developments in Elasticsearch, Logstash, and Kibana, as well as community stories and practical advice from data scientists using these tools. The blog post discusses a long-standing bug fix in Apache Lucene and provides updates on upcoming versions of Logstash and Kibana. It also highlights global meetups and conferences where Elastic's team is present, offering insights into the ELK Stack's application in various industries, such as utility companies and data analysis firms. Additionally, the text mentions various training sessions and workshops on Elasticsearch scheduled across multiple global locations, inviting readers to participate and enhance their skills.

Packetbeat, a lightweight network packet analyzer that parses various protocols and integrates with Elasticsearch and Kibana for data visualization, has gained significant traction as an open-source project. Created by Tudor and Monica, Packetbeat exemplifies innovation in open-source platforms, leading to their recruitment by Elastic to spearhead the newly formed Beats team. This team is dedicated to developing lightweight data shippers that complement Elastic's technology stack, such as Logstash and Logstash-forwarder, by leveraging libbeat, a core library for shared functionality. The first of these data shippers, Packetbeat, is designed to capture network data, and its code base has been relicensed to Apache 2.0 to encourage broader adoption. Elastic envisions expanding the Beats ecosystem to include various data shippers, such as Filebeat, which will succeed Logstash-forwarder by focusing on log file tailing and data shipping, as well as potential future Beats for event logs and metrics.

The article explores the file structure and data management within Elasticsearch's data directory, focusing on node, index, and shard levels. It details the various paths used by Elasticsearch, such as path.home, path.conf, and path.data, and explains the roles of both Elasticsearch and Lucene in handling data storage. The text examines node data files, like the global state file and node.lock, and index data, illustrating how index state files store metadata. It further delves into shard data management, including the transaction log's role in maintaining performance by delaying full Lucene commits, and the structure of Lucene index files. The article also discusses the use of the Lucene CheckIndex tool for resolving shard issues and outlines the snapshot storage process, which includes renaming files to avoid conflicts when indexes are recreated and resnapshotted. Additionally, an editor's note updates readers on a change in Elasticsearch version 6.0 regarding content type headers in curl commands.

As the Leap Second approaches on June 30, 2015, adding an extra second to Coordinated Universal Time (UTC) to keep it aligned with the mean solar time, IT infrastructures face potential challenges, particularly within Linux-based systems. The 2012 Leap Second caused significant disruptions, including high CPU usage on production servers, affecting major companies. To mitigate these issues, Linux vendors like RHEL/CentOS, Ubuntu, and SLES/OpenSUSE have released guidelines for preparing systems, and Elastic advises following these recommended procedures instead of untested solutions. Additionally, cloud providers such as Amazon and Google Cloud Platform have offered specific guidance for managing the Leap Second within their services.

Watcher, a commercial alerting and notification product, has been released as a public beta by Elasticsearch to enhance its capabilities by allowing users to automate monitoring and response actions based on real-time data changes. Unlike Elasticsearch, which requires continuous manual analysis, Watcher enables a "reactively proactive" approach by automatically monitoring data, notifying users of important events, and executing predefined actions, thereby freeing users to focus on other tasks. It introduces a set of APIs to manage "watches," which consist of triggers, inputs, conditions, and actions designed to monitor specific events, evaluate conditions, and execute actions like logging or sending notifications. This extensible system can handle a wide range of use cases, from simple to complex, and is integrated seamlessly with Elasticsearch to provide immediate value. Users are encouraged to register for the beta program, explore the online documentation, and provide feedback through a dedicated forum.

This week in Elastic highlights several developments and updates across the ELK Stack ecosystem, including the public beta release of Elastic Watcher for Elasticsearch alerting. The blog post features various contributions and tutorials from community members, such as analyzing weblogs using Elasticsearch in the cloud and insights into IoT data using Elasticsearch, Apache Spark, and es-hadoop. It also covers numerous meetups and conferences worldwide where Elastic team members and enthusiasts are presenting on topics related to Elasticsearch, Logstash, and Kibana. Additionally, there are mentions of upcoming training sessions on Elasticsearch in locations like Bangalore, Paris, Miami, and San Francisco. The post invites community engagement by encouraging readers to share their presentations and meetups and highlights Elastic's ongoing recruitment efforts.

In May 2015, Aaron Mildenstein introduced the heartbeat input plugin for Logstash 1.5, designed to monitor the flow of events through Logstash's single pipeline by sending periodic heartbeat messages. The plugin helps identify when the pipeline's flow slows or stops, which can occur if any output becomes backed up, halting the entire pipeline. Users can configure the heartbeat messages to include a string value, an epoch timestamp, or a sequence number, and these messages can be used to monitor system performance, track event timing, or detect pipeline delays. The heartbeat plugin provides a straightforward method for ensuring Logstash's operational status and offers basic monitoring capabilities, with future plans to expand its functionality to include an API for comprehensive pipeline visibility and performance metrics.

Logstash 1.5.0 has been officially released, marking one of the most significant updates with over 500 commits and 10 months of development. This version introduces vital enhancements, including improved plugin management, performance boosts, and integration with Apache Kafka. Notable advancements involve the separation and easier management of over 165 plugins, alongside the introduction of new plugins like Heartbeat input and CouchDB changes input. Performance enhancements are evident in the Grok Filter, which now processes data faster, and JSON serialization/deserialization using the JrJackson library, resulting in significant throughput improvements. Security measures have been bolstered with added authentication and encryption features, improving compatibility with Elastic's Shield security product. The integration with Apache Kafka facilitates more efficient data processing and analysis, while Windows users benefit from improved setup and plugin infrastructure. Documentation has been relocated to the Elastic website for easier access, and the development team is already planning the roadmap for Logstash 2.0, inviting community feedback and participation through various channels.

This week's update from Elastic highlights developments in the ELK Stack, including progress on Elasticsearch, Logstash, and Kibana, such as Elasticsearch's alerting feature called Watcher. The blog also shares insights into new releases like Elasticsearch .NET and NEST 1.5 and discusses best practices for using Elasticsearch in virtualized environments. The update promotes numerous global meetups and conferences where Elastic team members will present on topics related to the ELK Stack, and it announces upcoming Elasticsearch training sessions in various locations worldwide. Additionally, the post encourages community engagement through meetups and forums while offering support for users hosting events and mentions ongoing hiring opportunities within the company.

Elastic has transitioned from using multiple mailing lists to a centralized online discussion forum to better support its community as it scales development on open-source projects like Elasticsearch and Logstash. The company found the previous system of separate mailing lists, including those for non-English speakers, to be confusing for newcomers and noisy due to diverse user questions. The new forum, accessible via discuss.elastic.co, aims to streamline communication and provide a more user-friendly experience, allowing users to create profiles using various authentication methods such as GitHub, Facebook, and Google Apps, and manage email preferences for easier interaction. Elastic encourages users to join the conversation on the forum and provide feedback on the new tool to enhance their experience.

The article explores the use of the ELK stack—Elasticsearch, Logstash, and Kibana—for analyzing web logs in the cloud, specifically using Microsoft's Internet Information Services (IIS) logs. It highlights the process of configuring Logstash to parse and process log data from an IIS-based web search application, creating custom grok patterns for non-standard log formats, and utilizing Kibana for visual representation of data. The author details creating visualizations to analyze search terms and request patterns, emphasizing the importance of correctly indexing fields for better data insights. The piece concludes by encouraging further exploration of the ELK stack, offering resources for deeper learning and mastery of data aggregation in Elasticsearch.

In this blog post, Christoffer Vig outlines the process of using Logstash and Kibana with the Elasticsearch cloud service, formerly known as Found, to analyze weblogs. The article explains the initial setup of Logstash and Kibana on a Windows laptop, emphasizing the importance of using SSL connections for securing data transfer between the local machine and the cloud-hosted Elasticsearch cluster. It provides step-by-step instructions on creating an Elasticsearch cluster, configuring Logstash to connect with the cluster, and setting up Kibana to visualize data. The post highlights security measures, such as access control lists and user authentication, to protect the Elasticsearch cluster from unauthorized access. The article concludes with a preview of the next installment, which will cover the ingestion of IIS log files into Elasticsearch and the visualization of web traffic data using Kibana.

The article explores the complexities and best practices of using Elasticsearch in virtualized environments, highlighting the historical development of virtualization and its significance in modern computing. It details various platforms such as Amazon EC2, Microsoft Azure, and VMware vSphere, which can host Elasticsearch, and discusses their unique configurations and capabilities. The article emphasizes the importance of effectively managing resources like CPU, memory, disk, and network to ensure optimal performance, while addressing potential issues like CPU limits, memory allocation, disk contention, and network configuration. It also highlights common challenges, such as bugs in operating systems and resource limitations, and suggests solutions like forced awareness and shard allocation awareness to mitigate these risks. Ultimately, the article underscores the need to balance technical and financial considerations to choose the most effective and efficient virtualization strategy for Elasticsearch deployments.

On May 7, 2015, Suyog Rao announced the release of the fourth release candidate (RC4) for Logstash 1.5.0, addressing significant bugs found in RC3. The update rectifies a regression in filter chaining affecting periodic flush filters like multiline and metrics, and resolves an issue with the Lumberjack input plugin, which previously failed to properly timeout connections in back pressure scenarios, leading to increased memory usage and potential crashes. To mitigate this, an upper limit of 1000 Lumberjack connections has been implemented, although further improvements are planned. Users are encouraged to download RC4, test it, and provide feedback through GitHub issues, pull requests, or social media interactions.

The weekly roundup post "This Week in Elastic" provides updates on the ELK Stack, including new releases and improvements in Elasticsearch, Logstash, and Kibana. It highlights the release of Shield versions 1.2.1 and 1.1.1, which address a bug fix for Disk-based Shard Allocation, and the compatibility updates for various plugins with Elasticsearch 1.4.5. The post also shares insights into the Mars Curiosity Rover's use of Elasticsearch for cloud analytics and mentions tutorials on mapping user locations with GeoIP. Furthermore, it details Elasticsearch-related events and meetups happening globally, from training sessions and workshops to presentations and conferences, encouraging participation and collaboration within the community. The post concludes with a call for community members to share their events and highlights available job opportunities at Elastic.