October 2015 Summaries

In the blog post by Jay Greenberg, the integration of F5's High Speed Logging (HSL) with the Elastic Stack is explored to effectively analyze web traffic patterns and application behavior in near real-time. The F5 Local Traffic Manager acts as a load balancer, distributing logging traffic across Logstash servers and providing a unique vantage point for traffic analysis. This setup allows for detailed insights, such as HTTP error rates, traffic sources, and application response times, through visualizations like vertical bar charts, tile maps, and line charts. The blog explains how to configure HSL to work with Logstash, using a GROK pattern to parse log entries into Elasticsearch. Additionally, the importance of setting up health checks for Logstash instances is highlighted, alongside a discussion on the pros and cons of using UDP versus TCP for HSL. Alternatives for measuring HTTP response times without an F5 are also mentioned, emphasizing the flexibility of the Logstash and Beats tools in capturing similar data.

Improving search performance in Elasticsearch is often more effective through data modeling rather than merely tuning settings, as altering document modeling can dramatically enhance performance. For instance, using an n-gram tokenizer instead of wildcard queries can speed up searches by indexing every substring, transforming slow wildcard queries into faster term queries. When users search real-estate listings, aggregations can be optimized by pre-computing data, directly indexing price and surface ranges to replace range and histogram aggregations with terms aggregations. Further optimization is achieved by merging multiple terms aggregations into a single one, reducing the number of aggregations from six to two while maintaining the same information. Although this approach might limit flexibility, as changing ranges would require reindexing, it can complement the traditional method by retaining original data fields and allowing default set ranges for most requests, while using regular range aggregations for more detailed queries.

Logstash emphasizes testing as a crucial part of its engineering process, utilizing a variety of approaches to ensure software quality, including unit, integration, acceptance, functional, and regression tests. Written in JRuby, Logstash relies on Ruby- and Java-specific tools, with RSpec serving as the test runner and Jenkins as the Continuous Integration platform. The testing framework is well-organized, with dedicated directories for core and plugin tests, and employs advanced testing techniques like randomized testing through the ruby-flores library. Developers can run tests using specific rake tasks for the core codebase or the `bundle exec rspec` command for plugins. The blog post provides guidance on writing tests using RSpec, detailing the structure and conventions for organizing test files, and underscores the significance of automated testing across Elastic products.

Sense, initially conceived during a hackathon brainstorming session, is a tool designed to facilitate interactions with Elasticsearch's REST API, which traditionally relied on cumbersome terminal commands. The project started as a Chrome Extension to offer a more user-friendly and automatically updating interface, gradually evolving with features like a History Panel and multiple request handling, thanks in part to community feedback and collaboration. As Sense gained popularity, it was integrated into Marvel, Elasticsearch's management and monitoring solution, where it benefited from professional development support and underwent significant enhancements, including the development of a unified suggestion engine. With the release of Sense 2.0 as an open-source Kibana app, it now aligns with Kibana's vision as a flexible application platform, offering improved features such as support for Elasticsearch 2.0, multiple request execution, and the ability to copy and paste commands in a streamlined manner.

Elasticsearch 2.0.0 GA, based on Lucene 5.2.1, introduces numerous enhancements including the integration of Pipeline Aggregations, which simplifies client-side computations and facilitates powerful analytic queries like predictive analytics and anomaly detection. The update merges query and filter functions to optimize performance and introduces configurable store compression for better data management. Security is enhanced by running under the Java Security Manager and ensuring documents are fsynced before indexing. Performance improvements include lower heap usage, faster change propagation, and auto-throttling of merges. Core plugins, such as Shield and Watcher, now offer new features like field-level access control and Slack notifications, while Marvel, a monitoring tool, is free for production use. The Sense editor is made open-source, and an Elasticsearch Migration Plugin assists users transitioning from version 1.x to 2.0.

Beats version 1.0.0-beta4 introduces Filebeat, a new open-source log forwarder designed to efficiently transfer logs to Logstash or Elasticsearch, building on the Logstash Forwarder code base but with notable improvements. Filebeat aims to provide lightweight log forwarding with no dependencies, ensuring that no log lines are lost during network disruptions by using an "at-least-once" delivery approach, though this can result in duplicate messages. The new protocol includes enhancements like encryption, message batching, and a slow start mechanism to minimize duplicates. Filebeat supports Windows, addressing previous limitations, and uses YAML for configuration to facilitate migration from Logstash Forwarder, with an extensive guide available to assist users. While Filebeat was the focus, improvements were also made to Packetbeat and Topbeat, enabling data transfer to Logstash without Redis. Future updates plan to include features like multi-line support and Windows event log integration, aiming for a general availability release soon.

Chris Hart's guide demonstrates how to create a REST API powered by Elasticsearch for any dataset, using the Gender Inequality Index (GII) as an example. The guide illustrates setting up the project using Ramses, an open-source tool similar to "backend as a service," which allows users to run their own server and access the internals. It involves downloading the GII data, setting up a schema, configuring endpoints, and using Elasticsearch query DSL for data querying. The API facilitates various data manipulations, such as sorting, filtering, and performing aggregations, to extract insights about gender inequality across countries. The guide also shows how to handle pagination, conduct full-text searches, and perform aggregations to compute statistics like the average global gender inequality. Additionally, the text provides technical instructions for setting up the environment, configuring the server, and using command-line tools to interact with the API, alongside offering support through a Gitter channel for further assistance.

Elasticsearch, prior to version 2.0, was packaged as a JAR with some dependencies shaded to prevent version conflicts for Java users embedding it in their applications, but this approach created complexity and debugging challenges. From version 2.0 onwards, Elasticsearch opted to eliminate shading, requiring developers to manage dependencies like Joda and Guava manually to avoid conflicts. The text provides a detailed guide on how to handle these conflicts through dependency relocation and shading within a Maven project, offering an example of how to use the Elasticsearch transport client with different versions of Joda. The document also hints at future plans to reduce dependency conflicts by replacing some third-party libraries with native Java alternatives and potentially developing a lightweight Java client for Elasticsearch. Despite the challenges of implementing these changes, the aim is to simplify dependency management and improve the overall development experience.

Elastic is actively engaging with the global tech community through a series of events and meetups across multiple continents, showcasing their expertise in Elasticsearch and related technologies. Highlights include Alexander Reelen's session on Elasticsearch at CFCamp in Munich, and various meetups across Europe, North America, Africa, and Asia, where topics like search capabilities, securing Elasticsearch clusters, and the new features in Elasticsearch 2.0 are being discussed. Notable speakers such as Morgan Goeller and Paul Sorensen are participating, with the latter discussing the application of Elasticsearch in optimizing hotel search algorithms. Elastic encourages community involvement by offering support and promotional materials for those interested in hosting or speaking at meetups about their technologies.

Elasticsearch 1.7.3, based on Lucene 4.10.4, is a stable release that addresses several bugs affecting previous versions and is recommended for users experiencing these issues. Key bug fixes include resolving problems with synced flushes that reduced index buffer sizes, excessive throttling in Snapshot/Restore operations, heavy memory usage due to old cluster states in tribe nodes connected to slow clusters, and indeterminate application of analyzer settings that complicated transitions to version 2.0.0. Additionally, a networking layer bug related to SSL handling has been fixed by upgrading to Netty 3.10.5. Users are encouraged to download the release and provide feedback via Twitter or the forum, with problems reportable on the GitHub issues page.

Christina Thaute and Russ Savage introduce a new Google Sheets add-on that simplifies visualizing spreadsheet data using Elasticsearch and Kibana. The article details the process of setting up a hosted Elasticsearch cluster with Elastic Cloud, formerly known as Found, and utilizing the add-on to push spreadsheet data to the cluster for visualization in Kibana. Users are guided through installing the add-on, configuring connection details, and managing data indexing to ensure seamless integration with Elasticsearch. The piece highlights practical tips, such as selecting unique identifiers to prevent duplicate entries and offers resources like demonstration videos and GitHub code for further exploration. The emphasis is on the intuitive nature of the process, making data visualization accessible even for those new to Elasticsearch and Kibana.

The text provides a schedule of upcoming Elastic-related events and meetups across Europe, North America, and Asia for the week of October 12, 2015. Notable events include David Pilato speaking about advanced search at the Bordeaux Developer eXperience on October 16 and the EuRuKo 2015 conference in Salzburg from October 17-18. Meetups are happening in various cities, including London, Marseille, Tel-Aviv, Stamford, Chicago, Duluth, and Beijing, featuring talks from Elastic team members like Tal Levy from the Logstash team. The Elastic Team encourages individuals interested in hosting or speaking at meetups to contact them for support and promotional materials.

Logstash is a robust data processing tool that can be resource-intensive, and optimizing its configuration is essential for maximizing hardware efficiency. This blog post outlines a six-step methodology to fine-tune Logstash setups, emphasizing the necessity of tailoring configurations to specific data and hardware environments. The optimization process involves using sample data for repeatable testing, employing the Logstash metrics plugin for performance measurement, and adjusting filter efficiency by balancing computational resources with logical structures like conditional statements and type casting. Input optimization focuses on removing filter bottlenecks and testing throughput limits, while output optimization involves preparing test files and ensuring that output targets do not become bottlenecks. The article stresses the importance of incremental testing and adjusting configurations based on specific system needs, highlighting that there are no one-size-fits-all solutions and that thorough testing can lead to significant performance gains and reduced hardware costs.

Elasticsearch 2.0.0-rc1, based on Lucene 5.2.1, has been released as the final release candidate before the general availability (GA) of Elasticsearch 2.0, and is intended solely for testing purposes, emphasizing that it should not be deployed in production. This release candidate, which is not compatible with previous beta versions, focuses primarily on minor bug fixes and optimizations, including the points_only optimization for geo-shape fields and a new caching interface in the Shield plugin. The announcement encourages extensive testing to identify and resolve any issues before the GA release, providing links to download the release candidate and access change logs and documentation. Additionally, the Elasticsearch Migration Plugin is available to help users prepare their indices and mappings for compatibility with Elasticsearch 2.0. Feedback is encouraged via Twitter, forums, or GitHub to expedite the GA release process.

Elastic is actively engaging with the global tech community through a series of events and meetups across Europe, North America, and Asia. Highlights include their presence at significant conferences such as LinuxCon/Cloud Open in Dublin, where they will showcase their offerings at booth #22, and the AWS re:Invent in Las Vegas, where attendees can visit their booth in Hall C of The Venetian. They are also participating in PuppetConf 2015 in Portland and the Future of Web Apps Conference in London, where Thijs Feryn will present on ElasticSearch. Additionally, Elastic is nominated for the Best European Start-up award at Emerce Day in Amsterdam, where Chretien Minke will deliver a presentation. The company is also organizing local meetups, including events in Barcelona and Ho Chi Minh City, and is encouraging community involvement in hosting or speaking at meetups about their products.