November 2015 Summaries

Elastic's blog post details the investigation and resolution of a significant memory leak affecting Logstash deployments on Windows, which was reported by numerous users. Despite initial difficulty in identifying the issue due to standard JVM heap analysis tools showing no anomalies, the team employed Microsoft Sysinternals' VMMap, which revealed excessive "Unusable" memory related to small, un-freed allocations. The root cause was traced to the Logstash file input plugin, particularly the FileWatch library's use of Ruby's File#stat method, which caused continuous off-heap allocations in JRuby. By narrowing down JRuby versions, the leak was pinpointed to a regression introduced in JRuby 1.7.20. The JRuby core team, alerted via IRC, quickly identified the issue within the jnr-posix library and released a fix in JRuby 9.0.4.0 and 1.7.23, which was incorporated into upcoming Logstash releases.

Running an Elasticsearch cluster across multiple data centers is generally discouraged due to challenges such as high latency, limited or unreliable connectivity, and data availability issues, which can degrade cluster performance and complicate operations. The architecture of Elasticsearch assumes nodes are located on a local network, making it less suited for geographically distributed data centers where network disruptions are more common. Alternative solutions include using replicated queuing systems for real-time availability across regions, employing snapshots for disaster recovery without hot replication, keeping datasets local to each data center while using cross-cluster search for unified querying, and leveraging cross-cluster replication for data redundancy and geo-proximity reads. These strategies aim to address the limitations of distributing an Elasticsearch cluster across multiple data centers while maintaining data integrity and performance.

Elastic's "Where in the World is Elastic?" update highlights a series of events and meetups happening globally during the first week of December 2015. In North America, Elastic{ON} Tour will visit Los Angeles and Seattle, while meetups in San Francisco, Dallas, Seattle, and Madison will focus on technologies like NGINX, Apache Hive, and Kibana. In Europe, Elastic will feature at events such as DevTernity in Riga, GOTO Berlin, and a hands-on workshop in Brussels. In Asia, the team will participate in Strata+Hadoop World in Singapore and host the Spark Singapore Meetup. Additionally, a meetup in Johannesburg, Africa, will showcase integrating the Elastic stack into Java ecosystems. The Elastic team encourages the community to host or speak at future meetups, offering support and swag for those involved.

Elasticsearch offers a versatile array of features to manage cluster topologies, enabling the formation of heterogeneous clusters with varied node capacities. While it might be tempting to consolidate all workloads into a single, powerful cluster, the text argues for the benefits of managing multiple separate clusters. This approach simplifies performance management and limits the impact of failures by allowing different workloads, such as autocompletion, full-text search, and analytics, to be handled independently. It also facilitates safer upgrades through cloning and enables latency-based routing by distributing clusters globally. Additionally, separate clusters can accommodate bulk indexing tasks and experimental testing without risking production stability. Although managing multiple clusters involves additional complexity, tools like Elastic Cloud streamline the process by offering features such as easy cluster creation, snapshot/restore capabilities, and hourly billing, making it feasible to manage both short-lived and continuous production clusters efficiently.

Elastic, a company specializing in search technology, was invited to deliver a guest lecture at VU University in Amsterdam where Boaz Leskes, a core software engineer, introduced around 100 computer science and engineering students to the fundamentals of Elasticsearch. Despite most students being unfamiliar with Elasticsearch and Lucene initially, the lecture aimed to demystify the lifecycle of a document within Elasticsearch, detailing how documents are indexed, replicated, and retrieved across a cluster. Boaz engaged the students with humor and insightful explanations, sparking their curiosity and encouraging them to ask numerous questions, leading to an enthusiastic interaction. The session concluded with special recognition for the students who posed the most thoughtful questions, rewarding them with a book on Elasticsearch. Overall, the lecture was deemed a success, leaving both the presenters and students with a positive experience of revisiting academia.

Elasticsearch 2.1.0, released alongside a bug fix update for Elasticsearch 2.0.1, introduces several key enhancements based on Lucene 5.3.1, including the geo-centroid metric aggregation that offers a more precise overlay of geolocation data on maps, improving upon the previous row-and-column display. Three new pipeline aggregations are introduced to enhance the statistical processing of data, offering calculations like min, max, average, and specified percentiles. Further improvements to node restart times and shard rebalancing processes, such as synced flushing and delayed shard allocation, address previous flaws in shard management. Incremental changes include a soft limit on search results, default update behavior adjustments, and replacements for count and scan search types, aiming to optimize performance and user experience. Users are encouraged to download and test the new release, providing feedback via social media and GitHub.

On November 24, 2015, Elastic announced the release of the first production-ready version of the Beats platform, which includes Topbeat, Filebeat, and Packetbeat, designed to capture and ship operational data to Logstash and Elasticsearch. This platform, built on the libbeat Go library, aims to streamline the creation of lightweight shippers for data collection by handling common tasks such as secure event sending and load balancing. Topbeat centralizes system metrics across different operating systems, Filebeat replaces Logstash Forwarder for efficient log shipping, and Packetbeat provides real-time network traffic monitoring akin to Wireshark, facilitating visibility into application communications. The platform encourages community contributions, with several user-created Beats and added protocol support, and offers developer guides to foster new developments, highlighting its open-source nature and collaborative potential.

Zachary Tong explores the implementation of a statistical anomaly detection system using Elasticsearch, inspired by eBay's Atlas algorithm, which addresses the challenge of identifying subtle disruptions in large datasets. The article illustrates the difficulty of detecting anomalies in aggregated data, as the variability tends to smooth out significant changes, while monitoring numerous individual time series is impractical at scale. Tong explains how Atlas effectively aggregates correlated changes in data variance to alert on real disruptions by being robust to noise yet sensitive to genuine shifts. He details the process of simulating a data production cluster using Rust to create a realistic dataset with controlled disruptions, and outlines the use of Elasticsearch's aggregations, including moving averages and surprise calculations, to detect anomalies. Tong describes how this method calculates the largest deviations from the average, focusing on the 90th percentile of these deviations to identify significant anomalies, setting the stage for further development in subsequent articles.

In November 2015, Elastic organized a series of events and meetups across various regions, including Europe, Asia, and Africa, focusing on the newly released Elastic stack and its components like Elasticsearch, Logstash, and Kibana. Notable events included workshops and demonstrations in cities such as London, Paris, Warsaw, Copenhagen, and Tokyo, where attendees could learn about Elasticsearch 2.0's new features, explore use cases, and engage with open-source search experts. These meetups offered opportunities for networking, sharing knowledge, and discussing the challenges of building search products, with sessions featuring presentations from industry professionals and Elastic team members. Elastic also encouraged community involvement by offering support to those interested in hosting future meetups or talks related to their technologies.

StreamSets Data Collector is an open-source software designed to facilitate continuous data ingestion pipelines for Elasticsearch, addressing challenges such as data drift that can lead to data loss and unreliable analytics. StreamSets offers a robust alternative to traditional ETL by adapting to changes in schema, semantics, and infrastructure while providing features like in-stream data cleansing and error handling to maintain pipeline reliability. With its user-friendly drag-and-drop interface and a range of APIs for customization, StreamSets allows users to connect diverse data sources to Elasticsearch, supporting real-time analytics by ensuring high-quality, curated data flow. This integration empowers users to perform accurate and uninterrupted real-time analysis, critical for applications ranging from social data analysis to financial transaction monitoring. Both StreamSets and Elasticsearch are built for scalable, real-time, in-memory performance, forming a complementary backbone for modern data infrastructures. Arvind Prabhakar, a co-founder of StreamSets and a veteran in data integration, has contributed significantly to the development of open-source projects like Apache Flume and Sqoop, bringing his extensive experience to the creation of StreamSets.

Beats 1.0.0-rc2, the last release candidate before the general availability of Beats 1.0.0, has been announced with minor bug fixes, clean-ups, and enhancements. Notable changes include the transition from exporting "shipper" to "beat.name" for clarity, and the addition of "beat.hostname," requiring updates to Beats dashboards. Filebeat no longer exports the "line" field due to performance issues, recommending the use of the "offset" field instead. Compatibility-breaking changes are noted in the release notes of Topbeat, Packetbeat, and Filebeat. The development team encourages beta testers to download, test, and provide feedback to expedite the official release.

Elastic is actively engaging with its community through a series of events and meetups around the world, including notable appearances at QCon San Francisco and Devoxx Morocco, with dedicated Elastic{ON} Tour stops in New York and Chicago. These events provide opportunities for attendees to connect with the Elastic team, learn about product roadmaps, and participate in discussions on Elasticsearch and related technologies. The upcoming meetups span across Europe, North America, and Asia, featuring presentations on the latest developments in Elasticsearch 2.0 and other Elastic products. Elastic invites community members to host meetups and offers support for talks on Beats, Elasticsearch, Logstash, or Kibana, emphasizing their commitment to fostering a collaborative and informative environment.

In the latest update from the Beats team, significant progress has been made towards the upcoming 1.0.0 GA release, with key changes based on community feedback and efforts to simplify configuration. Notable updates include the introduction of Winlogbeat as a proof-of-concept project for shipping Windows Event Logs, the replacement of the "shipper" field with "beat.name" for clarity, and fixes for a memory leak in Topbeat. Furthermore, configuration options have been streamlined by removing the "enabled" field for outputs, and Filebeat no longer exports the "line" field due to performance concerns. A script has been implemented to automatically update shared files across all Beats repositories, and system tests for Topbeat have been added, enhancing the reliability and functionality of the platform.

Elastic recently concluded its Elastic{ON} Tour, with the final event taking place in Tokyo on November 16, 2015. The tour featured stops in multiple cities, including Washington D.C., Amsterdam, London, Paris, Munich, New York, Chicago, Los Angeles, Seattle, Sydney, and Melbourne, showcasing various Elastic products and engaging with the community. Although no events are happening this week, Elastic is promoting its upcoming Elasticsearch User Conference, Elastic{ON}16, which will feature numerous sessions and two pre-conference workshops. The Elastic team encourages individuals interested in hosting meetups or presenting on topics like Beats, Elasticsearch, Logstash, or Kibana to reach out for support and merchandise.

Timelion is a versatile time series composer for Kibana, developed as part of Elastic's {Re}search initiative, which allows users to aggregate and visualize time series data from various sources using a simple one-line expression language. Created by Rashid Khan during a transatlantic flight, Timelion enables the execution of complex queries and transformations by chaining functions and combining data from multiple sources, including the World Bank's Data API and Elasticsearch. With 25 built-in functions ranging from basic arithmetic to advanced analytics like moving averages and derivatives, Timelion fosters innovation by inviting users to contribute new functions and share their experiences. Timelion is readily available as a plugin for Kibana and can be tried through Elastic Cloud, offering users an accessible platform to explore their data-driven inquiries and generate insightful visualizations.

Decitre, a Rhône-Alpes library network that expanded into e-commerce, has enhanced the relevance of its search engine by leveraging Elasticsearch and the expertise of Zenika. The initiative involved logging user searches to gather data on search patterns, indexing this information in Elasticsearch, and utilizing Kibana for analysis. This process allowed Decitre to identify search queries yielding no results and make improvements, such as adding missing books or synonyms, and refining search analyzers. The team tested hypotheses about search relevance and adjusted configurations in a pre-production environment to measure changes in search outcomes. They used tools like graphviz to visualize the impact of modifications, which illustrated improvements such as a 3% reduction in zero-result searches. Monitoring tools and dashboards were employed to ensure changes positively affected search performance and relevance in production, with a significant portion of user clicks occurring within the top 20 search results, indicating successful enhancements.

Elastic's blog post provides a roundup of upcoming Elastic-related events and meetups occurring across various continents during the week of November 9, 2015. Highlighted events include Elastic{ON}Tour in Munich, Midwest.io in North America, and Playnode.io in Asia, along with meetups from user groups focused on technologies like Hadoop, AWS, Elasticsearch, and Python in regions such as Europe, North America, Asia, and Africa. The post concludes with an invitation for readers to host meetups or give talks on Elastic products like Beats, Elasticsearch, Logstash, or Kibana, offering support and promotional materials to those interested.

The de_dot filter was introduced to address a compatibility issue between Logstash 2.0 and Elasticsearch 2.0, as the latter does not support field names containing dots. This filter replaces dots with underscores or any specified separator, allowing users to maintain compatibility without altering source field names. It also offers an option to translate dotted fields into nested fields by setting a configuration parameter, providing flexibility in data structuring. Users can specify which fields to process to optimize performance, especially when dealing with nested fields. The filter aims to ease the transition to the new Logstash and Elasticsearch versions by mitigating the challenges posed by dotted field names. Users are encouraged to engage with the community for support and to provide feedback on the filter's functionality.

Beats 1.0.0-rc1 has been released, featuring numerous fixes and improvements based on feedback from the 1.0.0-beta4 version. A significant change is the renaming of the "timestamp" field to "@timestamp" to resolve confusion caused by integration with Logstash, which affects data compatibility with new Kibana dashboards; however, existing dashboards remain functional with old data. Upgrading requires reloading templates and dashboards in Kibana, and updating the Logstash Input Beats Plugin to accommodate the change. Installation of Packetbeat, Filebeat, and Topbeat is now easier via apt and yum repositories, with packages available on packages.elastic.co. Further testing is encouraged to expedite the release of Beats 1.0.0 GA, with feedback welcomed on social media and GitHub.

Felix Barnsteiner discusses the suitability of Elasticsearch as a time series database for performance monitoring, specifically for the open-source tool stagemonitor, which needed a replacement for the aging Graphite TSDB. He highlights Elasticsearch's ease of installation, scalability, and strong visualization capabilities, especially when paired with Kibana, making it an attractive choice despite initial doubts about its aptitude for time series data. Barnsteiner references a CERN performance comparison that favored Elasticsearch over InfluxDB and OpenTSDB, further supporting its use. Essential practices such as defining mappings, disabling source for metric documents, and employing doc_values are emphasized to optimize storage and performance, while aggressive optimizations like using index "no" for metric values and force merging indices are discussed for long-term data management. Barnsteiner also outlines improvements in Elasticsearch 2.0, such as pipeline aggregations and default doc_values, which enhance its flexibility for time series data applications.

Elasticsearch 2.0 introduced a series of improvements aimed at enhancing indexing performance and resilience. One of the significant changes is the introduction of adaptive merge IO throttling, which automatically adjusts the IO rate based on the workload, eliminating the need for manual throttling adjustments. The update also modifies how data is distributed across multiple IO devices by allocating entire shards to specific paths, improving resilience to device failures. The removal of the Auto-ID optimization, previously used for indexing without user-defined IDs, is offset by improvements in ID lookup performance and the adoption of Flake IDs, enhancing reliability and performance. Additional enhancements include enabling doc values by default, improved data compression techniques, and shifts in API functionality to enhance safety and reduce heap usage during merges. These updates collectively aim to optimize Elasticsearch's indexing capabilities and overall system performance.

Elastic's blog post highlights a series of events and meetups happening around the world, showcasing the company's active engagement with its community. Elastic{ON}Tour is making stops in London and Paris on November 3 and 5, respectively, where attendees will have the opportunity to learn about the company's product roadmap and hear customer experiences. Concurrently, in North America, Elastic will be present at DevOps Days in Silicon Valley on November 6-7. Further community involvement includes various meetups, such as the Scala-Romandie meetup in Switzerland and Elastic gatherings in St. Louis, Singapore, and China, all occurring on November 5 and 7. Elastic emphasizes its commitment to supporting and connecting with its community by inviting those interested in hosting or speaking at meetups to reach out for support and promotional materials.