September 2015 Summaries

The blog post provides a comprehensive guide on how to generate and implement coverage reports for system tests in Golang projects, specifically focusing on the Packetbeat product by Elastic. It outlines the challenges faced when unit tests cannot fully cover code dependent on external environments and provides a solution using the Golang toolchain to generate a coverage binary that tracks executed lines of code. The guide details a step-by-step approach, including the creation of a main test file, management of command line flags, execution of binaries, and the collection and conversion of coverage data into a human-readable format. It emphasizes the significance of system test coverage in microservice environments, allowing for detailed insights into which parts of each service were executed during tests. The post concludes with future plans to enhance the testing framework used across Elastic's Beats projects and invites contributions from the community.

Elastic is actively engaging with the tech community through various events and meetups as September transitions to October. Highlights include their participation in DevRelCon 2015 in London, where Shaunak Kashyap will discuss developer relations metrics, and Strata + Hadoop World in New York, where Elastic will host a booth. Additional meetups are planned in Oslo and Johannesburg. The team is also encouraging community members to host meetups or give talks on Elastic products such as Beats, Elasticsearch, Logstash, or Kibana, offering support and promotional materials in return.

Jurgen Altziebler explores the fascinating tennis rivalry between Roger Federer and Novak Djokovic, utilizing Elasticsearch, Logstash, and Kibana to create dynamic dashboards that visualize their match data. As of September 2015, the two players were tied with 21 wins each, with Federer initially dominating until Djokovic's rise in 2011. Their rivalry intensified over the past four years, with Djokovic winning 12 out of their 19 encounters, including four out of five grand slam meetings. Altziebler emphasizes the importance of knowing your data and creating coherent stories in dashboards, suggesting strategies such as grouping modules logically and testing designs with others. Despite Federer being in his mid-thirties, he maintains an impressive win rate, having won 83% of his matches over the last four years, including 310 matches in total, and the dashboards also highlight his encounters with Rafael Nadal while using Kibana's dark theme for improved visual presentation.

Elastic is actively organizing and participating in various events and meetups around the world, with notable activities in Europe, Asia, North America, and Australia. The Elastic{ON} Tour is a significant highlight, with a recent event in Washington, DC, and an upcoming one in Amsterdam, where proceeds will be donated to CoderDojo. Meanwhile, Elastic team members are sharing their expertise at events such as DevOps Days Berlin, Librecon.io in Europe, and the Korea Linux Forum in Asia. Additionally, Elastic is engaging with community meetups in cities like Munich, Vienna, and Sydney, while also supporting discussions about their tools and technologies. Elastic encourages community involvement and offers support to those interested in hosting meetups or talks about their products.

Asawari Samant introduces a newly improved GitHub repository designed to help users, both new and experienced, get started with the ELK Stack, which includes Elasticsearch, Logstash, and Kibana. The repository offers easy-to-use examples aimed at simplifying the process of transforming raw data into insightful Kibana dashboards by providing sample data, Logstash configuration files, Elasticsearch mapping templates, and pre-built Kibana dashboards, along with detailed instructions. The initiative encourages community involvement, inviting users to share their own examples or ideas by opening GitHub issues, with a promise to collaborate and potentially include these contributions in the repository. Elastic expresses enthusiasm for the project's potential to foster learning and innovation within its user community.

In September 2015, Elastic organized a series of events and meetups across Europe, North America, and Australia to engage with users and showcase Elasticsearch applications. Notable events included David Pilato's sessions in Amsterdam and Geneva, focusing on advanced search for legacy applications and hands-on experience with Elasticsearch. In North America, the Elastic User Group Meetups featured discussions on scaling document repositories and utilizing the ELK stack in locations such as New York City, Salt Lake City, and Dallas. European meetups in Paris, Bordeaux, London, and Amsterdam offered workshops and talks on Elasticsearch and related technologies, while the Sydney meetup in Australia catered to the local developer community. These events aimed to foster community interaction and share insights on leveraging Elastic's capabilities, with opportunities for attendees to network and learn from expert speakers. Elastic also encouraged the hosting of meetups and talks, offering support and swag for such initiatives.

Elasticsearch 2.0.0-beta2, based on Lucene 5.2.1, has been released for testing, marking the final beta before the release candidate of version 2.0.0. This beta version is not compatible with its predecessor, beta1, and is not intended for production use as there is no guarantee of compatibility with the future general availability release. The update primarily includes bug fixes and refinements, while introducing significant features in commercial plugins like Shield and Watcher. Shield now offers enhanced document- and field-level security, user impersonation capabilities, and support for custom authentication realms, while Watcher introduces pausable watches and new chatroom actions for Slack and Hipchat. Users are encouraged to test this release and report issues to facilitate the general availability release. The Elasticsearch Migration Plugin is available to assist with identifying necessary upgrades and potential issues prior to testing the beta version.

Shield and Watcher 2.0.0-beta2 releases introduce several new features and improvements for testing purposes, with Document and Field Level Access Control now a first-class feature in Shield, allowing more precise data security management by defining document and field accessibility per role. Additionally, Shield supports User Impersonation, enabling users to execute requests on behalf of others when authentication is managed outside Elasticsearch, and allows custom implementation of authentication realms. Watcher enhances its functionality by allowing users to deactivate watches during system maintenance and adds new communication actions via HipChat and Slack for templated notifications. These updates also include bug fixes and infrastructure enhancements, with further details available in the release notes and forums.

Elasticsearch 2.0 introduced significant improvements in storage requirements, including the best_compression option and default enabling of doc_values, which can reduce the hardware footprint of clusters by 15-25%. These enhancements leverage Lucene 5.0's DEFLATE algorithm to achieve compression ratios between 0.429 and 1.117, depending on configuration and data characteristics. The article discusses the implications of these changes, noting that while compression can lead to performance penalties during decompression, these are mitigated in specific query scenarios. Additionally, doc_values offer a more efficient way to manage analytics workloads by storing data off the JVM heap, thus reducing the constraints on hardware requirements. Users are encouraged to conduct their own experiments to understand the impact of these features on their specific data sets and configurations.

Russ Savage introduces an open-source Logstash plugin designed to efficiently extract Salesforce data into Elasticsearch, enabling advanced data analysis using Kibana. This plugin allows users to query almost any Salesforce object and incorporate the data into the Logstash pipeline, which can then be outputted to analytics tools like Kibana for more insightful visualizations. The system addresses challenges in sales and marketing operations by providing interactive time-series snapshot analysis across various dimensions, correlating Salesforce data with external data sources, and overcoming the limitations of Salesforce's built-in reporting. Users can customize the Salesforce query process, configuring specific Salesforce objects and fields to be imported and transformed within Elasticsearch, thereby enriching the data and enhancing its utility for trend analysis and decision-making. The setup process involves configuring Logstash with Salesforce credentials, enabling the extraction of data from Salesforce's API, and regular data collection can be automated through scheduled tasks. Visualization using Kibana facilitates trend analysis, allowing users to visualize data changes, such as daily pipeline variations, and identify specific opportunities that have changed over time, thus providing deeper insights into sales trends and customer health.

Elasticsearch 1.7.2, based on Lucene 4.10.4, has been released as a stable update focusing on bug fixes and performance improvements. Key enhancements include better shard allocation logic when using shard allocation awareness, a resolution for a rare recovery issue identified through randomized testing, and improvements to the snapshot/restore API, which now properly accepts index options like ignore_unavailable. Additionally, adjustments have been made to delayed shard allocation to further enhance functionality. Users who have encountered these specific bugs are encouraged to upgrade to this version, and feedback can be shared on Twitter or the company's forum, with issues reported on GitHub.

Elastic's announcement highlights various global events and meetups centered around Elastic products and technologies, scheduled for mid-September 2015. Key events in Europe include the Kiratech Event in Verona, Italy, where Emmanuel Brochard will discuss leveraging data, and the Jug Summer Camp in La Rochelle, France, featuring David Pilato on Big Data. In North America, several DevOps meetups will explore ELK stack use cases, with notable gatherings in Richmond, Reno, and Phoenix, where Trax Tech developers and LinkedIn's Lucas Ewalt will share insights. Additional meetups are happening across Europe, Africa, and Asia, showcasing the diverse interest in Elastic solutions. The announcement concludes with an invitation for those interested in hosting or speaking at meetups to reach out for support and promotional materials.

The blog post by Samir Bennacer discusses the "Hot-Warm" architecture for Elasticsearch versions 1.x and 2.x, which is tailored for large-scale, time-based data analytics. This architecture involves a tiered structure with three distinct types of nodes: master, hot, and warm nodes. Master nodes are dedicated solely to cluster management tasks, while hot nodes handle indexing and store the most recent, frequently queried indices, requiring high-performance SSD storage. Warm nodes, on the other hand, are optimized for storing older, less frequently accessed read-only data on larger, slower disks. The setup involves tagging nodes as either hot or warm to manage data allocation effectively, with Elasticsearch automatically moving indices to warm nodes as they age, which can be automated using tools like Curator. This approach enhances resource efficiency and ensures optimal performance by offloading older data to less resource-intensive nodes.

In the blog post "When and How To Percolate - Part 2," Dale McDiarmid delves into techniques for enhancing the performance of Elasticsearch's Percolator, building on the concepts introduced in the first part of the series. The piece discusses the use of filters, which attach metadata to percolator queries to minimize query sets and improve execution efficiency, resulting in significantly faster query evaluations. Strategies like sharding and routing further optimize performance by distributing queries across multiple shards, thus increasing throughput through parallel execution while balancing workloads. The multi-percolate API is introduced as a tool for bundling requests to optimize network performance, with replicas providing high availability and increased throughput, contingent on adequate resource allocation. Memory and CPU usage are emphasized as critical considerations, with recommendations for dedicated resources and environments to maintain consistent performance. The post highlights the importance of understanding and managing resource utilization, particularly in high-throughput scenarios, while providing links to exhaustive test results and methodologies for further exploration.

In the first week of September 2015, Elastic announced a series of events and meetups across the globe focused on Elasticsearch, Logstash, and Kibana. In Europe, notable events included talks at Java Zone in Oslo, Norway, where speakers Sigmund Hansen and Aleksander M. Stensby discussed configurable log analysis and data value unlocking with Elasticsearch. In North America, the Heartland Developers Conference in Omaha, Nebraska, featured a hands-on Elasticsearch workshop by Shaunak Kashyap. Various meetups were also scheduled in cities like San Francisco, San Diego, Boston, Dublin, and Canberra, offering opportunities for community engagement and knowledge sharing. Elastic invited individuals interested in hosting meetups or delivering talks to reach out for support and promotional materials.

Elastic has released Beats 1.0.0-beta3, introducing Topbeat alongside the improved Packetbeat, both of which serve as shippers of operational data to Elasticsearch with enhanced features. Topbeat functions similarly to the Linux/Unix "top" command, offering system-wide and per-process resource monitoring across Linux, Windows, and OS X platforms, and integrates with Kibana for visual monitoring. Packetbeat now supports DNS and Memcache protocols, with specific improvements for UDP protocols and Windows support, including native service operation and enhanced network interface selection. The release also eliminates the need for the libpcap runtime dependency on Linux, broadening compatibility, and offers new developer guides for creating custom Beats or adding protocols to Packetbeat, encouraging community interaction and development.

Logstash 1.5 introduces a powerful feature allowing users to add metadata to events, which remains transient within the Logstash pipeline and is not serialized in outputs, thereby simplifying configuration and enhancing log processing efficiency. This feature facilitates the addition of custom data, filtering, and conditional operations based on metadata without the need for temporary fields. Users can access metadata fields using a specific syntax and leverage them for various tasks, such as optimizing date filters, creating unique identifiers for Elasticsearch documents, and managing intermediate processing results to avoid unnecessary storage. By using metadata, Logstash configurations become less complex, reduce storage requirements, and enhance performance, making metadata a valuable tool in streamlining data processing workflows.

The _cat API is a valuable tool for command-line debugging and monitoring in Elasticsearch, providing a more streamlined alternative to the JSON-oriented main API. It allows users to efficiently gather information on node memory usage, thread pool activity, and index metrics, which are crucial for identifying and addressing performance issues. By utilizing command-line utilities like curl, sort, and watch, users can quickly pinpoint nodes experiencing high memory pressure, monitor thread pool operations, and track data migration between clusters. These capabilities make the _cat API particularly useful for managing Elasticsearch clusters in real-time, offering immediate insights into system performance and potential bottlenecks. Additionally, the article highlights the versatility of combining the _cat API with other command-line tools to create custom monitoring solutions, emphasizing its role in simplifying Elasticsearch administration and troubleshooting.