June 2016 Summaries

Elasticsearch 5.0.0-alpha4, based on Lucene 6.1.0, is the fourth pre-release in the series aimed at testing new features and gathering user feedback before the official 5.0.0 release. This alpha version introduces significant updates, such as the Java REST client, which offers a more stable and flexible HTTP client solution compared to the Java Transport client, and new APIs like Rollover and Shrink for better index management. While the release is not intended for production use, it provides various tools like the Elasticsearch Migration Helper to assist in transitioning from version 2.3.x to 5.0. The release also highlights improvements in search performance, such as the ability to wait for refreshes and support for aggregations in the search profiler. Users are encouraged to download the release, test their applications, and provide feedback on potential issues to help refine the final product.

Marcelo Rodriguez's blog post provides a detailed guide on using Elastic Shield to implement document-level security in Elasticsearch, demonstrated through a scenario involving three users with different access needs to regional documents. The process involves configuring authentication realms, indexing sample data, creating roles and users with specific access privileges, and setting up Kibana for data visualization. Users can view their respective data through a Kibana visualization based on their assigned roles, which restrict or allow access to documents from the EAST or WEST regions, or both for the manager. The guide also covers testing user access and notes that while the example uses basic configurations, it can be expanded to include more complex security setups, such as additional authentication realms or field-level security. It emphasizes the use of the Roles API for defining roles, suggesting that this method is preferable to using the roles.yml file since the latter will override API definitions if both are used.

Elastic Stack 5.0 alpha 4 introduces several new features and improvements across its components, including Elasticsearch, Kibana, Logstash, Beats, and ES-Hadoop, although it remains an alpha release not recommended for production use. Highlights include Elasticsearch's new low-level Java HTTP/REST client, Rollover and Shrink APIs for easier index management, and a Migration Helper tool for transitioning from version 2.3.x to 5.0. Kibana now supports direct CSV data import into Elasticsearch and improved monitoring through X-Pack, along with a new Management app for configurations. Logstash sees enhancements in plugin-level metrics for better debugging and optimizations for Elasticsearch output, while also supporting Apache Kafka version 0.10. Beats introduces a new MongoDB monitoring module and an Apache module dashboard in Kibana, and ES-Hadoop has also been updated to version 5.0.0-alpha4.

Kibana 5.0.0-alpha4 has been released, introducing several new features and improvements, including the ability to import CSV data directly into Elasticsearch, enhanced monitoring for Kibana through the X-Pack with tracking capabilities for requests and memory usage, and more flexible visualization options like drag-and-drop sorting of buckets. The release also features a revamped "Management" app, replacing the "Settings" app, which serves as a hub for configuring the Elastic stack and managing users and roles. Although this release is alpha software, intended for testing rather than production use, it paves the way for the upcoming beta release, with developers encouraged to report bugs and provide feedback.

The article by Nicolás Bevacqua explores the journey of integrating and utilizing the Elastic Stack, formerly known as the ELK Stack, comprising Elasticsearch, Logstash, and Kibana, for better data visualization and server metrics analysis. The author explains how this powerful trio can provide insights into platform operations by setting up Elasticsearch for indexing, Logstash for data streaming from various sources, and Kibana for visualizing the collected data. The narrative details the step-by-step process of upgrading to Elastic Stack 5.x, configuring services, and integrating Node.js and nginx logs into the system. The article highlights the practical applications of Logstash in processing diverse data inputs and outputs, as well as the use of Kibana to create visualizations such as pie charts, line charts, and geographic maps to understand application performance and traffic patterns. Additionally, the author discusses the importance of automation and security using X-Pack to enhance the Elastic Stack's functionality, ultimately offering comprehensive insights into server performance through a centralized dashboard.

Elasticsearch's percolator has undergone significant enhancements in version 5.0, making it more flexible and efficient. Key improvements include the ability to skip evaluating most percolator queries, reducing the execution time, and the introduction of a percolator field mapper that allows queries to be stored in any field and type, rather than a special .percolator type. This update is part of a broader refactoring effort that aims to enhance scalability and performance by ensuring that only relevant queries are evaluated, thereby alleviating memory pressure and improving speed. The previously used percolate and multi percolate APIs have been replaced by more versatile search and multi search APIs, which not only return more detailed information but also support additional features like pagination. Scoring has also been revamped to reflect how well a percolator query matches a document using the in-memory Lucene index. Despite these changes, existing percolate and mpercolate APIs are still operational but deprecated, with plans to phase them out in future releases. Users are encouraged to transition to the new system and explore the ongoing improvements to the percolator field mapper for enhanced query execution.

The text is part two of a multi-part series detailing the integration of Apache Kafka with the Elastic Stack, focusing specifically on operational aspects such as capacity planning and monitoring when using Kafka and Logstash in production environments to handle large data volumes. The article emphasizes the importance of Apache ZooKeeper for Kafka operations, discussing how to set up a quorum configuration of ZooKeeper instances for stability. It addresses the significance of Kafka brokers in data retention and replication, explaining how the number of brokers correlates with data storage capacity. Logstash is highlighted for its flexibility in scaling horizontally and managing complex data transformation tasks, while also stressing the importance of careful capacity planning for external systems rather than Logstash itself. The text also covers Kafka's offset management system and methods to ensure message delivery guarantees, including handling potential data duplication. Monitoring tools such as Kafka's CLI tool, JMX, and the Elastic Stack's Kafkabeat and Metricbeat are recommended for tracking system performance and consumer lag. The article concludes by mentioning forthcoming updates in Kafka, including new security features, and promises further exploration of these features in the next installment.

Bol.com, an online superstore with over 11 million products and 230,000 active sellers, faced challenges in its search functionality due to its flat document model, which limited the visibility of all relevant offers for products. To address this, the company explored using Elasticsearch for a scalable and flexible data model that could handle the high volume of searches and data updates, which could peak at 20 million offer updates per day. Various data models were tested, with the nested document model emerging as the most effective, though it posed challenges with update behavior. The testing, done in collaboration with Anne Veling, highlighted the need for improved query time performance and led to bol.com's decision to integrate Elasticsearch into its search infrastructure. Maarten Roosendaal, an IT Architect at bol.com, played a key role in this initiative, leveraging his expertise in scalable search and SEO solutions.

Running Elasticsearch on AWS involves a detailed process of provisioning and configuring EC2 instances to create a three-node Elasticsearch cluster using version 2.3.3. The guide outlines steps such as selecting the appropriate Amazon Machine Image (AMI), configuring instance details, setting up storage, tagging instances, and configuring security groups. Users are advised to install the necessary plugins, including the AWS cloud plugin, and to configure Elasticsearch to optimize performance, particularly by adjusting the JVM heap size. Starting up and verifying the cluster involves ensuring all nodes have the same version and configuration, with the aim of achieving a "green" status for optimal operation. While managing Elasticsearch on AWS requires a hands-on approach, Elastic Cloud offers an alternative with a managed service that simplifies deployment and management, supporting features like security, automatic scaling, and backup, and is available for a free trial without a credit card requirement.

The article by Nik Everett discusses the mechanics and implications of the "refresh" process in Elasticsearch, which is essential for making changes to documents available for search after they have been written to disk. While Elasticsearch automatically performs refreshes at regular intervals, when the indexing buffer is full, or on demand, each method has its own advantages and drawbacks. For applications where immediate searchability post-update is critical, forcing a refresh is an option but can lead to inefficiencies due to the creation of small index segments. The upcoming Elasticsearch 5.0 offers a hybrid solution with the ?refresh=wait_for option, allowing changes to become searchable without immediate segment inefficiency, although it requires waiting for the refresh cycle to complete. The article emphasizes choosing a refresh strategy that balances the need for search immediacy with the performance impacts of index segment creation and the specific requirements of the application in use.

Apache ZooKeeper, an open-source distributed coordination service initially developed by Yahoo and now managed by Apache, plays a crucial role in storing critical data, necessitating a reliable backup system. ZooKeeper databases contain persistent data, accessed via traditional CRUD activities, and ephemeral data, which is associated with state machine semantics and implies a specific state. In production, ZooKeeper is deployed on multiple processes, ensuring consistency by requiring that a majority of processes receive writes. The data stored can be transient, a source of truth, or stateful, with ephemeral nodes tied to client sessions that expire when the sessions do. Challenges in backing up and restoring ZooKeeper arise from its transient and stateful data, as it lacks built-in support for these processes. Improper restoration can lead to inconsistencies in the distributed state machine, as exemplified by scenarios where restored ephemeral nodes conflict with current client states. Recommendations for ZooKeeper backups include copying transaction and snapshot logs, filtering out ephemeral node transactions to prevent data inconsistencies, and ideally closing all clients before restoration to effectively manage session expirations.

Joshua Rich, a support engineer at Elastic, shares his experience working with the eBay team, highlighting the personalized approach Elastic takes by dedicating support engineers to specific accounts rather than rotating them. This method fosters deep relationships and intimate knowledge of customer projects, creating a collaborative environment akin to working with friends rather than mere colleagues. Rich first met Sudeep Kumar from eBay in person at the Elastic{ON} 16 event, after months of remote collaboration, and emphasized the excitement and challenges of working at eBay's large scale. He appreciates the opportunity to contribute to eBay's success and values the sense of community and passion within Elastic, where the distributed support team maintains a strong sense of togetherness and continuously learns from and supports their customer base.

The MyGene.info Development Team at the Scripps Research Institute has developed MyGene.info and MyVariant.info, utilizing Elasticsearch to streamline the fragmented landscape of gene and variant data, allowing researchers to efficiently access up-to-date genetic information in a consistent JSON format. Spearheaded by Dr. Chunlei Wu, these services aggregate data from millions of genes and variants across numerous databases, addressing the challenges of scalability and performance in bioinformatics research. By employing Elasticsearch, the team ensures that users can perform flexible, high-speed searches specific to their needs, such as filtering variant annotations or gene data. The services, which are free for public use but may have data source-specific restrictions, have proven capable of handling high traffic, with MyGene.info alone managing over 10,000 requests per minute from thousands of unique monthly users. The visualization of service usage is facilitated by Kibana, helping the team distinguish traffic sources and manage client requests. The development of these tools, part of Dr. Andrew Su's computational biology lab, aims to make genetic information more accessible and useful to the research community, with plans to expand the scope to cover other areas with fragmented data sources.

The Unicode Consortium, a non-profit organization established in 1991, endeavors to standardize and support global languages, ensuring their compatibility with digital platforms. Its Adopt-a-Character program generates funds to enhance technological support for almost 98 percent of the world's digitally disadvantaged languages, which often lack representation in operating systems, browsers, and apps. Elastic, a company reliant on Unicode's work, participates in this program by allowing its engineers to adopt characters, reflecting personal or professional significance. This initiative not only aids in language conservation but also encourages team members to express individuality and cultural connections through their chosen characters. Elastic's involvement highlights its commitment to supporting Unicode's mission of language inclusivity and technological standardization.

Nicolás Bevacqua shares his experience of integrating Elasticsearch as the search provider for his blog, Pony Foo, detailing the technical steps involved in the process. He begins by explaining Elasticsearch's role as a REST HTTP service that enhances Apache Lucene's indexing and search capabilities, enabling scalable and efficient searches without relying on Java. Bevacqua outlines the installation of Elasticsearch on different platforms, the setup of a Node.js client for interacting with the Elasticsearch instance, and the creation of indices and mappings to store blog data effectively. He highlights the importance of keeping the Elasticsearch index up-to-date with blog content changes, employing a bulk API for initial data synchronization, and creating complex queries using Elasticsearch's DSL to allow for advanced search functionalities. Additionally, Bevacqua discusses deploying Elasticsearch on a Debian environment and automating the deployment process to ensure seamless integration with his existing infrastructure. The article emphasizes the improved accuracy and relevance of search results and related articles on his blog, hinting at future plans to integrate Elasticsearch with Logstash for enhanced analytics through Kibana dashboards.

Marty Messer reflects on the unique and empathetic approach to customer support at Elastic, emphasizing the company's deep-rooted commitment to user success and satisfaction. Unlike traditional support models, Elastic integrates its support team closely with its engineering team, ensuring that the feedback loop between customer needs and product development remains seamless. This integration stems from Elastic's origins, where co-founder Shay Banon personally supported early users of Elasticsearch. The company's supportive culture is bolstered by hiring developers who are passionate about Elastic's products and were often users themselves before joining. Elastic's ethos is that its engineering serves as its marketing, striving for transparency and honesty in its product capabilities, which in turn allows support engineers to engage in meaningful and enjoyable interactions with customers. This approach not only enhances customer satisfaction but also creates a collaborative, family-like atmosphere among employees and clients, as evidenced by genuine interactions at Elastic's events.

Martin Smith, a DevOps Engineer at Rackspace, reflects on his experiences at Elastic{ON} 16, a conference focused on Elastic's product developments and open-source contributions, held in San Francisco. During his talk, Smith shared insights into the challenges of building Chef automation for Elasticsearch and invited new contributors to join the effort, highlighting updates to the Chef cookbook and the importance of collaboration in open-source projects. Smith appreciated the distributed nature of Elastic and enjoyed connecting with global peers and Elastic staff, gaining valuable knowledge about Beats architecture and Kibana changes. His travels as a digital nomad, participating in Remote Year, and his commitment to open source and community mentoring are underscored as key aspects of his professional and personal life.

Chad Pryor's article provides a comprehensive tutorial on setting up and using Elastic Cloud with a sample dataset, specifically focusing on Elasticsearch 2.x and Kibana Version 4.x. It guides users through the process of creating an Elastic Cloud account, setting up a hosted Elasticsearch cluster, securing the cluster by configuring user access, and enabling Kibana for data visualization. The tutorial includes detailed steps on importing data into Elasticsearch using Logstash with sample Apache logs and visualizing this data in Kibana by adding indices and importing a sample dashboard. The guide emphasizes securing credentials and provides useful links for further exploration of Elastic Stack components such as Beats and Logstash, while also pointing to updated resources and training options for users seeking to deepen their understanding of the platform.