April 2018 Summaries

In a blog post by Paul Coghlan, the process of using Minio for storing snapshots in Elastic Cloud Enterprise (ECE) is explored, offering a solution for on-premise storage needs. Initially, ECE supported snapshot backups mainly for public cloud services like AWS S3 and Google GCS, but with the release of ECE 1.1, Minio, an open-source object storage server compatible with the AWS S3 API, is now supported for local storage. This guide details the installation of a single Minio node using Docker, providing step-by-step instructions for both internet-connected and air-gapped environments. Configuring Minio involves setting up a snapshot repository within ECE, specifying details such as the bucket name, access key, and endpoint. While the blog covers a basic installation suitable for testing, it notes that production environments require a more complex setup using Docker Swarm or Kubernetes for high availability. Future updates are anticipated to include support for additional storage backends like NAS, Azure Blob Storage, and Google Cloud Storage.

Keeping up with Kibana is a series of posts detailing recent developments in the Kibana project, focusing on both upcoming releases and new features. The team is preparing for the release of version 6.3, while also integrating new components into the Elastic UI Framework, such as the EuiEmptyPrompt and EuiTabbedContent, and fixing issues with existing elements. Additionally, enhancements include support for significant term aggregations in tag cloud visualizations, which helps in identifying emerging topics during searches. Progress is also being made in Kibana applications, with updates like converting the dashboard list page to the EUI Framework, adding multiple color ramps to coordinate maps, and introducing HTML tooltips in Vega, alongside community-contributed features such as toggling the filter bar in the Discovery app.

In February, Elastic hosted a Django Girls workshop in San Francisco with the aim of introducing programming to underrepresented groups, particularly women, in a free and engaging way. The event, which was the company's second employee-organized workshop, brought together 45 attendees, 25 coaches, 10 volunteers, and 7 speakers for lightning talks, covering topics like imposter syndrome and programming career paths. Django Girls, a global non-profit organization, provides a framework for similar workshops worldwide, focusing on teaching Python and Django through setting up a personal blog site. The event saw a significant growth from the previous year, with doubled participation and improvements based on past experiences. Attendees participated to gain new skills, understand coding perspectives, or simply learn something new, while coaches enjoyed sharing their passion for programming. Elastic supports such initiatives through its volunteer policies and sponsorships, and the workshop was enhanced by contributions from event sponsors and community organizations. The workshop was marked by an energetic atmosphere filled with learning, collaboration, and "a-ha" moments, embodying Elastic's commitment to broadening tech accessibility and demystifying complex topics.

The blog post by Sherry Ger provides a comprehensive guide on setting up a three-node Elasticsearch cluster on Google Cloud using the Google Compute Engine (GCE) discovery plugin, which utilizes GCE APIs for automatic unicast discovery of nodes. It outlines the necessary prerequisites, such as having a Google Cloud Platform project ID and installing the Google Cloud SDK, before detailing the steps to configure firewall rules, create GCE instances, and install required software like the Java Development Kit (JDK) and Elasticsearch. The post further explains the process of configuring Elasticsearch with the GCE discovery plugin and setting up necessary configurations in the elasticsearch.yml file. Additionally, it covers creating a custom GCE image from an instance to facilitate the deployment of additional instances across different projects, elaborating on steps to snapshot the disk, create a compressed tar file, and store it in a Google Cloud Storage bucket. Finally, the post provides instructions on creating new instances from the custom image and verifying the successful setup of the Elasticsearch cluster, while hinting at future enhancements such as installing X-Pack for security and using the Google Cloud Storage repository plugin for snapshot capabilities.

Composite aggregations in Elasticsearch 6.1 are a significant advancement that enable efficient pagination of large datasets, as demonstrated through an example involving Sliceline, a fictional pizza delivery service. This feature allows for seamless navigation through large result sets, creation of new indices from aggregation results, and consistent API response times. Unlike previous versions, which required retrieving all preceding results, composite aggregations allow users to paginate through results without needing to skip and limit, thus reducing resource consumption and ensuring fast response times. The introduction of composite aggregations facilitates various tasks like building analytical services, creating histograms, and calculating averages, while also being integrated into upcoming Elasticsearch features such as data rollups and SQL group_by. This enhances the ability to handle large datasets efficiently and reliably, making it a valuable tool for data-heavy applications.

The text provides an overview of using language models as an alternative scoring method in Elasticsearch for information retrieval, contrasting with the default BM25 method. Language models, commonly used in Natural Language Processing, estimate the probability distribution of terms within a language, with the Unigram Language Model assuming term independence. In this context, each document in a collection is represented by its own language model, and documents are ranked based on their likelihood to generate a given query. The text explains how probabilities are estimated using term frequency and document length, with smoothing techniques like Jelinek-Mercer and Dirichlet smoothing applied to adjust these models by incorporating a background model, thus avoiding zero probabilities for unseen terms. Practical examples demonstrate how to implement these language models in Elasticsearch, including setting up indices and interpreting query scores, and the text concludes with a mention of future blogs intended to explore comparisons with other scoring methods.

The blog post, "How to Practical BM25 — Part 2: The BM25 Algorithm and its variables," delves into the intricacies of the BM25 algorithm, a widely used method for ranking documents based on relevance in search engines like Elasticsearch. The discussion begins with the mathematical structure of the BM25 formula, breaking it down into components such as query terms (q_i), inverse document frequency (IDF), term frequency (f(q_i, D)), and parameters like k1 and b, which influence how documents are scored. The post explains how IDF penalizes common terms and how document length relative to the average field length affects scoring. It also discusses the role of k1 in determining term frequency saturation, illustrating its impact through examples with varying k1 and b values in different document sets. The results show how these parameters influence the scoring of documents based on term frequency and document length, offering practical insights into optimizing relevance scoring in Elasticsearch. The post concludes by setting the stage for the final part of the series, which will explore considerations for selecting appropriate values for b and k1.

In the final installment of a series on Practical BM25, the focus is on optimizing the parameters b and k1 for Elasticsearch's similarity ranking algorithm. While the default values of b = 0.75 and k1 = 1.2 generally work well, fine-tuning these parameters is suggested only after exploring other relevancy improvements like boosting, synonyms, and fuzziness. The article details how b and k1 should be adjusted based on document length and term saturation, with b ranging from 0 to 1 and k1 from 0 to 3, depending on the corpus. It underscores the importance of experimentation and testing across various queries and documents when adjusting these parameters, using tools like Elasticsearch's Rank Eval API and Explain API to aid in this process. The piece concludes by highlighting that while BM25 is effective, no single algorithm or parameter set is universally optimal, and most tuning efforts should focus on Elasticsearch's expressive query language, index controls, and user feedback, with BM25 parameter adjustments as a deeper level of refinement.

The blog post is the first in a three-part series focused on the practical application of the BM25 similarity algorithm in Elasticsearch, particularly examining how the number of shards influences relevance scoring in text document searches. Elasticsearch defaults to five primary shards per index, and this setup can affect scoring because each shard holds a subset of the data, leading to variations in term frequencies across shards. For example, when searching for documents with the title "Shane," the distribution of documents across shards results in different relevance scores, even for similar entries. To achieve more consistent scoring, users can reduce the number of shards or employ the `search_type=dfs_query_then_fetch` parameter, which calculates scores by first gathering distributed term frequencies across shards. However, this method is not enabled by default due to the additional processing time required, making it unnecessary for cases prioritizing speed over scoring precision. The post sets the stage for deeper exploration of the BM25 algorithm and its variables in subsequent parts of the series.

At the Elastic{ON} 2018 conference, a session on getting started with Elastic APM (Application Performance Monitoring) highlighted its capabilities in monitoring application performance, tracking user requests, and diagnosing errors. The demonstration by Elastic APM Tech Lead Ron Cohen explained the straightforward setup process involving downloading the APM server, installing the APM agent into applications, and redeploying them to stream data into the APM UI in Kibana. This interface allows users to visualize performance metrics, analyze response times, and troubleshoot issues using detailed insights into application processes, SQL queries, and cache calls. The APM module provides comprehensive error data, including stack traces and error frequency, aiding in performance optimization and debugging. The session also introduced real user monitoring, which was in beta, and discussed the integration of APM data with machine learning for anomaly detection, offering advanced functionalities for enhancing application performance. Other topics from the conference included security analytics with Kibana, machine learning forecasting, and various Elastic Stack features.

At Elastic{ON} 2018, machine learning developments for the Elastic Stack were showcased, highlighting its evolution from automated anomaly detection to event forecasting using Elasticsearch time series data. Steve Dodson, the team lead for machine learning at Elastic, demonstrated new features using New York City taxi data, identifying anomalies such as low trip volumes during a snowstorm and forecasting future taxi trip times. The forecasting capabilities, which aggregate historical data to predict future trends, were a focal point, offering potential for operational optimizations such as disk space management and KPI trend analysis. These advancements, demonstrated through various sessions, illustrate how Elastic's machine learning tools can automate and enhance operations across different use cases, with further insights provided into the modeling techniques behind these capabilities.

Endgame released Ember, an open-source benchmark dataset aimed at advancing research in static malware detection by providing 1.1 million portable executable file sha256 hashes, along with metadata and derived features, without disclosing the actual files to protect intellectual property. This dataset, which includes both training and test samples categorized as malicious, benign, or unlabeled, allows researchers to measure the effectiveness of new machine learning techniques in identifying unseen threats, despite the evolving nature of malware. The Ember benchmark model, a gradient-boosted decision tree trained with LightGBM, achieves a high area under the ROC curve score, though it is not intended for production use. Instead, it serves as a research tool to explore improvements in feature selection, model parameter optimization, and the development of new featureless neural network-based models, encouraging further research in the field and providing a target for studies on countering machine learning-driven attacks.

In the blog post "Keeping up with Kibana: This week in Kibana for April 16, 2018," Jin Mu discusses the concepts of internationalization (i18n) and localization (l10n) in the context of adapting Kibana for non-English speaking users, such as those in China and Germany. The article explains the distinction between i18n and l10n, highlighting that internationalization involves designing a product for easy localization, while localization adapts a product to specific cultural and legal requirements. It details the challenges of localizing JavaScript applications, noting the lack of a standard approach and the presence of various frameworks, each with unique advantages and limitations. The blog outlines a comparison of six popular JavaScript localization frameworks—Angular-translate, React-intl, i18next, MessageFormat.js, Format.js, and Fluent.js—assessing their suitability for Kibana, which uses both React and Angular. The analysis considers factors like syntax ergonomics, pluralization support, and server-side compatibility. Ultimately, the decision was made to adopt a solution based on the FormatJS core libraries due to its support for server-side use, adherence to ICU message syntax, and ease of use across different frameworks, encouraging readers to evaluate their business needs when selecting a localization framework.

At the Elastic{ON} 2018 conference, a demo by Asawari Samant showcased how the Elastic Stack, including Elasticsearch, Kibana, Beats, and Logstash, can enhance security analytics by streamlining the detection and investigation of security breaches. Through the use of lightweight data shippers like Winlogbeat and Packetbeat, the Elastic Stack simplifies log collection, while Kibana offers visualization tools for identifying malicious activities. The anomaly explorer in Kibana allows users to zoom into anomalies, assess their severity, and explore potential threats by analyzing hosts and domains involved in suspicious activities. The time series visualization features help track events like unauthorized logins and access to honey files, providing a comprehensive view of network traffic and user actions. The collaboration of these tools enables fast and efficient analysis, allowing analysts to address breaches swiftly and effectively.

Pius Fung's blog post discusses the evolution and enhancements of the X-Pack reporting feature in Kibana since its initial release, highlighting key user-requested improvements such as CSV export capabilities, custom logos, and layout preservation in PDF reports. The post notes the transition from PhantomJS to Chromium as the default reporting backend, addressing previous issues like truncated legends and chart misalignment. It provides troubleshooting advice for common issues, such as font dependencies and encryption key mismatches, and suggests configuration adjustments to optimize report generation and manage reporting indices. Additionally, it offers guidance on enabling debug logging and handling older reports, emphasizing the importance of consistent settings across Kibana instances to ensure smooth reporting operations.

In the April 2018 update of the "Keeping up with Kibana" series, significant progress was reported in the Kibana project, focusing on new developments and enhancements. The ongoing EUIfication process has led to the creation and improvement of UI components, such as the new EUI Combo Box and expandable rows for EUIBasicTable, along with updates to EuiIconTip and other features aimed at improving responsiveness and accessibility. Efforts to improve the Elastic Map Service included refining map data, adding tilemap configurations, and updating world layers for better usability. Additionally, the Kibana query language has been enhanced with an auto-complete feature to suggest field names, operators, values, and conjunctions, reflecting a broader push to streamline the user experience.

Voxpopme, a leading video insight platform founded in 2013, has transformed its technology stack to cope with rapid growth, particularly in video survey processing, by integrating Elasticsearch to significantly enhance search performance. Previously reliant on a monolithic PHP application with separate MySQL and MongoDB databases, Voxpopme faced scalability issues that prompted a shift to Elasticsearch, enabling faster, more efficient data retrieval by consolidating disparate data into single JSON documents. Initial testing with Amazon Elasticsearch Service proved insufficient under load, leading to the adoption of Elastic Cloud, which delivered superior query performance and latency reduction. The transition not only improved user experiences but also provided new opportunities for data visualization and operational insights, allowing Voxpopme to better monitor server health and optimize resource use. The company continues to refine its processes and explore Elasticsearch for internal data analysis to support ongoing growth and efficiency improvements.

Securing personal data under GDPR is essential, and Elasticsearch, powered by X-Pack, offers robust security features to meet these requirements. The post outlines the importance of protecting 'Personal Data' as defined by GDPR, which includes both Direct and Indirect Identifiers, and discusses pseudonymization as a method to enhance data security. It emphasizes the need for encryption in data transit and at rest, and highlights X-Pack's capabilities in user authentication through integration with systems like LDAP and Active Directory. The primary focus is on implementing role-based access controls (RBAC) and attribute-based access controls (ABAC) in Elasticsearch, demonstrating how these methods can limit data access based on user roles, document-level security, and field-level security, ensuring compliance with GDPR. Examples illustrate how RBAC can manage access to datasets by country and how ABAC can use user metadata to control data visibility, offering a scalable solution for both small and large organizations.