April 2020 Summaries

Elastic Enterprise Search provides a flexible and scalable solution for modern customer service experiences by enabling brands to enhance their digital support channels with powerful search capabilities. The platform offers products like App Search and Workplace Search, which allow organizations to deliver fast, relevant search results across various applications, improving both self-service and agent-assisted channels. Elastic's solutions help reduce costs by lowering reliance on expensive support routes like phone, email, and chat while maintaining customer satisfaction through intuitive controls and tunable relevance settings that don't require coding. Additionally, built-in analytics provide insights into user behavior, helping identify content gaps and optimize support content, and the pricing model based on consumption offers a transparent alternative to traditional per-seat pricing. Elastic Enterprise Search empowers customer service teams to deliver efficient and effective support, aligning with the growing trend towards digital transformation while supporting the need for human interaction in complex issues.

Elastic Cloud on Kubernetes (ECK) 1.1.0 has been released, enhancing the management of Elasticsearch clusters within Kubernetes environments by introducing improved cross-cluster support and declarative users and roles. This update simplifies setting up cross-cluster search and replication, allowing for efficient data distribution across global environments, and facilitates the management of users and roles through Kubernetes Secrets, supporting GitOps-style deployments. The release also includes quality-of-life improvements such as resource validation at admission time, automatic Kibana encryption key specification, and role-based access control for more secure associations, thereby streamlining operations and bolstering security for users managing Elasticsearch deployments across various Kubernetes clusters.

Binary classification, a supervised machine learning method, is widely used to categorize data into two groups and has applications in areas such as fraud detection and security analytics. Elastic's implementation of binary classification reportedly outperforms other models when benchmarked against open datasets, showing a 5.1% improvement in mean accuracy on average. Unlike other algorithms requiring user-set parameters, Elastic's approach automatically estimates parameters through cross-validation. Elastic's method, based on gradient boosted decision trees, excelled in comparison to other algorithms across 31 datasets from the OpenML Curated Classification Suite, achieving top performance in 26 datasets. The process demonstrated efficient runtimes, averaging just over 2 minutes per analysis, and Elastic provides a 14-day free trial for users to explore its capabilities in binary classification.

In April 2020, Elastic Security, which includes endpoint security, SIEM, threat hunting, and cloud monitoring, participated in MITRE's APT emulation round 2, focusing on evaluating detection capabilities against APT29 techniques. Elastic Security excelled in the evaluation, identifying 100% of the attack steps and over 90% of the sub-steps, demonstrating its ability to eliminate blind spots and provide actionable telemetry, even as adversaries use techniques like PowerShell and WMI to evade detection. The evaluation, which aimed to assess visibility across a single attack rather than make direct vendor comparisons, reinforced Elastic’s mission to provide accessible security solutions by leveraging the Elastic Stack's capabilities for massive-scale data ingestion and analysis. Over the past six months, Elastic Security has made significant advances, including eliminating endpoint-based pricing and integrating new features, while being recognized as a strong performer in enterprise detection and response by Forrester. Elastic continues to work towards a unified security solution, inviting users to stay informed about upcoming developments.

MITRE's recent evaluation of 21 security products, focused on simulating attacks using the ATT&CK framework, allowed security teams to analyze detection capabilities across various vendors. This evaluation involved MITRE's red team executing orchestrated attacks, while vendors acted as the blue team tasked with detecting these threats. Unlike typical assessments, detection was only the starting point, emphasizing the importance of understanding the full attack chain. Elastic leveraged Kibana to visualize the results, offering users the ability to filter and analyze detection data by vendor. This approach highlights detection of both major and sub-steps in the attack scenarios and provides insights into the performance of products, excluding managed security service providers. The visualizations displayed not only detections but also misses, offering a comprehensive view of vendors' capabilities in detecting tactics and techniques across all evaluation steps. Elastic expressed gratitude towards MITRE for the evaluation, noting it as a valuable opportunity to refine their security solutions and provide an improved user experience.

Elasticsearch Service has expanded its presence on Microsoft Azure by launching in the London (uksouth) region, marking its sixth Azure region globally and second in Europe. Existing users can access the service immediately, while new users can sign up for a free 14-day trial. The service offers advanced features such as machine learning, index lifecycle management, and intuitive visualization with Kibana Lens, alongside use case-ready deployment templates for ease of configuration and scaling. The collaboration between Elastic and Microsoft aims to enhance the Elastic Stack experience for Azure customers, with plans to offer Elasticsearch Service subscriptions through the Azure Marketplace. Comprehensive resources and support are available for users looking to migrate their Elasticsearch data to the new region.

Elastic, a company with roots in open-source development, embraces a distributed model that is integral to its DNA, mirroring the architecture of its core product, Elasticsearch. This approach means Elastic can scale efficiently and hire talent from around the world, fostering diversity and inclusivity. The distributed nature of the company allows it to adapt swiftly to changes, ensuring resilience and growth. Elastic's workforce, known as Elasticians, operates without a central office, promoting a flexible work environment where employees manage their own schedules and utilize tools like GitHub, Slack, and Zoom for communication. This model necessitates trust in employees' self-motivation and commitment, supported by the Elastic Source Code, which emphasizes freedom and adaptability. As a result, Elastic has expanded its reach significantly, with employees in over 35 countries and customers in more than 80, demonstrating that its distributed mindset is not just a business strategy but a core philosophy.

Testing applications utilizing databases poses a challenge for backend developers, especially when aiming for unit tests rather than integration tests. While traditional methods like using real databases or Docker can be cumbersome, mocking offers an efficient alternative, particularly for isolating API behavior. The Elasticsearch Node.js client, designed for flexibility, can be mocked effectively by replacing its Connection component. The @elastic/elasticsearch-mock library facilitates this process by allowing developers to create custom mocks for requests, enhancing test speed and parallelization without altering other client components. This library supports both strict and loose mocks, dynamic paths, wildcards, and even simulates random failures, enabling robust testing scenarios. The approach ensures that only the HTTP layer is mocked, maintaining the integrity of other client functionalities, thus providing an efficient and error-free testing environment.

Elastic offers a variety of free, on-demand training courses covering topics such as observability, security, and Elastic Stack administration, catering to both beginners and advanced users. These online courses provide an immersive learning experience similar to traditional classroom settings, with expertly designed materials, engaging demos, hands-on lab exercises, and access to Elastic experts. With more than 40 courses available, learners can build and retain new skills at their own pace from anywhere with an internet connection. Additionally, Elastic offers a Training Subscription for those seeking comprehensive access to all course offerings and exclusive benefits, making it convenient for busy professionals to enhance their Elastic skills remotely.

Elasticsearch provides a flexible and scalable solution for storing, searching, and analyzing large volumes of data, making it suitable for various applications such as system observability and enterprise search. Designing an effective data storage architecture for Elasticsearch involves considering factors like data loss tolerance, performance, and downtime. The blog discusses several storage options, including different RAID configurations (RAID 0, RAID 1, RAID 5, RAID 6) and multiple data paths in Elasticsearch, each with its pros and cons in terms of data protection, performance, and capacity. RAID 0 offers high performance but no fault tolerance, RAID 1 provides data redundancy at the cost of capacity, while RAID 5 and 6 offer a balance between performance and data protection with the use of parity. Multiple data paths in Elasticsearch allow for flexible data storage but require careful management to avoid performance bottlenecks. For those who prefer not to manage these complexities, Elasticsearch Service on Elastic Cloud offers a managed solution that handles scalability and storage architecture, allowing users to focus on their data applications without worrying about infrastructure.

Kibana, part of the Elastic Stack, offers powerful tools for time series analysis, notably through the Time Series Visual Builder (TSVB), which is used to visualize observability data such as CPU or memory utilization and network transfer. The blog post by Chris Cowan explains the process of visualizing various rates using TSVB, focusing on positive rates, rate of change, and event rates. It details how to convert counters to positive rates by evenly distributing data across time intervals and handling resets with positive-only aggregation to avoid negative dips. For the rate of change, it emphasizes calculating the percentage change for gauges or counters to understand rising or falling trends, using derivatives and bucket scripts. Event rates are discussed in terms of the frequency of events within set time intervals, with steps on converting counts to per-second rates. The post also touches on the importance of understanding the shape of metrics, whether they are gauges, counters, or document counts, to select the appropriate visualization approach. Practical examples and visualizations are provided, which can be replicated using Kibana's TSVB, with additional resources available for those interested in trying these techniques on their data.

Elastic Security, previously known as Elastic SIEM, is a comprehensive solution that includes SIEM, endpoint security, threat hunting, and cloud monitoring. The platform leverages the MITRE ATT&CK framework to enhance security teams' understanding of attacker tactics and improve their threat detection capabilities. Elastic Security 7.6 introduced 92 detection rules aligned with ATT&CK, allowing users to generate "signals" based on risk and severity scores for effective triage. The system operates by running detection rules every five minutes, which output a signal when specific criteria are met. For effective threat analysis, users can configure and utilize Sysmon data, a Windows-specific application that logs various system activities, to trigger these rules. The document emphasizes the importance of live data testing through Sysmon to ensure the accuracy and reliability of threat detection, while cautioning against running tests on production systems without proper authorization. Proper setup involves using Sysmon in conjunction with Elasticsearch and Winlogbeat to ship logs in an ECS-compatible format, ensuring that Elastic SIEM can process and analyze the data efficiently.

Elastic Cloud Enterprise (ECE) 2.5 introduces several enhancements aimed at improving deployment management and scalability, including the addition of a dedicated coordinating layer for better performance and support for snapshot lifecycle management (SLM). This release allows users to configure independently scalable coordinating nodes to handle ingest pipelines and search queries, thereby optimizing resource utilization and maintaining data accessibility even in case of coordination layer issues. Additionally, ECE 2.5 integrates SLM for automated snapshot management and a more robust restore process, transitioning from the previous snapshot mechanism. The update also facilitates migrating from index curation to index lifecycle management (ILM) through a new wizard, simplifying the process and encouraging users to adopt ILM due to the deprecation of index curation. Additionally, ECE supports custom Kibana plugins, enabling users to extend functionalities with their own or existing plugins, with detailed guidance provided for implementation. The release also includes various improvements and bug fixes, with a free 30-day trial available for those not using ECE.

Elasticsearch Service has expanded to the Google Cloud Platform (GCP) South Carolina region, marking its 13th GCP region globally and 5th in North America. This expansion allows existing users to immediately access the new region, while new users can explore it through a 14-day free trial. The service offers advanced features like machine learning, index lifecycle management, and intuitive data visualization tools such as Kibana Lens and Canvas. Elastic, in partnership with Google, is committed to further expanding GCP regions and enhancing native GCP console integration. Customers can subscribe through the GCP Marketplace, benefiting from integrated billing and the potential to offset charges against prepaid GCP agreements. For those needing to move existing deployments, detailed migration documentation and webinars are available to assist users.

Elasticsearch 7.6 introduces classification-based supervised learning, enabling users to predict categories or preferences in datasets, such as movie genres or personal movie preferences. The blog post describes using this feature to build a model that predicts whether a user will like a movie based on its characteristics, using data from the Korean Film Council. The process involves creating a deployment in Elasticsearch, ingesting movie data, and using Logstash for configuration. Users can create a classification model in Kibana to analyze and predict movie preferences, with results accessed through Elasticsearch's inference capabilities. The author shares a personal experience of using the model to choose movies during the COVID-19 lockdown and encourages others to explore classification for various applications, such as predicting loan risks or detecting defects. Configuration and data files used in the demonstration are available in the author's GitHub repository.

Ingram Micro, a Fortune 100 company known for delivering technology and supply chain services globally, chose Elasticsearch Service to enhance the search and sales capabilities of its extensive ecommerce platform. With a catalog of over 20 million items and 70% of its visitors using the search bar directly from the homepage, Ingram Micro needed a robust search solution to accommodate complex indexing and variable factors associated with its global operations. By transitioning from Endeca to Elastic, the company significantly improved its ecommerce performance, with a 50% increase in conversion rates, 35% improvement in search relevance, and a 30% reduction in null search results. The implementation of Elastic's services, including Elastic APM, Heartbeat, and machine learning features, enabled real-time monitoring, anomaly detection, and performance visualization, contributing to a unified and scalable search experience that Ingram Micro expanded from Canada to the U.S., Europe, and is planning further implementation in Asia Pacific and Latin America.

The Elastic Stack, consisting of Elasticsearch, Kibana, Beats, and Logstash, has returned to its open-source roots, emphasizing its principles of being free and open to foster a collaborative and innovative community. This approach has cultivated a vibrant ecosystem where users worldwide can contribute to and benefit from the technology, whether for developing search-powered applications or enhancing security analytics. The community's creativity has sparked the development of tools like Kibana Lens, Elastic Maps, and Canvas, which enhance data visualization and storytelling. Elastic has also introduced solutions like Elastic Observability and Elastic Security, integrating free and open components to address key user challenges. The Elastic Stack's open development model encourages continuous improvement and innovation, inviting users to participate through forums, GitHub repositories, and various engagement platforms.

Elasticsearch machine learning allows for the development of regression and classification models that can analyze complex data, and the introduction of feature importance in Elastic Stack 7.6 enhances the interpretability of these models. Through the use of data frame analytics, which employs decision trees to predict outcomes based on historical data, feature importance provides a locally-accurate linear model representation that helps users understand and verify predictions. In a practical example using the World Happiness Report, feature importance was applied to explore factors influencing happiness across different countries, indicating that healthy life expectancy and GDP per capita are significant predictors of happiness, with the former being more influential overall. The methodology uses the SHAP (SHapley Additive exPlanations) algorithm to assign values to features, thus enabling the interpretation of model predictions and the discovery of data relationships. This approach reveals that while wealth is a factor in happiness, other aspects like social connections and health play crucial roles, particularly in countries at the extremes of the wealth spectrum. Feature importance, as part of the Elastic Stack, offers a valuable tool for gaining insights into data and understanding model predictions, fostering a deeper comprehension of the factors that contribute to outcomes such as happiness.

Elastic App Search is a free product designed to simplify the development of advanced search experiences using Elasticsearch, making it accessible not only to developers but also to non-technical users like website administrators and marketing teams. It offers out-of-the-box features such as typo tolerance, multilingual relevance, and an intuitive administrative dashboard, allowing users to customize search algorithms visually without needing expert knowledge. Elastic App Search aims to democratize powerful search capabilities by removing cost barriers, thus encouraging innovation and broad adoption. As an open-source solution, it can be run on personal infrastructure or deployed on Elastic Cloud, where users only pay for the resources they consume. This approach aligns with Elastic's philosophy of providing scalable, user-centric software that adapts to diverse business needs, further supported by free developer tools like the Search UI JavaScript library and native API Clients.