December 2019 Summaries

In 2019, Apache Lucene, a long-standing and dynamic open-source project, saw significant developments, including the induction of new committers and Project Management Committee members, and the release of nine new versions, notably version 8.0. The migration of Lucene’s repository to Git in 2016 facilitated an increase in unique contributors, as it simplified the process of contributing by allowing pull requests via GitHub. Key enhancements included the introduction of Block-Max WAND, which significantly accelerated certain query processes, and the integration of static scoring signals through the new FeatureField. Additionally, the Luke tool was incorporated as a Lucene module, ensuring timely updates with each new Lucene release, and a new "monitor" module was added, thanks to the donation of Luwak by FlaxSearch. The year also saw major improvements in indexing speed for multi-dimensional points, due to optimization inspired by radix sort, and various efficiency enhancements, such as improved FST lookups and memory usage reductions for BKD trees and FSTs.

The 2019 Elastic Advent Calendar concluded with a series of 25 articles exploring a wide range of topics related to the Elastic Stack, including Elasticsearch, Python, Auditbeat, ECS, JVM options, anomaly detection models, and SSL configuration. The series featured contributions from various authors in multiple languages, such as Finnish, Spanish, German, and French, and covered practical applications like monitoring Linux commands with Auditbeat, using Elasticsearch for email activity processing, and simplifying ingest pipelines with the new enrich processor. The articles provided insights into the latest features and improvements in Elasticsearch and Kibana, offering readers a comprehensive understanding of data handling and analysis tools available in the Elastic ecosystem. The series encouraged interaction and ongoing discussion on the Elastic Discuss Forums, inviting feedback and suggestions for future content while expressing optimism for continued innovation and community engagement in 2020.

Elasticsearch Service on Elastic Cloud has expanded its availability to the Google Cloud Platform (GCP) region in Montréal, marking its ninth GCP region globally and fourth in the Americas. This expansion allows existing users to access the new region immediately, while new users can explore a free 14-day trial to experience features like index lifecycle management and machine learning. The partnership between Elastic and Google enhances the service by integrating built-in technologies for various use cases such as logging, metrics, APM, and BI/analytics, and facilitates procurement through the GCP Marketplace, with charges consolidated into the user's GCP bill. The collaboration aims to continue expanding GCP region availability and integrate native GCP console features, supporting both Elastic Cloud Enterprise and Elastic Cloud Kubernetes.

The third week of the 2019 Elastic Advent Calendar offers insights into various Elasticsearch-related topics, including setting up Snapshot Lifecycle Management (SLM) with Minio.io to manage snapshots, leveraging Github events for enhanced data visualization in Kibana, and using the Elastic Stack for monitoring home networks. Additionally, it covers utilizing the new Enrich Processor to simplify data ingestion by enabling lookups on other indices, building efficient search solutions with App Search, and analyzing large datasets using choropleth maps and Elastic Maps, illustrated by the "Get It Done" initiative in San Diego. This series, aimed at sharing practical tips and insights from Elastic engineers, encourages engagement and learning through a variety of innovative applications and tools.

The blog post by Wylie Conlon discusses methods for displaying data as percentages in Kibana visualizations, which is crucial for effective numeric comparisons, especially when dealing with varying sample sizes. The post provides examples using Kibana's sample flights and ecommerce datasets to demonstrate how percentages can be utilized in different visualizations such as pie charts, single number metrics, tables, and time series with the Time Series Visual Builder (TSVB). For instance, it explains how to calculate the percentage of on-time flights using the Filter Ratio in TSVB and track changes in sales week over week using the Serial Difference aggregation. The article emphasizes configuring the appropriate data sets and visualization types to derive meaningful insights, suggesting that users can further explore Kibana's Canvas for more advanced percentage calculations and dashboard creation.

Mark Mager discusses the development of a ransomware testing framework called DCART, which is designed to enhance the detection and mitigation of ransomware using Elastic Endpoint Security. The framework focuses on decoupling the components responsible for collecting and analyzing event data, allowing for scalable and efficient testing of ransomware detection capabilities. By introducing a minifilter driver paired with a user space process, DCART efficiently logs file system events, enabling continuous analysis through a Python-based script that evaluates these events for anomalous activity using entropy and header analysis metrics. Although the framework effectively identifies suspicious file behaviors, it acknowledges the potential for false positives and emphasizes the importance of comprehensive testing across multiple file types. DCART is still a proof of concept requiring further development for production readiness, but its implementation and code are available on the Elastic GitHub repository, providing a foundation for future advancements in behavioral ransomware detection.

Prometheus has become a popular tool for monitoring and alerting in container systems, primarily due to its open-source nature and community support, but it faces challenges in large-scale deployments and cross-team collaboration, particularly with long-term data retention and high cardinality dimensions. While Prometheus excels in efficient server-side metric storage, its local storage model limits scalability and availability, necessitating complementary long-term storage solutions. Elastic Stack offers a way to enhance Prometheus's capabilities by providing centralized, scalable storage and allowing integration of various operational data types, such as logs and traces, for improved observability. With features like data rollups and advanced security measures, Elasticsearch enables long-term retention and secure access to operational data, overcoming some limitations of Prometheus's approach. Elastic Stack's capability to manage high cardinality dimensions without compromising storage efficiency and its support for streaming metrics via Metricbeat provide a more holistic approach to system observability. By integrating these tools, organizations can break down operational silos, gaining a comprehensive and secure view of their data across distributed environments.

The Elasticsearch Service is now generally available on Microsoft Azure following a successful public beta, enhancing the collaboration between Elastic and Microsoft. This development allows existing users to deploy services on Azure via their current accounts, while new users can access a 14-day free trial. The service offers robust features, such as one-click, zero downtime upgrades, and the latest software releases and security patches. Additionally, Elastic continues to support Microsoft technologies through various integrations, including Azure Active Directory and .NET support. The partnership aims to ensure seamless technology integration, fostering innovation opportunities for users. The service's expansion includes new Azure regions in Washington and Singapore, with plans for further global support. Customer testimonials, such as from Bell and Howell's Dr. Haroon Abbu, highlight the service's capability to enhance data management and predictive maintenance. A promotional period offering free data transfer and storage is currently available to encourage new deployments on Azure.

The Elastic Advent Calendar 2019 offers a diverse range of content related to the Elastic Stack, with the second week covering topics such as logging in Elasticsearch and Elastic Cloud, smart query cancellation in Kibana, SSL/TLS configuration, improvements in Kibana Maps, data transforms, diagnosing sluggish web apps, and getting started with Elastic Machine Learning. Each topic presents practical insights and updates, such as the flexibility of logging in Elasticsearch, the introduction of data transforms to create summary indexes, and enhancements in anomaly detection through machine learning. The series is designed to engage users with a variety of expertise levels, offering new features and best practices while inviting feedback and encouraging further exploration of the Elastic Stack's capabilities.

Elasticsearch has significantly enhanced its geo_shape field indexing technology by transitioning from the legacy prefix tree indexing to the new block k-d tree (BKD) technique, delivering improved performance and accuracy. This change, introduced in Elasticsearch 7.0, guarantees 1 cm accuracy and accelerates searching and indexing processes, addressing previous user challenges with spatial accuracy and performance configurations. Historically, spatial indexing relied on rasterization and quadtrees, which increased index sizes and complexity. The new approach maintains geometries in vector space, using triangular tessellation, which reduces the number of terms and preserves the original accuracy of geometries. This involves encoding triangles into a BKD data structure optimized for spatial searches. The BKD approach, utilizing selective indexing and a compact dimensional encoding for triangles, results in a more efficient index that is less than half the size of traditional methods, albeit with a minor spatial error of 1 cm. Future improvements and optimizations in BKD structures are planned, with ongoing benchmarks and community feedback encouraged to further refine the system.

Adobe utilizes Elasticsearch and its AI platform, Adobe Sensei, to power advanced image recognition and search capabilities across its products, particularly in Adobe Stock. By managing their own Elastic Stack deployment, Adobe hosts over 10 billion documents and supports a high ingestion rate, enhancing functionalities like face detection, object detection, and auto-tagging in real-time. Adobe has developed custom Elasticsearch plugins to improve image recognition and search ranking, enabling users to search for similar images based on attributes such as color and composition. The integration of convolutional neural networks allows for the creation of deep learned representations, enhancing the user experience in discovering and sorting through Adobe Stock's vast digital content library.

Elastic Stack 7.5 introduces new modules in Metricbeat and Filebeat that enhance support for Azure services, allowing users to efficiently collect and analyze metrics and logs from Azure Monitor and Azure activity logs. The Azure module in Metricbeat enables the retrieval of metrics from various Azure resources, with customization options such as resource filters and metric aggregations, while the Azure module in Filebeat facilitates the collection of Azure activity and AD activity logs through event hubs. Users can configure these modules with authentication credentials obtained from Azure AD and leverage built-in dashboards and visualizations to monitor key metrics and activities. These enhancements aim to streamline the integration and monitoring of Azure services within the Elastic Stack, with plans for further service-specific integrations in the future.

Elasticsearch 7.5.0 introduced a new enrich processor for ingest nodes, allowing users to enrich documents with additional data from reference datasets during the ingestion process. Ingest nodes pre-process documents before indexing by using pipelines composed of various processors, each performing specific operations like splitting or removing fields. The enrich processor uses data from another index to enhance documents, such as adding a user's full name based on an email address match. For example, flight data can be enriched with airport data by creating an enrich policy that manages reference data efficiently, like indexing US airport codes to enrich flight information with airport-specific details. This policy creates a system index for quick document enrichment, and once executed, a pipeline can be defined to enrich and clean up documents by removing redundant fields and renaming others. The enriched data is then incorporated into the documents, replacing fields with enriched JSON objects containing additional information. This feature allows users to enhance their data workflows and is accessible by upgrading to Elasticsearch 7.5 or using Elasticsearch Service.

In the December 2019 update for Kibana, several notable changes and improvements were highlighted across different components. Kibana's security was enhanced by allowing it to be instrumented with APM, and deprecated Node.js APIs were cleaned up to prevent future errors. The platform introduced new APIs, including "uiCapabilities" for plugins, and validated configuration before running migrations. Enhancements to Stack Services Alerting enabled KQL nested queries and allowed filtering of alerts by action type, while some email actions were whitelisted. The Kibana Platform migration efforts included various tasks like applying filter actions and state management improvements, alongside bug fixes. In the Kibana App section, Elastic Charts saw progress with pie charts, series identifiers were refactored, and Dependabot was configured for automatic dependency updates. Elastic Maps improved symbolization consistency by using extended statistical metadata, providing users with more reliable visual representations while allowing configuration for style adjustments. Additionally, Canvas updates were noted, EuiBasicTable was converted to TypeScript, and DataGrid enhancements were made. The operations updates ensured CI changes were aggregated into a single comment to reduce PR noise.

Elastic's Elasticsearch Service (ESS) offers a cost-effective solution for handling data such as logs, metrics, and Application Performance Monitoring (APM) by minimizing hidden network costs associated with data transfer. Unlike competitors like New Relic and DataDog, ESS provides extensive cloud provider and regional presence across 22 regions and three major cloud providers (Azure, GCP, and AWS), reducing data transfer costs by leveraging data locality. This is crucial because outbound data transfer across cloud regions incurs significant charges, especially for services that lack local endpoints and rely on US or EU regions, leading to higher fees. The document exemplifies how ESS's unique resource-based pricing model and regional availability can result in substantial cost savings compared to other SaaS solutions. By using scenarios involving different volumes of log data, metrics from various hosts, and APM data, the text demonstrates the potential financial advantages of choosing ESS over other vendors, emphasizing the importance of regional data processing to avoid punitive data-out charges.

The blog post is part of a series that guides users through setting up Elastic Security (formerly Elastic SIEM) for home and small business environments using Elastic Stack 7.4 and later versions, specifically focusing on data collection from Windows systems using Beats. It highlights the process of installing and configuring Winlogbeat, Packetbeat, and Auditbeat on a Windows 10 computer to collect log files, network data, and system activities. The article provides detailed instructions on software setup, configuration file adjustments, and the use of PowerShell commands to manage and verify the services, emphasizing the importance of selecting appropriate log sources and configuring the GeoIP ingest pipeline for enriching data. The guide also advises on potential issues, such as script execution policies and GeoIP processor limitations, and suggests using specific PowerShell commands to determine and configure event log sources. Additionally, it discusses the use of Sysinternals System Monitor (Sysmon) and Npcap for enhanced data collection and includes tips for logging configuration and troubleshooting common errors encountered during setup. The series aims to empower users to develop a robust security solution by leveraging Elastic's tools for data visibility and analysis.

Elastic's Advent Calendar 2019 presents a diverse array of topics related to the Elastic Stack, shared by their engineering team through daily posts in multiple languages over the first 25 days of December. This initiative, inspired by previous years' success, covers subjects ranging from Elasticsearch's integration with Python for data scientists, to monitoring shell commands in Linux using Auditbeat, processing Mailgun activity with Elasticsearch, and exploring the Elastic Common Schema (ECS) with Kibana. Additionally, the series delves into configuring JVM options in Elasticsearch and the intricacies of Elastic's unsupervised machine learning anomaly detection models. Each post aims to engage the community with insightful technical content and invites feedback to enhance future offerings.

In the blog post, Darren LaCasse discusses how the Elastic Stack, specifically using Elasticsearch and Watcher, can be used to enhance information security alerts by enriching them with additional data, such as MITRE ATT&CK information, and storing them in a separate index for improved reporting and analysis. The process involves transforming alert payloads using a Watcher payload transform to inject new fields, which are then indexed into Elasticsearch, allowing for detailed reporting and visualization through a Canvas dashboard. This approach not only facilitates more meaningful reporting by breaking down alerts by MITRE ATT&CK Techniques and other parameters but also aids analysts by linking key fields to relevant resources, such as MITRE ATT&CK Technique pages and internal triage playbooks, thus speeding up investigation processes. The enriched alert data and its visualization provide deeper insights into potential security threats, enabling a more effective response and enhancing overall security detection capabilities.

In the blog post by Mike Barretta, the process of automating the installation of Elastic Cloud Enterprise (ECE) on AWS using Ansible is detailed. Barretta, a newcomer to Ansible, outlines the steps for setting up a small ECE environment, suitable for a proof of concept, testing, or a small production cluster. The process involves creating a security group, launching three EC2 instances across different availability zones, and installing Ansible on the user's machine. The post guides readers through downloading the ECE Ansible role, setting up an Ansible project, creating an inventory, and writing a playbook to deploy and configure ECE. Barretta emphasizes the ease of using Ansible for this purpose, despite his initial unfamiliarity, and concludes by celebrating the successful setup of a three-node, three-zone ECE environment. Readers are encouraged to explore their new setup and potentially deploy Elasticsearch clusters using ECE.

UserCentric, an Australian digital consultancy, developed a tailored recruitment solution for the Postgraduate Medical Council of Victoria (PMCV) using Elasticsearch Service on Elastic Cloud, significantly enhancing the efficiency of matching doctors and nurses to job vacancies. This system allows PMCV administrators to manage the recruitment process through a single dashboard, enabling real-time updates and configurations specific to each job role, thereby streamlining the placement of healthcare professionals across 142 hospitals. By incorporating Kentico Kontent for content management and Kibana for business intelligence reporting, the solution facilitates sophisticated data insights and rapid report generation, reducing the time required to produce reports from a week to real-time. This has improved the recruitment processing time by 50%, ensuring that Victoria's healthcare system remains fully staffed, while freeing up technical resources to focus on enhancing user experience and addressing core engineering challenges.

Elastic's Sales Development Representatives (SDRs) play a pivotal role in fostering business growth by engaging with users and potential customers to integrate the Elastic Stack and related products into their operations. The team is notable for its diversity, with members spread across 12 cities worldwide and speaking over 20 languages, which enhances their ability to connect with a wide range of clients. Unlike many companies that adopt a uniform approach to hiring, Elastic values the varied cultures, languages, and experiences of its SDRs, believing this diversity strengthens the team. John Black, Vice President of Sales Development, and Ira Casteel, SDR Manager, emphasize the unique career paths available to SDRs at Elastic, noting that many have advanced into roles in sales, marketing, solutions architecture, and more, underscoring the company's commitment to career development. SDRs at Elastic are described as investigators, connectors, and educators who adeptly handle large amounts of information to provide timely and relevant customer support, positioning them as future leaders in the tech industry.

Kibana has introduced several new features and fixes, enhancing its platform and user experience. A significant update includes the implementation of an absolute session timeout, providing better session management compared to the previous idle timeout feature. Additionally, issues like infinite redirects when using base-paths have been resolved, and new APIs have been introduced to customize HTTP configuration options and enhance the SavedObjectsClient. The platform has also seen improvements in telemetry and alerting, with migrations to the new Kibana platform and enhancements in alert filtering and UI. The app architecture has progressed with the migration of key components like the FilterBar, Search Source, and Index Patterns to the Kibana Platform. Visualization and dashboard functionalities have been improved, including debugging of TSVB saving regressions and the release of Elastic-Charts as PNG. The efforts towards the Kibana Platform migration continue, focusing on enhancing the overall infrastructure and user interface.

Elastic has announced the general availability of external collection for Elastic Stack Monitoring via Metricbeat, which enhances the reliability and flexibility of monitoring Elasticsearch, Kibana, Logstash, APM server, and Beats. This new approach allows monitoring data to be collected and sent directly to the monitoring cluster through Metricbeat, eliminating the need to route it through the production cluster, thus reducing the workload on the production system and increasing the resilience of the monitoring setup. Previously, internal collection required services to gather and ship their own data, often through the production cluster, which was not ideal during times of duress. The introduction of external collection simplifies the process of monitoring Elastic Stack services, and Elastic provides a Migration Wizard to assist current users in transitioning from internal to external collection. Future versions of Elastic Stack will phase out internal collection entirely, and users are encouraged to switch to Metricbeat to ensure preparedness for this transition and to benefit from the improved monitoring capabilities.

The Beats 7.5.0 release introduces significant enhancements geared towards improving observability by offering turnkey data integrations and monitoring solutions for critical infrastructure metrics. This release includes expanded support for Microsoft Azure, enabling users to seamlessly ingest metrics and logs from Azure services through new Metricbeat and Filebeat modules, complemented by prebuilt Kibana dashboards for swift analysis. Additionally, Kubernetes monitoring is enhanced with improved Heartbeat capabilities, featuring hint-based auto-discovery to better manage the dynamic nature of its services. These updates build upon existing integrations with platforms like Kubernetes, Prometheus, and AWS, further solidifying Beats as a vital tool for organizations seeking comprehensive observability solutions. Users are encouraged to explore the new features, report feedback, and engage with the Elastic community for any support or inquiries.

Elastic Security 7.5.0, released in December 2019, integrates Elastic Endpoint Security and Elastic SIEM to provide comprehensive threat detection and response capabilities. This update follows Elastic's acquisition of Endgame, enhancing their security offerings by combining threat hunting and analytics with prevention and response features. Elastic Endpoint Security, now part of the standard Enterprise subscription, facilitates faster incident response by reducing the mean time to remediate from seven days to 30 minutes, as experienced by Texas A&M University. The 7.5 update introduces new machine learning jobs for identifying anomalous activities, with SIEM app enhancements such as improved UI widgets and pre-built ML jobs to support security analysts in threat hunting. Elastic Endpoint Security further streamlines root-cause analysis and incident management through integration with the Elastic Stack, offering features like automated attack visualization and real-time endpoint isolation. The release allows users to visualize endpoint event data using Kibana dashboards and adds functionality for dismissing alerts with specific reasons, enhancing collaboration and workflow efficiency among security teams.

Elastic Observability 7.5 introduces significant enhancements to its monitoring and analytics capabilities, focusing on seamless integration between APM, logging, and security data to support comprehensive observability initiatives. The release features the Metrics Explorer for real-time analytics, streamlined data integrations for key infrastructure metrics like Kubernetes, Prometheus, AWS, and new support for Microsoft Azure metrics and logs. It also introduces improved navigation between APM traces and logs using unique identifiers, enhancing analysts' ability to diagnose incidents efficiently. The update further offers a curated user interface for log rate anomaly detection, facilitating the identification of significant trends and events. Additionally, it enhances Kubernetes service monitoring with flexible configurations and introduces Kibana Lens, a user-friendly tool for visualizing observability data through drag-and-drop functionality. The update aims to unify disparate datasets into a cohesive system, increasing productivity for analysts by providing a single view for logging, metrics, and tracing data.

Elastic Logs 7.5.0 introduces significant enhancements, including log rate anomaly detection and expanded support for AWS and Azure log events, available on the Elasticsearch Service or as part of the Elastic Stack. This release features a curated UI for dataset-based log rate anomaly detection, aiding operators in identifying important trends with a single click, though this requires a Platinum license. The update supports ingesting AWS Elastic Load Balancer logs from S3 and processes key metrics such as request processing time and TLS handshake time, covering Classic, Application, and Network Load Balancers. Additionally, a new module for Azure Event Hub logs is introduced, which handles logs related to Activity, Active Directory Sign-in, and Audit Logs, though it is currently in beta and not recommended for production use. Users can access Elastic Logs 7.5.0 by creating a new cluster or upgrading an existing one on the Elasticsearch Service on Elastic Cloud, or by downloading it as part of the default Elastic Stack distribution.

Elastic Stack 7.5.0 introduces several significant features and enhancements, including the debut of Kibana Lens, which revolutionizes data visualization with its intuitive drag-and-drop interface and smart suggestions, making it accessible to users without technical expertise. The release also enhances Elasticsearch with a new Enrich processor for efficient data enrichment at indexing time, allowing for precise data augmentation tasks like IP address identification and metadata addition. Elastic Enterprise Search now includes one-click integrations with Microsoft services and a new ServiceNow connector to streamline content unification. Observability tools are expanded with turnkey integrations for Azure metrics and logs, enhancing the real-time analytics capabilities of Elastic Metrics. Elastic Security introduces endpoint security data directly into the SIEM app and continues to leverage machine learning for threat detection, while removing per-endpoint pricing to offer unlimited protection with Enterprise subscriptions. These updates reflect a comprehensive effort to enhance data visualization, search integration, observability, and security across the Elastic Stack.

Elastic Enterprise Search 7.5, now renamed Elastic Workplace Search, introduces enhanced connectivity for enterprises by integrating with four new platforms, including Microsoft SharePoint Online, OneDrive, Office 365, and ServiceNow. This iteration focuses on simplifying the connection and content unification process for organizations using Microsoft and Google products, while also supporting IT and business operations through platforms like Salesforce and Zendesk. The updated version boasts significant improvements in speed, allowing document indexing up to 700% faster and delivering relevant search results in half the time compared to its predecessor, Beta 3. Elastic Enterprise Search continues to align its updates with the broader Elastic Stack releases, ensuring that enterprises can enjoy seamless integration and improved search capabilities across their entire organization. Users are encouraged to provide feedback to help refine the product during its beta phase.

Logstash 7.5.0 has been released, bringing significant improvements to plugin management by introducing "integration plugins" that consolidate input, output, and filter plugins for specific technologies into a single, manageable entity. This change enhances consistency and ease of use for users while streamlining the development and maintenance process for developers. The release debuts with integration plugins for Kafka and RabbitMQ, ensuring that existing configurations remain unaffected and continue to function smoothly. Users are encouraged to download the new version, provide feedback, and report any issues through various channels, including Twitter and GitHub, as the transition to integration plugins expands in future releases.

Elastic APM 7.5.0 introduces several enhancements to simplify application performance monitoring, including expanded navigation between APM traces and logs, allowing trace unique identifiers (IDs) to be written into log messages for all Elastic APM agents using the Elastic Common Schema. This update, part of Elastic Stack's default distribution and available on Elasticsearch Service, also marks the general availability of aggregate service breakdown charts, offering a comprehensive view of application performance hotspots. New configuration options enable users to adjust transaction capture body and the maximum number of spans from the UI without altering agent configuration files, accompanied by visual feedback indicators. Additionally, JVM instance-level visibility is improved with a tabular view displaying metric data, enabling detailed analysis of CPU, memory usage, thread count, and garbage collection for specific JVMs.

Elastic Uptime Monitoring 7.5.0, part of the Elastic Stack, introduces several enhancements including hint-based auto-discovery for Kubernetes services, search auto-complete support in the Kibana UI, and pagination for managing high volumes of monitor status results. The release simplifies Kubernetes monitoring by allowing users to attach metadata to Docker or Kubernetes containers, enabling Heartbeat to automatically configure monitoring based on these hints. Additional features include notifications for SSL certificate expiration and support for non-privileged ICMP checks, making the tool more accessible in environments with strict security policies. These updates are designed to improve the efficiency and effectiveness of monitoring dynamic infrastructures, and the latest version of Elastic Uptime can be accessed via the Elasticsearch Service or downloaded as part of the Elastic Stack distribution.

Elastic Maps 7.5.0 introduces several enhancements and new features designed to improve the user experience in mapping and data visualization. Key updates include the general availability of GeoJSON upload, enabling users to easily integrate custom vector shapes into their maps for detailed analysis. The release also allows styling of map symbols and icons by date and time fields, facilitating better representation of time series data such as asset tracking. Users can now prioritize important data by ordering Elasticsearch document layers based on sortable fields, ensuring critical data remains prominent. Additionally, the update enhances tooltip customization with a redesigned interface, offering more control over layout and display, thus enriching the overall mapping experience. Elastic Maps 7.5.0 is available for deployment via Elastic Cloud or through self-managed environments, with Elasticsearch and Kibana 7.5 ready for download.

Elastic Metrics 7.5.0, previously known as the Infrastructure app, has been released with significant enhancements, including a new Azure integration for collecting metrics and logs from Azure workloads, reflecting the app's expanded scope beyond infrastructure monitoring. The release introduces a range of new features, such as persistent views in Inventory and Metrics Explorer for more efficient resource and metrics browsing, as well as support for bar charts in Metrics Explorer to better visualize certain types of data. Additional improvements include AWS tags and metadata integration to enhance workload context in AWS environments, and expanded Kubernetes observability with resource quotas metrics and new ECS fields for improved correlation between Docker metrics and Kubernetes logs. The Azure module, a key addition, facilitates hybrid cloud observability by automatically collecting metrics from Azure accounts and workloads, complemented by preconfigured Kibana dashboards for streamlined monitoring. Users can access the latest version of Elastic Metrics either through the Elasticsearch Service on Elastic Cloud by upgrading or creating a new cluster, or by downloading it as part of the default Elastic Stack distribution.

Kibana Lens is a new feature introduced in the 7.5 release of the Elastic Stack, designed to simplify and enhance data visualization and exploration within the Elastic Stack. This tool enables users to effortlessly gain insights from Elasticsearch data through an intuitive drag-and-drop interface that provides immediate previews and smart suggestions for visualizing data. This development focuses on making data analysis accessible to a broader audience, including engineers, analysts, and executives, by removing the complexity of Elasticsearch terminology and allowing for easy switching between chart types and data sources. Lens is available for free as part of the default distribution, aiming to provide a powerful yet simple user experience by leveraging data visualization best practices and enabling users to perform real-time analyses on large data volumes. The release of Lens represents a significant milestone in Elastic's ongoing journey to deliver innovative data visualization solutions, with plans for future enhancements that include supporting more chart types, smarter suggestions, and increased customization options.

Elasticsearch 7.5.0, based on Lucene 8.3.0, introduces several enhancements in search and analytics, cluster management, machine learning, and more, now available on Elastic Cloud. Notable features include the new enrich processor, which simplifies ingest processes by enabling data enrichment from existing indices, and higher-resolution geotile analysis with composite aggregations, allowing more efficient processing of geographical data. This release also enhances SQL functionality with shape support, improves snapshot lifecycle management with retention capabilities, and strengthens security through an API keys app and the create_doc index privilege. Cross-Cluster Replication (CCR) is enhanced with pause and resume API endpoints, facilitating easier upgrades in bi-directional replication architectures. Additionally, the release includes a Machine Learning Classification API for binary classification, with several features available under Elastic’s free Basic subscription tier, while others, like CCR enhancements, require a Platinum subscription.