May 2020 Summaries

Shay Banon, addressing the Elastic company, emphasizes the importance of solidarity and action in response to the tragic events of racial injustice in the U.S., highlighted by incidents involving George Floyd, Ahmaud Arbery, and Breonna Taylor. He acknowledges the disproportionate impact of the pandemic on the Black community and calls for the same level of urgency in addressing systemic racism. Banon cites inspiration from Martin Luther King and encourages Elasticians to engage in anti-racism work through donations, collective actions, and personal education. He offers resources for support and emphasizes the importance of taking time for self-care, particularly for Black employees. Banon designates a company-wide Black Lives Matter day, inviting team members to show solidarity and engage in meaningful conversations and actions against racism.

Elasticsearch Service is now accessible on Amazon Web Services (AWS) in the Ohio region, allowing existing and new users to leverage its capabilities for search, observability, and security solutions. Users can try the service with a free 14-day trial and benefit from features like machine learning and index lifecycle management, while intuitive tools like Kibana Lens and Canvas facilitate creative visualizations. The service also offers use case-ready deployment templates for easy provisioning and scaling, and AWS customers can subscribe via the AWS Marketplace to streamline billing and integrate with existing AWS agreements. Additionally, the Private subscription tier provides secure connections for hosted deployments, and comprehensive documentation and webinars are available to assist users in migrating existing Elasticsearch data.

Elasticsearch Service is now accessible on Microsoft Azure in the Ireland region, allowing users to leverage Elastic Cloud's capabilities for search, observability, and security solutions. This expansion offers both existing and new users the ability to utilize features such as machine learning, index lifecycle management, and intuitive visualization tools like Kibana Lens. The partnership between Elastic and Microsoft aims to enhance the Elastic Stack experience by continually adding new Azure regions for the service. Azure customers will soon have the convenience of subscribing to Elasticsearch Service through the Azure Marketplace, with comprehensive support and resources available for migrating existing deployments.

Elasticsearch offers a flexible solution for enhancing existing SQL applications by adding powerful search capabilities without requiring significant architectural changes. This adaptability is particularly appealing to system administrators managing legacy systems, where replacing infrastructure can be costly and time-consuming. Through a five-part series, David Pilato demonstrates how to integrate Elasticsearch into a Spring Boot/MySQL application, showcasing its ability to perform CRUD operations, handle search and analytics, and use tools like Kibana for reporting. The series includes practical examples that can be tried at home using resources available in a GitHub repository, with options to deploy either on Elastic Cloud or locally. The content, initially presented at virtual events such as Codemotion's Dev Lunch Box, serves as both an educational guide and a leisure activity during social distancing periods.

Jamie Smith's blog post, the first in a series, explores how to effectively monitor system metrics on personal computers using Elastic Observability. It emphasizes the importance of metrics as a part of observability, providing insights into system usage for optimization and growth planning. The post explains that system metrics, which are numeric summaries of resources used by hosts and virtual machines, offer a time-series view of system states. Smith discusses various tools for monitoring these metrics, such as Task Manager, Activity Monitor, and command-line programs like vmstat and iostat, highlighting their limitations in long-term data gathering. By collecting and correlating metrics such as CPU, memory, disk, and network activity, users can diagnose system inefficiencies and plan upgrades. The post also points out the benefits of centralizing metrics from multiple hosts to detect trends and automate alerts through machine learning. In subsequent posts, Smith promises to guide readers on centralizing infrastructure metrics using the Elastic Stack to enhance scalability and monitoring precision.

The blog post by Alex Marquardt discusses strategies for enhancing search relevance in Elasticsearch through the use of boolean queries, match queries, and match phrase queries. It explains how Elasticsearch ranks search results using a scoring algorithm that considers factors like term frequency, inverse document frequency, and field length. The post provides examples of how different query operators, such as OR and AND, can affect the relevance and ranking of search results, demonstrating the flexibility of Elasticsearch in tuning search outputs to match specific use cases. By combining these queries with a boolean query's should clause, users can balance between exact matches and broader results, with the option to boost specific clauses to prioritize certain results. The article also introduces search templates for managing complex queries and suggests additional resources for further relevance tuning.

Elasticsearch has expanded its partnership with Google Cloud, introducing new purchasing options and easier access for its Elasticsearch Service through the Google Cloud Marketplace. Users can now opt for monthly or annual Gold and Platinum subscriptions, which offer advanced features such as machine learning and alerting, along with enhanced support service-level agreements. These subscriptions can be integrated with Google's consolidated billing system, allowing users to leverage their committed Google Cloud spend. The service is conveniently accessible via the Google Cloud console under "Partner Solutions," and additional support has been added for new Google Cloud regions, including Taiwan, Finland, and the Netherlands, enhancing resource distribution and availability.

Elastic has announced the general availability of the Elasticsearch Service API, which allows users to automate the management and scaling of their deployments on Elastic Cloud. The API integrates seamlessly with CI/CD pipelines and various infrastructure-as-code tools like Ansible, Chef, Puppet, and Terraform, facilitating the separation of use cases and minimizing resource contention between teams. Users can manage deployment tasks programmatically, such as creating, scaling, and shutting down deployments, without needing to access the Elastic Cloud console. The API supports predictable and usage-based scaling workflows, ensuring efficient resource allocation during varying demand periods. Additionally, Elastic provides a command-line interface, ecctl, to simplify interactions with the API, allowing for a more user-friendly approach to deployment management.

Logstash 7.7.0 has been released, introducing several enhancements and features aimed at improving performance and usability. One of the key updates is the reintroduction of proper event ordering using the Java execution engine for pipelines running with a single worker, offering more control through the new pipeline.ordered setting. The release also focuses on enhancing logging capabilities, including expanded log messages that provide detailed information about plugins and pipelines, aiding in easier troubleshooting. Additionally, the update includes improvements to AWS S3 integrations, such as the ability to ingest files from AWS Glacier and more flexible gzip file determination, along with new retry configurations to prevent infinite upload loops in error scenarios. Furthermore, Logstash 7.7.0 expands its platform support to include Windows Server 2019, reflecting its broad compatibility with various operating systems. Users are encouraged to download the release and provide feedback through various channels, including social media and community forums.

The release of Elastic Uptime Monitoring 7.7.0 introduces several new features aimed at enhancing user experience and efficiency. Key updates include proactive monitor status alerting, which notifies users if their monitors go down and integrates with platforms like Slack and PagerDuty. The release also features response duration anomaly detection powered by the Elastic machine learning engine, designed to highlight potential issues without requiring constant user monitoring. Additionally, support for custom index patterns and cross-cluster search has been added, allowing for more flexible data management. The Monitors Overview page has been improved for better readability, and users can now adjust the number of monitors displayed by default. This version is available through the Elasticsearch Service on Elastic Cloud or by downloading the Elastic Stack.

Elastic Workplace Search, announced on May 13, 2020, is a unified search platform designed to streamline information retrieval across various applications and tools used by organizations. As part of the Elastic Enterprise Search solution, it centralizes access to knowledge scattered across platforms like Microsoft 365, Google G Suite, Salesforce, and more, promoting efficient collaboration in increasingly distributed work environments. The platform emphasizes secure, personalized search results with features like typo-tolerant search, natural language querying, and automatic filtering, all powered by Elasticsearch. While initially released at the Platinum license level, a free version is planned to accommodate organizations of all sizes transitioning to remote work. Elastic Workplace Search integrates seamlessly with existing systems, offering APIs for custom content integration and ensuring fast, relevant search experiences as implementations scale, all without complex third-party integration challenges.

Elastic APM 7.7.0, released on May 13, 2020, introduces several significant enhancements, including service maps, inferred spans with an async profiler, alerting, agent central configuration, and custom links. Service maps provide a high-level, real-time view of Elastic-instrumented services, facilitating impact analysis and troubleshooting by visualizing service connections and dependencies. The inferred spans feature reduces upfront instrumentation efforts by providing granular method-level insights using a low-overhead async profiler, even in production environments. The release also integrates Kibana alerting for easy configuration of threshold alerts for service metrics and introduces agent central configuration within Kibana to simplify agent settings management. Custom links allow users to create dynamic, data-driven links for tasks like integrating with GitHub or Jira, enhancing workflow efficiency. These features are available via the Elasticsearch Service on Elastic Cloud or through the Elastic Stack download.

The release of Elastic Logs 7.7 introduces several new features aimed at enhancing observability and log management for Pivotal Cloud Foundry (PCF) operators and application developers. Key additions include support for PCF observability through a beta release, a new Metricbeat module for PCF monitoring, and the availability of PCF application logs via Filebeat input. This release also addresses common challenges in PCF environments, such as identifying noisy apps and managing tile lifecycle, by offering an easy-to-install and scalable integration developed as a PCF app. Improvements to the Logs app include a refined date/time picker for more intuitive navigation during troubleshooting and enhanced log categorization with sample log lines for easier analysis. Additionally, the update brings general availability for MQTT protocol support, facilitating log retrieval from IoT devices using lightweight messaging. Looking ahead, Elastic plans to introduce alerting features in the Logs app. The new version is accessible via the Elasticsearch Service on Elastic Cloud or by downloading the Elastic Stack.

Elastic Security 7.7, released in May 2020, introduces significant advancements in cybersecurity by enhancing threat detection, response, and management capabilities. This version features embedded case management to streamline incident responses and reduce mean time to respond (MTTR) through a built-in workflow that facilitates organizing and escalating security cases. The integration with ServiceNow IT Service Management (ITSM) ensures seamless coordination and tracking of incidents, while new alerting features and machine learning enhancements improve anomaly detection and notification systems. Additionally, Elastic Security 7.7 expands data source support with new Filebeat modules for Okta and Microsoft 365, further enhancing visibility into cloud and application environments. The release also includes prebuilt detection rules aimed at countering the latest security threats, offering organizations automated protection against sophisticated attack techniques. Elastic Security 7.7 is available on Elasticsearch Service on Elastic Cloud and as part of the Elastic Stack, providing users with a robust platform to protect against hidden adversaries.

Elastic Metrics 7.7.0 has been released, featuring enhanced Prometheus integration, support for various cloud services, and new alerting capabilities. This update introduces PromQL queries and OpenMetrics support, allowing users to store, transform, and visualize Prometheus metrics in Elasticsearch for improved observability in containerized and hybrid environments. The release also includes new integrations for monitoring cloud services across Azure, AWS, GCP, and PCF, enabling comprehensive metrics collection and analysis from container services, serverless functions, databases, and load balancers. Notably, the new alerting functionality is free and offers a streamlined, customizable experience within Kibana, allowing users to create precise alerts across dynamic infrastructures. This version also brings support for IBM MQ, Redis Enterprise, and Istio monitoring, while offering the ability to define custom metrics for more tailored infrastructure views.

Celebrating its 20th anniversary, Apache Lucene has a promising future as key contributors and project members share their hopes and ongoing developments for the project. Doug Cutting, Lucene's founder, and other community members highlight recent advancements like the implementation of block-max WAND, which enhances efficiency by finding top hits without evaluating all documents matching a query. This improvement, although seemingly minor, could lead to significant savings in computing power given Lucene's widespread deployment. Discussions about integrating support for high-dimensional data are ongoing, indicating the project's potential to adapt and grow. The enthusiasm and momentum within the community suggest that Lucene will continue to play a crucial role in powering digital search experiences globally.

Elastic's latest release, Enterprise Search 7.7, integrates App Search and Workplace Search into a unified solution that offers flexibility, control, and scalability for search deployments. The update introduces configurable default options for various settings, allowing users to scale their App Search deployments without performance trade-offs often associated with traditional hosted search experiences. This version addresses the "noisy neighbor" issue by enabling users to adjust limits such as document size, schema field count, and query length based on their unique needs, ensuring that performance is not affected by other users' resource consumption. Available on Elastic Cloud with a transparent pricing model, Enterprise Search 7.7 can also be downloaded for self-managed deployments, offering a versatile approach to enterprise search solutions.

Version 6.8.9 of the Elastic Stack has been released, providing recommended upgrades that include fixes and small enhancements across the stack. Users are encouraged to upgrade to this latest version to benefit from these improvements. For detailed information on changes for individual products within the stack, users can refer to the 6.8.9 release notes, which cover Elasticsearch, Logstash, Kibana, and Beats.

The new alerting framework introduced in Kibana 7.7 as part of the Elastic Stack aims to enhance user experience by providing a comprehensive and integrated alerting solution across Elastic Observability, Elastic Security, and the Elastic Stack. This release, available in public beta, supports seamless alert creation and management directly within applications like SIEM, APM, Metrics, and Uptime, with a focus on making alerts more actionable. Users can configure alerts for various use cases and easily integrate third-party solutions such as Slack, PagerDuty, and Microsoft Teams, among others, with intuitive interfaces and connectors. The framework emphasizes the detection of significant signal shifts, enabling responsive actions through a distributed task manager that allows for scalability, and supports organizing alerts within Kibana Spaces, with plans to enhance authorization models.

Elastic Enterprise Search 7.7 marks a significant milestone with the introduction of a combined download experience for App Search and Workplace Search, streamlining the user experience and integrating with Elasticsearch. The release also sees Workplace Search move to general availability, offering companies an enterprise-grade search experience across various productivity tools. Elastic Enterprise Search aims to simplify building powerful search experiences for both customers and employees, addressing the need for high-quality search in customer service and internal communications. Elastic App Search continues to enhance its offering with increased customization capabilities for deployment scaling, reflecting its core role in providing excellent customer-facing search experiences. The release underscores the balance between scalability and ease of deployment, supported by positive feedback from beta testers and an ongoing commitment to customer support and future development.

Elasticsearch 7.7.0, based on Lucene 8.5.1, introduces several new features and enhancements aimed at improving performance and usability for users. Key updates include asynchronous search capabilities, which allow long-running queries to be executed in the background with progress tracking and partial result retrieval. This feature is particularly useful for searching large datasets, such as those required for regulatory audits or threat hunting. The release also significantly reduces heap memory consumption for time-series data by moving the terms index of the _id field off-heap, enhancing cluster management efficiency. Security improvements include the option to password-protect the Elasticsearch keystore, while the beta release of the Painless Lab in Kibana offers a platform for easier script testing and debugging. Additionally, performance enhancements for time-sorted queries and support for new platforms like Red Hat Enterprise Linux 8 and Windows 2019 are included, alongside the availability of official builds for ARM architectures and the introduction of Elastic Helm charts for Kubernetes deployment.

Elastic Stack 7.7 introduces a range of enhancements and new features, demonstrating the team's resilience and commitment to delivering value amidst challenging times. This release includes the general availability of Workplace Search, offering a unified search experience across various work content, and a reimagined alerting framework in Kibana, enhancing user interaction with predefined actions and third-party integrations like Slack and PagerDuty. Elastic APM now features service maps for better visualization of service dependencies, while asynchronous search in Elasticsearch 7.7 optimizes cost and latency trade-offs for large data queries. Elastic Security 7.7 enhances incident response with embedded case management in SIEM, integrating with ServiceNow for efficient cross-organizational ticketing. Additionally, the update brings expanded integrations with technologies like Prometheus, AWS, Google Cloud, and others, simplifying instrumentation and increasing visibility across systems.

Ingesting data into Elasticsearch can be enhanced by enriching documents with additional information at the time of ingestion, a process that involves merging data from authoritative sources into documents. This enrichment allows for efficient query-time operations such as geographical searches using tools like the GeoIP Processor. Historically managed by Logstash, enrichment can now be directly executed in Elasticsearch since version 7.5.0, using the enrich processor on ingest nodes, eliminating the need for additional systems. The enrichment process can be automated using CSV files containing Master Data, which can be imported via Kibana's Data Visualizer. The process involves creating an enrich policy, executing it to form an enrich index, and setting up an ingest pipeline to merge Master Data into incoming documents. This enriched data, which can include information like device location and type, enhances the utility of documents for searching and visualization purposes, and can be seamlessly integrated into production environments by configuring index settings to apply the ingest pipeline automatically.

The blog post provides a detailed guide on integrating new security data sources into Elastic SIEM, now known as Elastic Security, specifically focusing on adding CrowdStrike Falcon endpoint data. The author, Tony Meehan, shares his experience on the process, which involves using Filebeat to collect data from log files and an Elasticsearch ingest pipeline to convert these logs into the Elastic Common Schema (ECS). This transformation allows users to leverage prebuilt SIEM detections and create custom detection rules to enhance security measures. The guide outlines the configuration of Filebeat for log data collection and JSON decoding, followed by mapping CrowdStrike fields to ECS using an Elasticsearch ingest pipeline, which categorizes alerts as malware events. It highlights the community-driven nature of Elastic Security, encouraging contributions and participation through the community Slack workspace, and hints at further enhancements, such as a new Filebeat module for CrowdStrike data coming in future releases.

Elasticsearch Service has expanded its presence on Google Cloud Platform (GCP) by becoming available in the Finland (europe-north1) region, providing users access to advanced search, observability, and security features, as well as tools like Kibana Lens for visualization. Existing users can utilize this new region immediately, while new users have the option to start with a free 14-day trial. The collaboration between Elastic and Google aims to enhance the service through native GCP console integration and the introduction of more GCP regions. Customers can subscribe to the Elasticsearch Service via the GCP Marketplace, benefiting from integrated billing and deductible usage charges. For those looking to migrate existing clusters or deployments, detailed documentation and webinars are available to assist with Elasticsearch data migration.

Software services are integral to modern businesses, necessitating service reliability to meet user expectations and maintain competitive advantage. The blog discusses the critical role of Site Reliability Engineering (SRE) and incident response, emphasizing the use of Elastic Observability to ensure service reliability and minimize downtime. SRE involves maintaining service level objectives through metrics like availability, latency, quality, and saturation, while incident response encompasses the lifecycle of prevention, discovery, and resolution of service disruptions. Elastic Observability enhances this process by providing continuous monitoring, alerting, and a unified search experience to quickly address and resolve incidents. It uses the Elastic Common Schema for standardized data management, offering integrations with various data sources to streamline incident response in complex, distributed environments. The blog illustrates how Elastic Observability aids in reducing mean time to resolution and safeguarding service reliability through practical examples and highlights its success stories, such as Verizon's significant reduction in MTTR using Elastic's solutions.

Elasticsearch Service has expanded its availability to the Google Cloud Platform (GCP) Taiwan region, enabling users to utilize its search, observability, and security solutions with ease. New users can enjoy a 14-day free trial, while existing users can start using the service immediately. The service offers advanced features like machine learning, index lifecycle management, and intuitive visualization tools such as Kibana Lens and Canvas. Elastic's collaboration with Google enhances the service with native GCP console integration and marketplace accessibility, allowing integrated billing for convenience. Elastic continues to expand GCP regions and provides resources for those transitioning to the new service, including detailed documentation and webinars to assist with Elasticsearch data migration.

Elasticsearch Service has expanded its availability to the Google Cloud Platform (GCP) in the Netherlands, allowing both existing and new users to take advantage of this development. Users can log in or sign up for a free 14-day trial to access features such as machine learning, index lifecycle management, and intuitive visualization tools like Kibana Lens. This expansion is part of a collaboration between Elastic and Google to enhance native GCP console integration and provide seamless user experiences with use case-ready deployment templates. Additionally, GCP Marketplace customers can benefit from integrated billing, consolidating charges into existing GCP accounts. Elastic continues to support users with resources, including documentation and webinars, for migration and deployment of Elasticsearch data on the new platform.

Elasticsearch users often face challenges with heap memory usage when attempting to store large amounts of data per node, leading to potential stability issues as they push their storage limits. The problem arises because Lucene, the underlying library used by Elasticsearch, requires some data to be stored in memory for efficient disk access, such as the terms index, which maps term prefixes to disk offsets. Traditionally, this has led to significant memory consumption, but recent changes in Elasticsearch version 7.7 have dramatically improved efficiency by moving more data structures from the JVM heap to disk, relying on the filesystem cache for frequently accessed data. This shift has significantly reduced heap memory requirements, allowing users to store more data per node and reducing costs, with some datasets experiencing up to a 100-fold decrease in memory usage. The improvements, particularly evident in datasets like Geonames and NYC taxis, result from changes that optimize the storage of Lucene indices, such as moving the terms index for the _id field on-disk, benefiting users who primarily index logs and metrics. Elasticsearch 7.7 promises to enhance storage efficiency, encouraging users to test the updates and provide feedback.