September 2020 Summaries

Machine learning within the Elastic Stack offers a powerful method for anomaly detection in extensive datasets, though understanding the root cause of anomalies often requires further analysis beyond initial detection. By utilizing custom URLs, users can extend their investigative capabilities in Kibana by linking anomaly records to additional dashboards or external sites, providing essential context for thorough analysis. This approach is demonstrated with a security use case, using Auditbeat data to detect unusual user activity and processes, and is facilitated by the machine learning job wizard in Kibana. Custom URLs allow seamless navigation between Kibana views and external tools such as Grafana, enabling users to filter data and time ranges based on specific anomaly records, thus enhancing the efficiency and depth of anomaly investigation workflows. Elastic Stack's capabilities, including prebuilt jobs and the Security solution, further support this comprehensive analysis approach.

Elastic Stack version 7.9.2 was released, featuring various fixes and enhancements, and users are encouraged to upgrade. The update includes a known issue with Elasticsearch on older Linux operating systems, as Elasticsearch 7.9.2 comes with JDK 15, which requires glibc 2.14. This version is not available on older systems like Oracle Enterprise Linux 6, RedHat Enterprise Linux 6, and CentOS 6, thus making the bundled JDK unsupported. In response, Elasticsearch 7.9.2 will automatically detect the absence of glibc 2.14 and will use JAVA_HOME to find the suitable Java runtime for these systems, requiring users to ensure JAVA_HOME is properly configured. For comprehensive updates, users should consult the release notes for each Elastic Stack product, including Elasticsearch, Kibana, Logstash, Elastic Enterprise Search, Elastic Observability, and Elastic APM.

ElasticON Global, a free virtual user conference held on October 13-15, aims to bring the Elastic community together across time zones, offering more than 100 sessions from Elastic customers, partners, and experts. The event features inspirational keynotes, product roadmaps, demos, and discussions on topics such as "Power and Privilege in Technology," hosted by Elastic's Senior Inclusion Officer Karen Penn. Notable speakers include comedian Trevor Noah and soccer star Megan Rapinoe. Attendees can engage in technical deep dives, networking opportunities, and entertaining sessions like a magic show and yoga class. Elastic is also supporting COVID-19 relief efforts by donating to charities for each participant.

In "Monitoring infrastructure and microservices with Elastic Observability," Dimitri Mazmanov explores the evolving landscape of software infrastructure, highlighting the shift from traditional physical machines to virtualized environments, cloud infrastructures, and containerized platforms like Kubernetes. This evolution has led to increased complexity in monitoring due to the dynamic nature of modern deployment environments, which require comprehensive and adaptable monitoring solutions. The Elastic Observability solution, powered by the Elastic Stack, provides a robust framework for handling this complexity by offering integrations for data ingestion from diverse sources, auto-discovery capabilities for containerized deployments, and a unified data model through the Elastic Common Schema. Elastic's tools enable organizations to gain actionable insights from logs, metrics, and other operational data, supporting anomaly detection and alerting through machine learning, thus enhancing infrastructure visibility and reliability. The platform's flexibility allows users to create custom visualizations and dashboards, seamlessly integrate with existing monitoring tools, and even combine security and observability efforts, effectively breaking down silos and improving organizational efficiency.

Elastic has introduced a new way for users to purchase its managed service, Elastic Cloud, directly through the Microsoft Azure Marketplace, offering monthly subscriptions in Standard, Gold, and Platinum tiers. This integration allows users to access Elastic's Enterprise Search, Observability, and Security features while benefiting from consolidated billing, as Elastic charges appear as a line item on the Azure bill, contributing to the user's annual Azure spending commitment. The service can be easily deployed through a configuration wizard available in various supported Azure regions, with detailed documentation provided to assist users in the process.

The blog post delves into the functionalities of the Go client for Elasticsearch, focusing on encoding and decoding JSON payloads and using the esutil.BulkIndexer for efficient data indexing. It highlights the use of different packages, such as encoding/json, tidwall/gjson, and mailru/easyjson, for handling JSON, each with varying levels of convenience and efficiency. The esutil.BulkIndexer is particularly emphasized for its ability to streamline the bulk indexing process by managing serialization, batching, and concurrency, which can significantly enhance throughput. The post also includes practical examples, such as the xkcdsearch application, to demonstrate real-world implementation of these techniques, and encourages experimentation with various configurations to optimize performance according to specific environments. Additionally, it provides insights into setting up a comprehensive indexing environment using Docker and Kafka, presenting a complete ecosystem for testing and deploying Elasticsearch applications.

Enrique Kortright's blog post illustrates the use of Elastic Observability to monitor Java applications, focusing on creating multiservice traces and correlated logs using Java ECS logging and APM log correlation. The process involves configuring a sample Java Spring application with a data-access microservice backed by MySQL, then adding a second microservice to generate comprehensive traces. The post guides readers through setting up the environment, using Filebeat to ship logs to Elasticsearch, and visualizing data with Kibana's APM app, emphasizing the importance of structured logging for effective log correlation. Additionally, it provides examples of custom instrumentation with the Elastic Java Agent API, showcases the potential of Elastic's machine learning features for anomaly detection in application performance data, and highlights the use of service maps and visualizations for detailed analysis of application behavior. By integrating these tools, the post demonstrates how Elastic APM can provide a thorough monitoring solution for complex Java applications, enhancing performance insights and facilitating troubleshooting.

Enriching data from internal private IP addresses with geolocation information can be challenging due to the nature of private networks, which do not inherently provide geographic data. The blog post by Sachin Frayne outlines a method for enriching these IPs using the enrich processor in Elasticsearch, which allows users to associate private IPs with specific geographic data such as city, country, and continent. This involves creating a custom index and policy in Elasticsearch to match private IPs with predefined geographic locations, which are then used to enrich documents with geolocation data. The method requires maintaining a lookup index of IP ranges and utilizes an ingest pipeline with processors for expansion, enrichment, and cleanup of IP data. Although currently limited to exact matches, Elastic is working on facilitating enrichment with IP ranges, which would streamline the process and reduce maintenance. Users are encouraged to test this method in their environments or on Elastic Cloud, with future updates promised to further simplify the enrichment process.

François-Clément Brossard provides an in-depth guide on setting up OpenID Connect (OIDC) on Elastic Cloud using Azure, Google, or Okta as identity providers, enabling a seamless single sign-on (SSO) experience. OIDC, an authentication layer based on the OAuth 2.0 protocol, utilizes JSON web tokens (JWTs) to communicate user identity between the identity provider and Elastic Cloud services like Elasticsearch and Kibana. The guide walks through the process of generating OAuth client credentials and configuring realms for Elasticsearch and Kibana, with specific instructions for each provider. It also covers creating role mappings to control user access based on email addresses, ensuring secure deployments. The guide notes that OIDC support is available only for Platinum and Enterprise subscriptions, and also mentions alternative authentication methods such as SAML and Kerberos for Elasticsearch Service.

As remote work surged in 2020, the challenge of locating information across numerous digital tools became more pronounced, leading to the introduction of Elastic Workplace Search as a solution for unified searching across diverse content sources. The 7.9 release of Workplace Search included a prebuilt connector for Gmail, allowing users to search their private Gmail messages alongside other integrated platforms like Google Suite, Microsoft 365, and Salesforce, all within a single interface. This integration ensures that Gmail results are displayed with relevance alongside other connected data sources, providing immediate access to new messages. Users benefit from features like autocomplete, typo tolerance, and robust filtering, which enhance the search experience. Admins can configure content prioritization to align with team preferences without needing extensive IT involvement, and Workplace Search can be accessed through a free 14-day trial on Elastic Cloud or via a self-managed download, supported by a free training guide.

The blog post by Aravind Putrevu discusses how to automate Elastic Cloud workflows using an SDK and the Elasticsearch Service API, which has recently become generally available. The API aids in automating tasks such as deployment creation and scaling, and it supports the Open API Specification, allowing users to generate SDKs in various programming languages using tools like Swagger. The article provides a tutorial on generating a Java Client SDK and using it in a sample application to perform operations like creating and listing Elasticsearch Service deployments. It also emphasizes the importance of securely managing API keys, which are necessary for performing API calls, and suggests further resources for learning and community support, including a GitHub repository and a Slack channel.

Karel Minařík's article provides an in-depth exploration of configuring and customizing the Go client for Elasticsearch, focusing on its numerous features and options for managing connections, security, logging, and retries. The client can be tailored to point to remote clusters, authenticate with API keys or Cloud IDs, and verify server certificates using custom certificate authorities. It offers extensive logging capabilities, allowing developers to track request and response details using various logger components, including options for structured JSON output suitable for production environments. The article also covers the implementation of custom retry logic, node discovery, and the development of custom transport layers for advanced use cases, such as performance optimization or client mocking in unit tests. With flexibility provided by the Elasticsearch configuration struct, users can adapt the client to meet specific deployment needs, whether for simple local use or complex, security-focused environments.

The article by Alex Marquardt explores the process of debugging broken grok expressions in Elasticsearch ingest processors, focusing on using Kibana's Grok Debugger for error identification. It illustrates a methodical divide-and-conquer approach to pinpoint errors in grok patterns by progressively isolating the problematic segment, exemplified through parsing an Elasticsearch slow log entry. The article emphasizes the importance of accurate grok patterns for structuring data to enhance observability and security operations. Additionally, it mentions alternative methods like the Dissect Processor for data extraction, which foregoes regular expressions for a simpler syntax. Through this exploration, readers gain insights into constructing and troubleshooting grok patterns, enabling them to effectively parse and structure their data.

Elastic Maps has introduced a new administrative regions boundaries layer to enhance geospatial analytics, allowing users to visualize data with location as a core component, such as tracking fleet vehicles, applying geofences, and identifying security threats through geographic coordinates. This layer, which includes boundaries for nearly 5,000 subdivisions across hundreds of countries, can be integrated with indices via the Elastic Common Schema's region ISO code. Available in Kibana's Elastic Maps, this feature enables detailed geographic analysis of log data, such as website visitor locations, and is compatible with various versions of Kibana. Users can utilize Elastic Maps Service layers beyond web logs for applications like APM, infrastructure monitoring, and security, with data derived from Natural Earth and OpenStreetMap. However, due to potential biases in world maps, users should verify the data's compliance with local laws, adhering to Elastic Maps Service Terms of Service.

Elastic Stack version 7.9.1 has been released, and users are strongly encouraged to upgrade to this latest version due to various bug fixes, most notably addressing a memory leak issue in Elasticsearch that occurs during document updates with a forced refresh and cluster state updates. This release is part of routine maintenance to enhance the overall performance and stability of Elastic Stack's components, including Elasticsearch, Kibana, Beats, and Logstash. For detailed information on all updates and fixes included in this version, users are advised to check the release notes for each individual product.

Implementing autocomplete suggestions in Japanese using Elasticsearch presents unique challenges compared to English due to the language's lack of whitespace between words and the complexity of its writing system, which includes kanji, hiragana, and katakana. Morphological analysis is necessary to parse Japanese text because words are not clearly delineated, and kanji characters often have multiple readings. The process involves converting input to romaji and handling incomplete or mistyped entries to generate relevant suggestions. Elasticsearch's native suggesters are not ideal for Japanese, so custom analyzers, including multi-field configurations and specialized token filters, are utilized to address these challenges. The implementation strategy involves creating a dedicated index for suggestions, utilizing character filters and tokenizers like kuromoji for morphological analysis, and applying techniques such as edge n-gram filtering and synonym management to handle various input forms and enhance suggestion accuracy. The document concludes with considerations for optimizing suggestion accuracy and user experience by incorporating search history and popularity metrics.

Java application monitoring aims to reduce the mean time to detect and resolve issues by leveraging Elastic Observability tools, particularly the Elastic APM Java Agent, to collect and analyze data. This process involves using the agent to instrument a Java Spring application that interacts with a MySQL database, allowing real-time monitoring of code spans, JVM metrics, and error traces via the APM app in Kibana. The agent can automatically gather span information, metrics, and errors from Java applications, supporting various Java frameworks and technologies. By integrating logs, metrics, and traces, Elastic Observability creates a comprehensive view of system behavior, enabling efficient incident response and monitoring. The Elastic Stack, with its powerful search capabilities, supports this process by managing and analyzing the collected data. The blog series illustrates these concepts through a practical example, setting up a Java Spring application to monitor database interactions and track performance metrics, with insights into extending monitoring capabilities to multiservice applications in future posts.

Elastic has announced the global launch of the Elastic Contributor Program, initially piloted in Brazil, to recognize and encourage open-source contributions within the Elastic community. The program, which officially begins on September 1, aims to reward contributors with incentives like swag, training passes, and event tickets for their efforts in sharing knowledge and enhancing the Elastic Stack. During the pilot phase, the program saw significant engagement, with hundreds of submissions and contributors being recognized at various levels. Participants, such as Felipe Queiroz, have expressed gratitude for the program's role in their personal and professional growth, highlighting its impact on local communities and the global Elastic community. The pilot's success led to increased community submissions and progress on key development requests, like converting Elastic UI Framework components to TypeScript. The initiative encourages diverse contributions, including code submissions and event organization, with top contributors to be recognized in March 2021.