May 2021 Summaries

Elasticsearch plays a crucial role in Adyen's payment platform, powering essential functions such as payment search, monitoring, and log retrieval. By leveraging Elasticsearch's distributed nature, speed, and scalability, Adyen provides merchants with near real-time payment search capabilities, allowing them to quickly verify successful transactions. The company employs an in-house streaming/consumer framework to efficiently index data into Elasticsearch, ensuring that processed payments are promptly searchable. For platform monitoring, Adyen uses the Elastic Stack to create solutions that deliver real-time alerts and end-to-end monitoring for various use cases across teams. This system integrates customizable monitors and analytical events to detect issues such as transaction spikes or anomalies. Additionally, Elasticsearch supports Adyen's log management infrastructure, aiding in troubleshooting and maintaining operational continuity. A recent smooth migration from Elasticsearch v1.7 to v7.6 with 12TB of data highlights its integral role in Adyen's architecture.

King & Wood Mallesons (KWM), a globally recognized law firm with a strong presence in Asian markets, utilizes Elastic to enhance its security operations, allowing them to identify and respond to threats effectively. Under the leadership of Chief Information Security Officer John Reeman, the firm leverages Elastic for threat hunting, log analysis, and endpoint monitoring to gain insights into potential security events. The firm is in the process of refining its security platform by incorporating machine learning to analyze domain name system activities and user behavior, which helps establish a baseline for recognizing abnormal activities. Reeman highlights the importance of understanding normal behavior to identify threats and emphasizes the critical role of endpoint data in the firm's security strategy, as it represents the last line of defense in their cybersecurity posture.

Elastic Observability 7.13 introduces several enhancements aimed at improving integration, scalability, and analysis capabilities. A key highlight is the seamless integration with Azure, facilitated by a strategic partnership with Microsoft, which allows users to manage Elastic deployments directly from the Azure console, simplifying the onboarding of logs and metrics from Azure services. The introduction of Fleet Server offers better scalability and flexibility in data ingestion, while new features in the APM workflows enhance root cause analysis with tools like time comparison views and scatterplot visualization. Additionally, the release supports enhanced management of APM Server and introduces exploratory data views, native OpenTelemetry support, and beta features like the synthetics agent, all designed to streamline observability and security data management within Elastic Stack 7.13.

Kibana 7.13 introduces enhanced capabilities for data exploration, offering users greater flexibility and speed in data analysis workflows through features such as the Kibana runtime fields editor, which allows for the creation, editing, and removal of runtime fields within Discover, Kibana Lens, and index patterns. The update also sees the general availability of Elastic Maps Server and supervised machine learning, enabling offline geodata analytics and advanced data modeling for complex queries. Kibana Lens, now the recommended tool for ad hoc analysis, facilitates intuitive data visualization and exploration, including pivot functionality for tables and metric-specific filters. Additionally, the release improves administrative functions with native alert data integration and a streamlined API key management interface, all aimed at optimizing workflows and enhancing the user experience. Users can explore these features through a free trial of Elasticsearch Service on Elastic Cloud or by downloading the latest Elastic Stack version.

Elastic Enterprise Search 7.13 introduces several enhancements aimed at improving workplace, website, and mobile app search experiences. The release features broader Dropbox integration, including support for Dropbox Paper, allowing users to search across content from various apps like Salesforce, GitHub, and Gmail in a unified experience. It also includes automatic syncing of Dropbox permissions to ensure proper access and introduces a beta precision tuning API for search admins to adjust recall and precision in search results. New custom source management APIs facilitate easier ingestion and management of content from various sources, while a new Logstash output plugin simplifies data ingestion from Elasticsearch indexes. Elastic Cloud customers can access these features directly, with options for new users to explore through training courses or a free trial.

Logstash 7.13.0 has been released, introducing new features and enhancements, particularly in its integration with Elasticsearch data streams, which were first introduced in Elasticsearch 7.9.0. This integration improves the management and querying of time series datasets by allowing users to write to data streams through the Elasticsearch output plugin, adhering to Elastic's data stream naming scheme to optimize storage and performance. The release also continues the effort towards ECS compliance by introducing an ECS compatibility mode for several plugins, such as CEF codec and File input, which is currently disabled by default. Users are encouraged to try the new version, provide feedback, and report any issues through various community channels.

Elasticsearch 7.13 introduces a range of enhancements, including the general availability of the frozen tier, which enables efficient searching of petabytes of data stored on low-cost object storage like Amazon S3, Microsoft Azure Storage, and Google Cloud Storage. The release also features significant performance improvements in aggregations, with up to a 92% increase in term aggregation speed under certain conditions, and the introduction of the combined_fields query for better multi-field search capabilities. Enhancements in supervised machine learning include transitioning data frame analytics and inference to general availability, allowing users to train and deploy models for tasks such as outlier detection, regression, and classification. Additional security measures have been implemented, such as audit ignore policies, improved metadata for API keys, and warnings for unsecured clusters. Elasticsearch 7.13 also introduces model aliases for easier management of trained models and provides tools to transform runtime fields into indexed fields for better query performance.

Elastic Security 7.13 introduces significant enhancements, particularly in the realm of osquery support and threat intelligence integration, aimed at improving the efficiency and effectiveness of security analysts. The release streamlines osquery management, allowing for seamless installation and query execution across various operating systems, thereby reducing the complexities and DevOps investments typically associated with osquery deployment. This version centralizes security analytics by integrating osquery results with other log and event data, enabling a comprehensive view of host activity, which is crucial for detecting cyber threats. Additionally, Elastic Security 7.13 enhances threat intelligence capabilities, including the introduction of a row renderer for alerts and support for the MalwareBazaar threat feed, which aids in rapid threat detection and response. Machine learning advancements in the Network Module further bolster the capability to detect sophisticated threats by analyzing network behavior for anomalies indicative of malicious activity. The update also introduces new prebuilt detection rules, improved endpoint security features, and expanded data integrations, offering a more robust and versatile security solution for organizations.

Version 6.8.16 of the Elastic Stack, released on May 25, 2021, includes fixes and minor enhancements across its components. Users are encouraged to upgrade to this latest version for improved performance and stability. Detailed information about the changes made to each product within the stack, such as Elasticsearch, Kibana, Beats, and Logstash, can be found in the 6.8.16 release notes.

Elastic 7.13.0 introduces a range of enhancements across its Enterprise Search, Observability, and Security solutions, which are integrated into the Elastic Stack, including Elasticsearch and Kibana. Key features include the introduction of searchable snapshots and the frozen tier for more cost-effective data search, enabling users to search petabytes of data stored on object storage. The release also offers runtime fields in Kibana Lens and Discover for on-the-fly data enhancements, and expanded Microsoft integrations to streamline deployment and management within Azure. Additionally, Elastic 7.13 includes new capabilities for Elastic Workplace Search with Dropbox integration, a precision tuning API for Elastic App Search, and improved data ingest architecture through the new Fleet Server. Security enhancements include central management of osquery with Elastic Agent, facilitating richer data analysis. These updates are available on Elastic Cloud, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes, providing users with flexible deployment options.

Elasticsearch has been integrated into the Microsoft Azure portal, allowing users to deploy and manage Elasticsearch directly from Azure, thereby enhancing enterprise search, observability, and security within the Azure environment. This integration enables users to purchase Elastic Cloud plans through the Azure marketplace, consolidate billing, and manage Elasticsearch deployments seamlessly with familiar Azure tools. It also offers features such as ingesting Azure logs with ease, single sign-on capabilities, and improved search and visualization for Azure applications. The partnership provides cloud-optimized features like Azure Private Link for secure data routing, autoscaling for performance optimization, and cost-effective data retention through Microsoft Blob Storage. Users can benefit from enhanced cross-cluster search, replication across regions, and simplified billing by incorporating Elasticsearch into their Azure infrastructure.

The latest updates to Elastic Cloud, version 7.13, enhance its integration with Microsoft Azure by introducing the public preview of Azure portal integration, allowing users to easily deploy and manage Elasticsearch within Azure. This integration offers benefits such as simplified log ingestion, consolidated billing, single sign-on, and support for Azure Private Link, which is now in private beta, providing secure, private connectivity for Elastic deployments. Additionally, Elastic Cloud has added support for two new Azure regions and has made the frozen tier generally available, enabling cost-effective data storage using object storage solutions like Azure Blob Storage. The frozen tier, powered by searchable snapshots, allows users to store large volumes of data without the need for deletion, while Fleet, now in beta, offers scalable management of Elastic Agents independently from Kibana. Users can access these features through the Elastic Cloud console or sign up for a trial to explore the improvements.

Since its introduction in 2019, the Elastic Common Schema (ECS) has rapidly evolved to facilitate consistent data structuring in Elasticsearch, growing from 279 fields in ECS 1.0 to 762 fields in ECS 1.9. ECS enables broad application of analytics content, such as dashboards and machine learning jobs, by offering customizable data structuring that supports analysis from diverse sources. The community-driven evolution of ECS has led to the addition of new fields and categories, enhancing its application in IT operations, security analytics, and application performance monitoring. The schema's extensive adoption across various sectors, including its integration into Elastic Security and Elastic Observability, underscores its utility and adaptability. However, the complexity of mapping events to ECS remains a challenge, prompting ongoing efforts to streamline this process and improve compliance validation. The introduction of a request for comments (RFC) process has been instrumental in refining ECS, leading to significant developments such as the inclusion of Threat Intelligence fields. As ECS continues to develop, it remains a pivotal tool for organizations like Uber, enhancing their security investigations and enterprise defense strategies.

The blog post discusses the ProblemChild framework, which aims to detect "living off the land" (LOtL) malware attacks using the Elastic Stack. These attacks exploit standard tools within a target environment to avoid detection, making them challenging to identify. The ProblemChild framework leverages machine learning to analyze Windows process event data and classify events as either malicious or benign. By extracting features from event metadata and training a supervised model, the framework enhances detection capabilities. It further applies anomaly detection to prioritize unusual events for analysts. The post outlines the process of feature engineering, model training, and event enrichment while emphasizing the importance of anomaly detection for refining the identification of rare malicious activities. The framework is tailored for Windows process events, with potential for expansion to other operating systems and event types in the future.

Elastic and Swimlane have formed a partnership to enhance security operations centers (SOCs) by combining Elastic's high-speed, cloud-scale analytics with Swimlane's security automation platform. This collaboration aims to optimize workflows, support security information and event management (SIEM), and improve threat detection, incident response, and other security operations through joint integrations. The partnership helps reduce dwell times, mean time to response (MTTR), and false-positive rates, allowing for quicker adaptation and response to threats. By integrating Elastic's scalable search capabilities across diverse data sources with Swimlane's extensive playbooks and workflow management, SOC teams can achieve higher security ROI and better utilize existing security investments. Both companies emphasize an open approach to security, with Elastic offering open-source code and free SIEM functionality, and Swimlane providing extensive integrations and community-shared best practices, fostering transparency and collaboration in the security community.

Public sector organizations, particularly in state and local government and education, face increasing data management challenges and security threats, exacerbated by the COVID-19 pandemic. Elastic has responded by providing solutions that allow organizations to quickly analyze data, unify intelligence, and standardize datasets to enhance security and operational efficiency. Key strategies discussed during the State and Local Government & Education ElasticON Public Sector event include leveraging Elastic's tools for a unified analyst experience, improving cybersecurity through automation and risk-based approaches, and increasing network visibility to preemptively address performance and security issues. Case studies from various institutions, such as the State of Arizona and Salt Lake County, illustrate how Elastic's capabilities have transformed their data handling and threat response processes, emphasizing the importance of a comprehensive, insights-driven approach in overcoming challenges related to data volume, infrastructure optimization, and digital ecosystem protection.

Elastic's new frozen data tier in Elasticsearch decouples compute from storage, utilizing low-cost object stores like Google Cloud Storage, Azure Blob Storage, or Amazon S3 to efficiently manage and query large volumes of data without rehydration. This innovative approach allows for scalable data management, enabling searches over vast datasets, such as petabyte-scale data, with significant cost savings. The frozen tier is designed for infrequent access scenarios, balancing performance with storage costs by leveraging an on-disk least-frequently-used (LFU) cache to optimize query speeds. Benchmark tests demonstrate that although the frozen tier's initial query performance is slower than traditional tiers, repeated queries benefit significantly from caching, achieving performance levels similar to those of the hot, warm, and cold tiers. This makes the frozen tier ideal for operational, security, and historical analyses, where fast access to large data sets is necessary without the overhead of maintaining vast amounts of local storage. The feature is integrated into Elasticsearch's index lifecycle management and is available for both self-hosted deployments and Elastic Cloud.

The ElasticON US Federal event highlighted the transformative role of Elastic's search capabilities in the federal public sector, emphasizing the need for tools that adapt quickly to increasing data volumes to ensure mission success. The event showcased Elastic's applications in security, cloud, supply chain, and compliance, with a focus on creating a unified analyst experience to enhance threat detection and response. Discussions included the impact of Elastic Cloud Enterprise on federal agencies, leveraging advanced analytics for document review, and supporting the Navy's Mission Assurance Decision Support System for real-time impact assessment. The event also featured a fireside chat with the US Air Force's Chief Software Officer, exploring the integration of DevSecOps and the importance of a holistic data view to improve security and efficiency. The event underscored Elastic's role in enabling insights-driven operations and fostering community connections within the public sector.