March 2021 Summaries

Elastic Cloud, a FedRAMP authorized service at the Moderate Impact level, is designed for U.S. government users and partners to leverage enterprise search, observability, and security for mission-critical decision-making. It offers a managed Elasticsearch service, enabling quick deployment and customization to meet specific needs. The platform is based on the Elastic Stack, which includes Elasticsearch for data storage and search, Kibana for data visualization, Beats for data shipping, and Logstash for data processing. Elastic Cloud supports various use cases such as enterprise search, observability, and security, providing tools for unified search experiences, monitoring, and cybersecurity. Users can take advantage of a free 30-day trial, during which they can create and customize deployments, ingest data, and build visualizations in Kibana. Elastic Cloud operates in a secure, isolated environment on AWS GovCloud, and users can enhance network security with AWS PrivateLink and IP filtering. The service aims to provide government organizations with scalable, secure, and efficient data management solutions.

Amazon has introduced the EC2 D3 instances for Elasticsearch Service on Elastic Cloud, offering enhanced processing and storage capacities for warm and cold data tiers. These instances, now available in major AWS regions like US East (N. Virginia), US West (Oregon), US East (Ohio), and Europe (Ireland), replace the previous D2 generation by providing up to 30% higher storage capacity through RAID configuration optimization, 2.5 times faster networking, and 45% better disk throughput. The D3 instances also feature a denser RAM-to-disk ratio of 1:190, leading to a 24–34% reduction in price per GB of storage. This launch underscores Amazon's commitment to offering the latest and most cost-effective instance types across cloud providers to its Elastic Cloud customers, who can easily access these benefits by creating a deployment in the Elastic Cloud console.

In the realm of security operations, differentiating between common and rare host processes is vital for identifying potential security threats. Elastic Security offers tools to detect unusual processes through machine learning, focusing on rare anomalies to streamline threat detection. By using Elastic's unsupervised machine learning, users can create models that highlight and score atypical processes, prioritizing those most unusual in the environment. The system provides prebuilt anomaly detection jobs and allows for customization to fit specific data sources and needs. Understanding and analyzing these rare processes involves investigating whether they pose a security threat or are benign, providing feedback to refine detection. Elastic facilitates this with out-of-the-box configurations and the ability to create tailored anomaly detection jobs, emphasizing the importance of modeling normal activity to identify abnormalities effectively.

Kafka, a distributed event streaming platform, can be run in various environments, including containerized setups, and is often used alongside ZooKeeper for storing configuration information. Monitoring Kafka involves tracking both the Kafka brokers and the associated ZooKeeper instances using tools like Elastic Observability, Metricbeat, and Filebeat. Kafka operates on a publish/subscribe model where events are published to topics, and consumers subscribe to these topics to receive updates. The system can be configured to control message distribution among consumers using consumer groups and partitions, with consumer lag being a critical metric indicating the need for additional consumers if it perpetually increases. The blog emphasizes the importance of using hints-based autodiscovery for dynamic monitoring in containerized environments, allowing Metricbeat and Filebeat to gather service-specific logs and metrics without manual reconfiguration. Additionally, Jolokia is used for retrieving JMX-based metrics from Kafka brokers, which are then processed through an Elasticsearch ingest pipeline to enhance data visualization capabilities in Kibana. This setup enables comprehensive monitoring and visualization of Kafka and ZooKeeper performance, ensuring efficient management of the streaming platform.

Version 6.8.15 of the Elastic Stack was released on March 23, 2021, featuring important security vulnerability fixes. Users are encouraged to upgrade to this latest version to benefit from these enhancements. Detailed information regarding the changes in this patch can be found in the release notes for each component of the Elastic Stack, including Elasticsearch, Beats, Logstash, and Kibana. For additional security details, the security page provides further insights.

Elastic has introduced a new "frozen tier" in version 7.12, which allows users to decouple compute from storage by directly searching data stored in object stores like AWS S3, Microsoft Azure Storage, and Google Cloud Storage without needing to rehydrate it. This advancement significantly reduces storage costs, offering up to 90% savings over traditional hot or warm tiers and 80% over the cold tier, while still maintaining searchability through Elasticsearch and integration with Kibana dashboards. The frozen tier eliminates the need for local data storage by utilizing searchable snapshots, and it retains a local cache for recently queried data to enhance performance. This development is part of Elastic's broader data lifecycle management strategy, which includes hot, warm, cold, and now frozen tiers, providing a cost-effective solution for storing and searching large volumes of IT data for observability, security, and compliance purposes. Elastic also offers a repository test kit for validating S3-compatible object stores, although official support is limited to specific platforms. The frozen tier is currently available as a technical preview in Elastic 7.12 and is accessible on Elastic Cloud, with further enhancements like a user-friendly slider expected soon.

The Logstash 7.12.0 release marks the latest stable version of the software, introducing several enhancements targeting Elastic Common Schema (ECS) compliance, ARM support, and JDK 15 compatibility. This release continues the journey towards achieving full ECS compliance across Logstash by introducing an ECS compatibility mode for plugins, which ensures that both explicitly and implicitly created fields align with ECS standards, and introduces a pipeline-level ECS compatibility option. Additionally, Logstash 7.12.0 graduates ARM/AARCH64 support for Linux to beta, accommodating the rising demand for ARM-based architectures, and extends compatibility to JDK 15. Users are encouraged to download the new version, explore its features, and provide feedback for further improvements.

Kibana 7.12 introduces several enhancements to improve data management and analysis within the Elasticsearch ecosystem, including the ability to manage long-running searches in the background, the general availability of location-based alerts, and the use of runtime fields in Discover and Kibana Lens. These updates allow users to save and manage search sessions for later use, share results easily, and enhance the analysis of spatial data with geo-fencing alerts. Additionally, the release incorporates mapping capabilities into machine learning tools, aiding in the visualization of geographic data and anomaly detection. The update also simplifies dashboard creation and management with improved panel saving options, encouraging reuse of visualizations across different dashboards. Kibana 7.12's new features can be explored through a free trial of Elasticsearch Service on Elastic Cloud, while further details are available in the release notes and associated blogs.

Elastic 7.12 introduces several significant enhancements across its Enterprise Search, Observability, and Security solutions, all integrated within the Elastic Stack, comprising Elasticsearch and Kibana. This release offers schema on read as a generally available feature, allowing users to create schemas at query time for greater flexibility and reduced time-to-value. The frozen tier, now in technical preview, enables cost-effective searching through object stores like Amazon S3, while autoscaling in Elastic Cloud automatically adjusts resources to maintain performance. In Elastic Enterprise Search, architectural improvements enhance storage efficiency and indexing speed, while Elastic Observability introduces a new correlation capability to streamline application performance analysis. Elastic Security benefits from analyst-driven correlation using Event Query Language (EQL) to improve threat hunting and investigation, and the Elastic Agent now includes behavioral ransomware prevention. These features collectively enhance the flexibility, efficiency, and security capabilities of the Elastic Stack, providing users with improved data management and analysis tools.

Version 7.12 of Elastic Observability introduces several enhancements aimed at improving performance analysis and infrastructure monitoring. Key updates include the addition of correlations in Elastic APM, which automatically identifies factors contributing to slow application performance, and native support for OpenTelemetry, streamlining the integration of observability data. The release also features autoscaling in Elastic Cloud to efficiently manage dynamic demands, and support for ARM architectures in Beats and Agent, catering to the growing use of ARM-based devices in cloud environments. Users can now search directly from low-cost object stores with the frozen tier, and the introduction of "save search to background" in Kibana enhances user experience by allowing long-running tasks to proceed without disruption. These updates reflect Elastic’s commitment to providing flexible and scalable solutions for observability, with future expansions planned for autoscaling and OpenTelemetry support.

Elastic Cloud's 7.12 release introduces several new features aimed at enhancing data storage, scalability, and performance for its users. The upcoming frozen data tier, available in the Enterprise tier, allows users to store large volumes of data at reduced costs while maintaining search capabilities by utilizing cost-effective object storage options like Amazon S3 and Google Cloud Storage. Autoscaling is now generally available, offering automatic resource adjustment based on storage and memory usage, with options to prevent uncontrolled cluster growth. New infrastructure improvements include D3 instances on AWS and Ls-Series virtual machines on Azure, providing higher performance at lower costs. Additionally, cross-cluster replication and search are now supported, allowing for data redundancy and reduced latency across different environments. The AWS Marketplace subscription for Elastic Cloud users now offers instant access to advanced features and improved support agreements. Furthermore, users can customize deployment endpoints with human-readable aliases to simplify management and sharing of resources.

Elastic Security 7.12 introduces a range of enhancements designed to improve cybersecurity defenses, including analyst-driven correlation and behavioral ransomware prevention. These features allow practitioners to better understand attack progressions and reduce false positives by leveraging sequence-based analysis and cross-index correlation. The update also includes advanced ransomware prevention for Windows systems via Elastic Agent, which uses low-level process data analysis to combat various ransomware families. Enhanced data management capabilities, such as schema on read and runtime fields, make it cost-effective to extend data visibility. New integrations, such as with Cisco AMP and ServiceNow, facilitate seamless data ingestion and incident response. The release also expands detection rules for macOS and Linux, extends MITRE ATT&CK coverage, and enhances threat intelligence capabilities through updated threat intelligence modules and new aggregation types in threshold rules. These improvements aim to streamline workflows, increase detection accuracy, and empower security teams to respond to threats more effectively.

Elasticsearch 7.12, based on Apache Lucene 8.8.0, introduces several new features and enhancements, making it the latest stable release available for deployment on Elastic Cloud or through download. Key updates include the general availability of runtime fields, which provide flexibility in data onboarding and searching by allowing fields to be defined dynamically, alongside the introduction of a frozen tier for cost-effective data storage and searchability on object stores like Amazon S3. The release also brings improved support for ARM architectures, advanced aggregation functionalities, and enhancements in index lifecycle management (ILM) with a refreshed user interface. Additionally, Elasticsearch 7.12 supports new queries for geoshapes, adds a data retention policy for continuous transforms, and offers a repository test kit for validating S3-compatible object stores, emphasizing flexibility and innovation across the Elastic Stack.

Elastic has released version 1.0 of its APM PHP Agent, which is part of its application performance monitoring solution designed to offer insights into application performance, error tracking, and end-user browser experience. The Apache-licensed agent, developed with community feedback, supports PHP versions 7.2-7.4 on Linux and provides auto-instrumentation for frameworks like Laravel, PHP Data Objects, and HTTP clients such as curl and Guzzle. It includes both a PHP extension and a PHP component, allowing for manual instrumentation and customization through a public API. Installation involves downloading a Linux package from GitHub, with configuration available via php.ini or environment variables, and monitoring data accessible in Kibana. Elastic encourages community feedback and contributions via their discussion forum and GitHub repository as they continue to enhance the agent with features like central configuration support and span type analytics.

Elastic Enterprise Search 7.12 introduces significant enhancements, including cloud autoscaling and improved storage efficiency, which offer up to 70% better storage utilization and a 40% reduction in indexing latency. The release also features a redesigned data architecture to enhance performance and relevance for App Search and Workplace Search, along with an advanced web crawler that now supports robots.txt files for better web content extraction. These updates underscore Elastic's commitment to providing scalable, efficient search solutions, further recognized by its recent classification as a Challenger in the 2021 Gartner Magic Quadrant for Insight Engines. Users can access these features via Elastic Cloud or self-managed downloads, with support for seamless integration of Elastic Stack's index lifecycle management policies.

Elastic has been recognized as a Challenger in the 2021 Gartner Magic Quadrant for Insight Engines, highlighting its innovative approach to creating modern search experiences with Elastic Enterprise Search. The platform is designed to offer ease of use, scalable solutions, and transparent pricing, enabling users to implement powerful search capabilities without complex setups or vendor lock-in. Elastic leverages its Elasticsearch foundation to provide speed, flexibility, and extensibility, allowing seamless integration with popular productivity tools and cloud services. By emphasizing a low-friction user experience and predictable pricing, Elastic aims to deliver fast and relevant search experiences that can adapt to the evolving needs of enterprises. The company’s commitment to innovation is evident in its frequent software updates and expansion of global cloud regions, ensuring that users benefit from the latest features and integrations.

TierPoint, a prominent provider of secure data center and cloud solutions, has adopted the Elastic Stack to enhance its IT security capabilities. Tasked with maintaining TierPoint's information security program, Mark Sellers highlights the necessity of a robust analytics platform to process and analyze vast amounts of data from varied sources, enabling faster and more informed decision-making in response to increasingly complex cyber threats. By leveraging Elastic Stack, TierPoint has significantly reduced investigation times and improved its ability to detect anomalies and emerging threats. The platform's centralized data analysis and real-time updates enhance TierPoint's protection measures. Additionally, Elastic's flexibility extends beyond security, offering advanced analytics for optimizing network performance and demonstrating its broader business value.

Elastic has been recognized as a Leader and a Fast Mover in the 2021 GigaOm Radar Report for Cloud Observability, highlighting its strengths in total cost of ownership, performance, scalability, and adaptability. Among 14 evaluated solutions, Elastic stood out for its broad telemetry and open telemetry-based full-stack observability capabilities, which cover application performance monitoring, infrastructure, services, and networks. The report also praised Elastic's deployment flexibility, especially for hybrid and multicloud environments, and its ability to conduct federated searches across clusters and clouds. This recognition underscores Elastic's progress in developing a comprehensive observability solution built on its Elastic Stack foundation, emphasizing the importance of observability in modern digital-first environments, especially as organizations have accelerated digital transformation due to recent global challenges.

Elastic Security explores the use of memory signatures as an effective method for detecting Cobalt Strike, a popular tool for red team operations and adversary simulation often exploited by malicious actors. While traditional machine learning models and behavior-based methods are valuable for identifying novel malware, signature-based detection offers a nearly zero false positive rate and prioritizes alerts, albeit with limitations in identifying unknown threats. The widespread use of packers and loaders poses challenges for long-term signature efficacy, but focusing on in-memory content extends their utility. The text outlines how memory signatures can successfully identify Cobalt Strike's Beacon payload, even when configured with advanced obfuscation techniques like the obfuscate-and-sleep option. By leveraging industry-standard tools like YARA, Elastic demonstrates that memory signatures can detect Beacon's presence despite its stealth features, emphasizing the importance of in-memory scanning as a robust detection strategy.

Elastic Cloud offers significant business value through cost savings, efficiency improvements, and enhanced scalability for organizations using Elasticsearch. By migrating to Elastic Cloud, companies can reduce IT infrastructure and storage costs, minimize server maintenance and data center overhead, and lower costs associated with overprovisioning, data duplication, and software licensing. Additionally, Elastic Cloud helps streamline IT operations by reducing the workload on DevOps teams, simplifying upgrades, and mitigating risks related to critical vulnerabilities and service disruptions. The platform's built-in security and compliance features further decrease the risk of data breaches and regulatory violations. The adoption of Elastic Cloud can lead to improved productivity and availability, with case studies indicating a substantial return on investment for businesses that have transitioned to this managed service.

The Elastic Common Schema (ECS) offers a standardized model for organizing data within the Elastic Stack, facilitating consistent examination through search, visualizations, and automated analysis. While Elastic provides numerous integrations that adhere to ECS standards, custom data sources can also be normalized to this schema, though it can be a time-consuming process. The Elastic Security detection engine aids in identifying ECS non-compliance, generating alerts for events lacking the ecs.version field or using improper field values. Detection rules can also ensure that network events have both source and destination fields populated, and that fields processed by ingest pipelines, such as user agent fields, are complete. These detection rules help refine data ingestion by promptly highlighting non-compliant events, which can be addressed systematically. A free trial of Elastic Cloud is available for users to experiment with these detection rules and improve ECS compliance in their data processes.

Managing Elasticsearch data across multiple indices can be streamlined using Filebeat, Index Lifecycle Management (ILM), and data streams, as demonstrated by Hung Nguyen. Filebeat is a lightweight log shipper that efficiently collects, parses, and manages data from various sources into Elasticsearch, offering prebuilt Kibana dashboards for visualization. The process involves creating specific index lifecycle policies for different data retention needs, such as keeping Google Cloud data for 30 days and Azure and AWS data for 365 days. By leveraging data streams and Elasticsearch index and component templates, users can avoid common sharding issues and ensure data is separated based on its source, enhancing management efficiency. The approach enables application of distinct data retention policies, selective data access, and tailored data availability, all while simplifying complex document-level security configurations. The blog provides a practical example of managing cloud data sources and offers a step-by-step guide to setting up the system, encouraging users to try it and explore additional features like Elastic Observability for comprehensive ecosystem insights.

Version 7.11.2 of the Elastic Stack, which includes Elasticsearch, Kibana, Beats, Logstash, Elastic Enterprise Search, Elastic Observability, APM, Elastic Security, and Elastic Cloud, was released on March 10, 2021. This update is recommended for users as it incorporates fixes and minor enhancements across the stack. For detailed information on specific changes for each product within the Elastic Stack, users are advised to consult the 7.11.2 release notes.

Organizations face significant challenges in detecting threats in AWS CloudTrail logs due to the nature of Cloud API transactions, which often lack conventional network or host-based evidence and can be indistinguishable from benign activities. The blog post by Craig Chamberlain explores the limitations of traditional search-based detection rules and highlights the use of machine learning techniques, specifically Elastic's anomaly detection, to identify unusual activities in cloud environments. Through case studies, the post illustrates how attackers have exploited cloud vulnerabilities, such as exfiltration via snapshots and multistage lateral movement, which conventional methods might miss. It emphasizes the effectiveness of combining machine learning-based anomaly detection with traditional search rules to enhance threat detection. Elastic's tools are designed to assist security teams in monitoring cloud API logs, identifying rare and suspicious activities, and efficiently managing alerts.

The article by Jason Tedor and Jason Zucchetto discusses the merits and distinctions between Elastic's searchable snapshots and AWS's UltraWarm for managing large volumes of data in Elasticsearch. As data storage needs increase, the cost-effective management of this data becomes crucial. Elastic's approach involves a tiered architecture, including hot, warm, cold, and frozen tiers, to optimize storage costs and performance, with searchable snapshots offering the flexibility to separate storage from compute by using services like Amazon S3. This approach allows users to balance between cost and performance effectively, with data tiers designed to cater to various data access needs. Conversely, AWS's UltraWarm offers a two-tiered system with limitations such as higher costs and lack of adaptability, available only on AWS, which may lead to vendor lock-in. Elastic provides more extensive support, open development, and efficient data search capabilities, making it a preferred choice for users requiring flexibility across different cloud environments. The article emphasizes that Elastic's solution offers a more comprehensive and adaptable option, especially for those looking to manage long-term data storage without compromising on performance or cost.

On March 2, 2021, Microsoft released a security update addressing several 0-day exploits targeting on-premises Microsoft Exchange servers, revealing four remote code execution vulnerabilities, identified as CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065. Microsoft urged users to patch their systems promptly due to active threats exploiting these vulnerabilities, with adversaries aiming to obtain credentials, maintain persistence, conduct reconnaissance, and steal data. Elastic Security corroborated these findings, identifying indicators of compromise (IoCs) and offering a technical overview of their Elastic Endpoint and Elastic Endgame capabilities on their Discuss forum, which outlines eight existing and two new Elastic Endpoint rules, six existing Elastic Endgame rules, and three additional EQL queries to detect attack patterns. They also provided a list of five IoCs based on telemetry observations and encouraged users to visit their forum for comprehensive details.

Elasticsearch employs multiple caching mechanisms to enhance data retrieval speed, focusing on page cache, shard-level request cache, and query cache. The page cache operates at the operating system level, storing frequently accessed data in memory to reduce disk reads, while the shard-level request cache stores full search responses, particularly useful for Kibana visualizations, to avoid redundant processing. The query cache is more granular, caching segments of queries that are repeatedly used across different searches, utilizing bit sets for efficient memory usage. These caches are designed to prevent stale data by aligning with the lifecycle of the data and are applicable whether Elasticsearch is self-hosted or used via Elastic Cloud. The article also highlights upcoming advancements in Linux and Java, such as io_uring and Project Loom, which could further optimize asynchronous I/O operations. Monitoring these caches is crucial to ensure they are effective and not frequently purged due to data changes, with Elasticsearch providing tools for observing cache usage and performance impact.

Elastic and Grafana Labs have announced a partnership to enhance user experience by jointly developing an official Grafana Elasticsearch plugin, which combines the robust visualization capabilities of Grafana with the comprehensive functionalities of Elasticsearch. This integration, freely available to all users, marks a significant step in expanding interoperability and user choice, aligning with community feedback for more comprehensive integrations. The partnership underscores both companies' commitment to providing a seamless and enriched experience by including advanced features such as broader query language support and space-saving constructs like rollups. This collaboration builds on Grafana's evolution from a Kibana fork to a standalone solution, capable of composing dashboards across various data sources, and is a testament to their dedication to meeting the diverse needs of their users.

In this article, Mark J. Hoy discusses the new analytics capabilities introduced in the 7.11 release of Elastic Workplace Search, which enhance the exploration of query and click data through advanced dashboards and visualizations using Kibana. The update includes new analytics fields such as query refinements, filters, and feedback events, allowing for more sophisticated data analysis and visualizations. Hoy provides examples of how to create and utilize these new fields for metrics like average click position, mean reciprocal rank, and top queries with no clicks, using tools like Elasticsearch transform indices and Vega scripts for custom visualizations. The article emphasizes the flexibility of these tools, encouraging users to customize analytics to fit their specific needs and improve Workplace Search's effectiveness. Additionally, Hoy highlights the importance of metrics in understanding user engagement and retention, offering insights into how these can be visualized and analyzed to enhance workplace search functionalities. The piece concludes with an invitation to experiment with these analytics tools and provide feedback for future improvements, while also promoting a free trial of Elastic Workplace Search on Elastic Cloud.

Elastic Cloud has introduced several updates for its Elasticsearch managed service on the AWS Marketplace, enhancing subscription and billing processes. Users can now choose from four subscription tiers—Standard, Gold, Platinum, and Enterprise—directly in the AWS Marketplace, with immediate access to features and support levels corresponding to their chosen tier. The service employs resource-based pricing, allowing monthly billing based on usage, which provides flexibility for users to adjust their subscription needs without long-term commitments. Additionally, Elastic Cloud has streamlined billing by consolidating Elastic charges into a single line item on the AWS monthly bill, deductible from AWS Enterprise Discount Program agreements. Users can now leverage a new cost analysis page to review usage history, compare bills, and gain insights into spending patterns, enabling more accurate cost forecasting and better resource utilization understanding.

Autoscaling is now available on Elastic Cloud, allowing for automatic adjustment of resource capacity in response to growing data and machine learning requirements. This feature monitors the storage utilization of Elasticsearch data nodes and memory capacity for machine learning jobs, ensuring deployment can handle expanding use cases without user intervention. Autoscaling is integrated with index lifecycle management policies and can be configured through the Elastic Cloud console, API, or CLI to set thresholds that prevent unchecked scaling. Users can enable autoscaling on new or upgraded deployments from version 7.11 or above, and Elastic Agent can be used to ingest data, triggering automatic scaling of data nodes and machine learning memory. The feature aims to optimize resource usage and costs by providing only the necessary capacity, and users can find more information or seek assistance through the Elastic Community and Discuss forums.

Elastic machine learning offers valuable tools for detecting unusual patterns in large datasets by employing rare analysis techniques, which are particularly beneficial in identifying security threats, unauthorized access, or potential issues within time series data. The rare function, a key component of Elastic's anomaly detection, helps identify infrequent occurrences by analyzing frequency over time or within a population, making it distinct from outlier detection, which focuses on distance and density metrics. Rare analysis can be applied in various contexts, such as detecting unusual user activity or new error occurrences in applications, and is especially useful in security threat hunting by highlighting anomalies in login frequencies or unexpected process executions. The effectiveness of rare analysis relies heavily on the chosen bucket span for aggregating data, influencing the sensitivity of the detector to frequency changes. Additionally, the severity scoring system in Elastic's anomaly detection provides insights into the rarity of detected anomalies, aiding security practitioners in evaluating potential risks. Elastic machine learning's rare function is a crucial tool for identifying unique signals in data, with further resources available for users via Elastic Cloud trials and future posts on specific use cases.

The 2021 Elastic Contributor Program winners were announced at the Elastic Community Conference, celebrating individuals who have significantly contributed to the Elastic community by organizing events, delivering presentations, contributing code, writing or translating articles, and creating video tutorials. These contributions not only strengthened the community but also expedited the implementation of a requested feature for an Elasticsearch ingest pipeline and integrated four security rule contributions into version 7.12. The program awarded Elastic training subscriptions, certification attempts, and digital badges to the winners, recognizing their dedication and impact across various regions, including Europe, the Middle East, Africa, East Asia, South Asia, Southeast Asia, Oceania, North and South America.