June 2021 Summaries

Elastic's Information Security (InfoSec) team plays a crucial role in enhancing the Elastic Security product by acting as both a quality assurance extension and a data custodian. During a three-day observation, the team demonstrated their use of various tools and workflows, such as cross-cluster search and custom machine learning jobs, to process and analyze internal data from multiple sources, including cloud and network logs. This approach enables them to effectively manage detection alerts and triage processes. The symbiotic relationship between InfoSec and Product Management allows Elastic to refine features and address usability issues, while also providing the InfoSec team with early access to new software versions. Additionally, the team operates an internal malware sandbox for secure file testing and manages the Elastic Bug Bounty Program, which is currently being revamped for an upcoming relaunch.

In the era of remote work, finding business-critical files has become a significant challenge for many employees, impacting productivity, confidence, and future opportunities. A study by Wakefield Research for Elastic revealed that 58% of surveyed US-based knowledge workers identified searching for documents as a top work-from-home issue, with 81% experiencing difficulty in locating documents when under pressure from bosses or clients. This problem leads to time wasted, reduced productivity, and diminished employee confidence, as workers struggle to find necessary information to protect themselves professionally. Furthermore, the time spent searching for files could be better used for professional growth and building relationships. To address these issues, organizations are encouraged to implement unified workplace search solutions, which can streamline content findability, reduce frustration, and enhance productivity, ultimately allowing employees to focus on meaningful work and opportunities for career advancement.

Kubernetes, an open-source container orchestration system, has become the standard for automating application deployment, scaling, and management, but its transition to microservices has complicated monitoring. The Elastic Stack, comprising Elasticsearch, Logstash, and Kibana, offers robust solutions for monitoring infrastructure and applications, including those in Kubernetes environments. Many users prefer integrating existing open-source tools such as Prometheus, a widely-used metric toolkit, and Fluentd, a popular log collection tool, with the Elastic Stack for enhanced observability. The blog discusses deploying Fluentd as a DaemonSet and Prometheus using Helm Charts to collect and store logs and metrics in Elasticsearch, visualized through Kibana. It details using Metricbeat as a Prometheus remote storage adapter and emphasizes the potential of Kibana for advanced alerting, including leveraging machine learning. The approach is demonstrated using a simple multi-container application, Cloud-Voting-App, with examples available in the author's GitHub repository, suggesting that users can begin monitoring Kubernetes by utilizing Elastic Cloud or self-hosting the Elastic Stack.

Current users of Swiftype App Search can migrate to Elastic Cloud to take advantage of improved performance, flexibility, scalability, and easier log and analytics management. The process involves automating the transfer of engines from Swiftype to Elastic Cloud using Python scripts and APIs, ensuring that the engines on Elastic Cloud mirror those on Swiftype. Key steps include downloading engine metadata, creating new engines on Elastic Cloud, optionally backfilling documents, and migrating search settings, curations, and synonyms. Users must also manually migrate result settings and user roles, switch client applications to Elastic Cloud, and ensure data parity between the old and new engines. Elastic offers consulting services for complex migrations, and interested users can explore a free trial or a webinar for further guidance.

In the first part of a multi-part series on CPU time performance tuning for security software, the author focuses on identifying and mitigating high CPU usage issues, using a specific example on macOS. The article explains how to monitor CPU usage with tools like Activity Monitor and how to identify performance bottlenecks using macOS's spindump profiling tool. The example involves a demo program that experiences high CPU consumption due to the use of Apple's SecStaticCodeCheckValidityWithErrors function, a necessary but computing-intensive task, especially when validating large bundles like Xcode. The author demonstrates how changing a specific input parameter can significantly reduce CPU usage by avoiding the validation of bundle resources, although this introduces security trade-offs. The piece emphasizes the importance of carefully balancing performance improvements with potential compromises in security, and it hints at future discussions on other time-related performance topics in security software.

Elastic Cloud Enterprise (ECE) has introduced usability enhancements to its administration console, designed to streamline operations for administrators. Key updates in version 2.10 include a shift to host-based management and dedicated role pages for ECE proxies and the control plane, allowing for simplified management of host roles and providing a centralized view of all physical or virtual hosts. The update also includes renaming the Coordinator role to Controller, which is now part of the Control Plane tab, and changing the term "Runners" to "Hosts" in the user interface. Each host now has an overview page detailing its health, version, and roles, with additional pages for Allocator, Proxy, and Control Plane roles, offering enhanced visibility and management capabilities. Future updates promise more component-specific drilldowns and expanded dashboards, along with an upgrade to the logging and metrics cluster to version 7.12.1, incorporating the latest Elastic Stack features.

Alex Marquardt discusses improving search relevance in Elasticsearch by combining stemmers and multi-word synonyms. The approach involves using stemmers to reduce synonym lists by extracting word stems before applying synonyms, thus simplifying the process of handling different word forms. The article provides a detailed explanation of implementing custom analyzers and mappings in Elasticsearch, showcasing how to define index-time and search-time analyzers with a synonym graph token filter. By applying this setup to a test index, the blog demonstrates enhanced search results, where synonymous and stemmed terms are effectively matched, improving the accuracy and relevance of search queries. Additionally, Marquardt mentions Elastic Cloud and Elastic App Search as tools for users seeking a simplified, out-of-the-box search experience.

The Kibana team has revamped the dashboard creation process to make it more user-friendly and efficient by introducing new features in version 7.12, aimed at reducing the time and complexity involved in building dashboards. The update introduces "by-value" embeddables, which allow visualizations to exist solely within a dashboard, making them easier to manage, edit, and delete without affecting other dashboards. This contrasts with the older "by-reference" embeddables that required storing visualizations as saved objects, often complicating retrieval and management. The streamlined process enables users to create, save, and return directly to dashboards quickly, enhancing the user experience. Kibana now offers flexibility by allowing users to create visualizations that can either be added to a single dashboard or saved for reuse across multiple dashboards. This new workflow empowers users to efficiently create and manage their data visualizations, ultimately enhancing their ability to glean insights from data. Users can access these features by upgrading to version 7.12 or by trying a free trial of Elastic Cloud.

The University of California at Davis, an institution with over 30,000 students and significant research commitments, faced a complex digital security challenge due to its vast infrastructure, including 5,000 servers and 170,000 user accounts. To address the inefficiencies of their legacy security tool, ArcSight, UC Davis adopted Elastic to enhance their Security Operations Center (SOC). This transition was driven by the need for a centralized, cost-effective, and easily maintainable security logging platform with improved data visibility and federated access control. The migration to Elasticsearch, which took six months, allowed the university to handle an average of 800GB of data daily, optimizing log retention for cost benefits while empowering student analysts with valuable skills for future employment. Elastic's implementation reduced costs, delivered high performance, and bolstered the university's security, positioning UC Davis to further integrate security and machine learning into its research and educational endeavors.

Elastic is enabling the Department of Defense (DoD) to effectively harness data as a strategic asset by aligning with the eight guiding principles of the DoD's Data Strategy. Elastic's platform allows for the secure ingestion and near real-time querying of vast, diverse datasets across distributed environments, ensuring that trusted information is available swiftly and securely to warfighters. The platform supports collective data stewardship through the Elastic Common Schema, facilitating interoperability across data sources while maintaining robust data ethics through monitoring and anomaly detection. Elastic's tools enhance data collection by securely capturing data at its creation and integrating seamlessly with other systems, thereby supporting enterprise-wide data access and availability. By utilizing cross-cluster capabilities, Elastic addresses challenges related to speed and scalability in data analysis, offering a global data mesh that simplifies AI and machine learning processes. Elastic's general-purpose data access layer ensures data is fit for purpose and aids in designing for compliance by centralizing data monitoring and auditing within a unified platform, thus enhancing compliance effectiveness.

Elasticsearch's point-in-time (PIT) reader, introduced in version 7.10, provides a method to query data as it existed at a specific moment, offering a consistent view across multiple queries, unlike the scroll API, which is less efficient for deep pagination and relies on a single query context. While the scroll API temporarily holds data in a frozen state to ensure consistency during a search, it requires resource management and is not ideal for dynamic datasets. PIT, on the other hand, allows users to execute various queries against a fixed dataset in time, making it particularly useful for scenarios requiring stateful queries, such as using the Event Query Language (EQL) for time series data. It achieves this by decoupling the search request from the creation of the PIT, which is managed through an API that maintains the data's consistency across search requests, even if modifications like deletions occur after the PIT is established. Although PIT is advantageous for maintaining data consistency, especially in analytics, it requires careful resource management to avoid excessive load on frequently updated indices, with recommendations to periodically refresh the PIT id to balance resource usage and data freshness. Future enhancements are anticipated, including integration with slice queries to further optimize data retrieval.

Setting up Elastic Cloud for optimal performance involves configuring deployments for full observability, which includes enabling logs and metrics, setting up monitoring via Kibana, and configuring alerts for automated performance monitoring. The process begins with creating a deployment, updating configurations for Elasticsearch proxies, and enabling security audits, which can be adjusted for deeper traffic analysis. Users can monitor deployment health through Kibana's Stack Monitoring and investigate issues using the Observability Logs Stream, which provides real-time error tracking. Elastic Cloud's capabilities facilitate the consolidation and standardization of data, while Kibana Alerts automate notification processes, allowing for custom actions such as Slack notifications. These setups ensure that users are alerted to performance issues, providing them with the necessary tools and data views to troubleshoot effectively, while Elastic Cloud assists with maintenance and upkeep responsibilities.

Process Ghosting is a newly identified executable image tampering attack that exploits a timing gap in Windows' process creation notifications to evade security products. This technique allows attackers to write malware to disk in a delete-pending state, map it to an image section, and then delete the file while still executing it as a regular file. Unlike previous methods such as Process Doppelgänging and Process Herpaderping, Process Ghosting does not rely on code injection, process hollowing, or Transactional NTFS. It manipulates the process creation sequence to bypass security scans by deleting the file before antivirus software can examine it. Demonstrated to circumvent common security measures, Process Ghosting poses a challenge to defenders who rely on traditional malware detection strategies. Elastic Security has developed methods to detect such tampering techniques by monitoring for abnormalities during process creation. Despite its potential to bypass existing defenses, Microsoft's response indicates that this vulnerability does not meet their criteria for immediate servicing.

The blog post outlines how security teams can leverage Prelude Operator and Elastic Security to enhance their defense against cyber adversaries by conducting adversary emulation exercises. It emphasizes the importance of continuously testing detection capabilities using various tactics, techniques, and procedures (TTPs) to identify gaps in security measures. Prelude Operator, designed for simplicity and transparency, allows security teams to emulate attacks across different platforms, using features like TTP Editor and Cloud Redirector, while integrating seamlessly with Elastic Security for real-time monitoring and analysis. The collaboration between Prelude and Elastic focuses on empowering security teams with open-source tools to manage and track their security posture effectively. The article highlights how Operator facilitates team collaboration, allows for detailed tracking of adversary activities, and provides graphical reporting through tools like Canvas, ultimately fostering a robust security environment.

Version 7.13.2 of the Elastic Stack has been released, bringing with it a series of fixes and minor enhancements aimed at improving the overall functionality of the stack. This update includes notable bug fixes such as resolving issues with Kibana's embeddables migration function that caused failing upgrades, enhancing Elasticsearch's cluster coordination by ensuring the next cluster state is fully written on all failures, and preserving field mappings from the user agent processor in Elasticsearch when ECS is disabled. Users are encouraged to upgrade to this latest version to take advantage of these improvements, and more detailed information can be found in the release notes for each component of the Elastic Stack, including Elasticsearch, Kibana, Beats, Logstash, Elastic Enterprise Search, Workplace Search, and Elastic Observability APM.

Elastic Cloud on Kubernetes (ECK) simplifies the deployment and management of the Elastic Stack on Kubernetes, facilitating the setup of Elasticsearch, Kibana, and Beats. The guide outlines configuring ECK with SAML authentication using Auth0 as an identity provider and demonstrates the implementation of a hot-warm-cold architecture for managing data lifecycle stages. Utilizing Google Kubernetes Engine (GKE), the setup involves creating specific node pools for different data tiers, each with tailored hardware configurations and storage classes. Key features include node and pod affinity, dedicated storage classes, and readiness probes, ensuring optimal data distribution and system resilience. The process also involves creating Kubernetes resources like ConfigMaps for SAML metadata and setting up ingress controllers with SSL for secure access. The detailed configuration covers the creation and deployment of necessary resources, including Elasticsearch and Kibana instances, ensuring a robust and scalable production environment for data management.

The South Dakota Bureau of Information and Telecommunications (BIT) has adopted Elastic Security to enhance endpoint security and improve incident response times for the state's IT infrastructure. The BIT, tasked with providing reliable and secure services to South Dakota's 885,000 residents, found its previous SIEM solution inadequate for monitoring over 9,000 systems. Elastic was chosen for its cost-effectiveness and ability to provide better visibility into both local and remote endpoints. This transition allows the BIT to implement detection rules for threats like malicious URLs and failed login attempts, significantly improving their ability to quickly identify and respond to potential security incidents. The deployment of Elastic, with tools like Logstash and Kibana, enables the BIT to efficiently monitor critical logs and prevent malware intrusions, ensuring a secure and responsive IT environment for the state government.

ProblemChild is a framework developed using the Elastic Stack to detect living-off-the-land (LOtL) attacks, which are difficult to identify because they exploit benign software within a target environment. The framework utilizes a supervised machine learning model to analyze Windows process event data and identify potentially malicious processes based on process lineage information. Once identified, anomaly detection is used to highlight rare processes, and detection rules alert on unusual parent-child process activities, indicative of LOtL attacks. The framework's components, including the detection model, anomaly detection jobs, and detection rules, are available in the detection-rules repository on GitHub, and users can integrate these into their Elastic cluster using the provided command-line tools. The supervised model is designed to flag processes with a prediction of potential maliciousness, and users can further refine their alerting strategy by configuring anomaly detection jobs and modifying detection rules to suit their specific environment, thereby reducing false positives and focusing on significant threats. The framework is supported by Elastic's community channels for feedback and troubleshooting, and it offers a 14-day free trial on Elastic Cloud for experimentation.

Elastic's security and observability solutions offer significant performance improvements and cost savings, according to a Total Economic Impact study conducted by Forrester. The study highlights a 10x performance boost and up to 75% reduction in licensing costs, leading to a 358% return on investment. It quantifies $19 million in benefits across various categories and nearly $15 million in net benefits after costs, with a payback period of less than six months. Users reported enhanced productivity, reduced labor costs, and significant risk reduction in data breaches, underscoring the platform's impact on efficiency and effectiveness. The ability to forecast potential issues, reduce false positives, and streamline development processes was also noted, suggesting Elastic's comprehensive utility in improving business operations.

Gabriel Landau's blog post discusses a Windows exploit that allows attackers to perform highly privileged actions typically requiring a kernel driver by exploiting a vulnerability in the DefineDosDevice API to tamper with the KnownDlls cache. This exploit, affecting Windows 10 version 21H1, enables attackers to inject a DLL into a Protected Process Light (PPL) process, thereby performing actions with WinTcb privileges, such as dumping enterprise credentials and disabling security products. The blog highlights the release of PPLDump, an open-source tool showcasing this exploit, and its subsequent adaptation into Sealighter-TI, which accesses restricted Threat-Intelligence feeds. To address the vulnerability, the blog introduces PPLGuard, a tool that hardens the KnownDlls object directory by applying a dynamic access control list (DACL) to block the exploit. The post emphasizes the potential for offensive tools to exploit this vulnerability and hints at a future discussion on using Elastic Security to detect such attacks.

Przemysław Witek's blog post discusses the use of Elasticsearch transforms to track the most recent customer orders by creating an entity-centric index that contains only the latest document for each entity. The introduction of the "latest" transforming function in Elasticsearch version 7.12 optimizes the process by eliminating the need for a full scan of the source index, unlike the previous pivot and scripted_metric aggregation method, which was resource-intensive and slower. The post outlines the steps to implement this transform using Kibana's "Sample eCommerce orders" data set, including setting up the destination index template and configuring the transform to run continuously, ensuring the destination index is updated with the latest orders. Additionally, it highlights the importance of setting a retention policy to clean up outdated entries, thus maintaining an efficient and current index. The new functionality is presented as useful for various applications, such as identifying inactive customers, tracking machine statuses, and monitoring air pollution data.

In June 2021, Elastic announced changes to its licensing model for Elasticsearch and Kibana, moving from Apache 2.0 to a dual licensing system under Elastic License 2.0 (ELv2) and SSPL, allowing users to choose their preferred license. ELv2, described as a permissive, fair-code license, permits free use and modification with minimal restrictions. This shift aims to reduce confusion caused by Amazon's Elasticsearch fork, ensuring users have a consistent experience with Elastic's products. Since the announcement, Elastic has released multiple updates, enhancing features and integrations, and ensuring products like Beats and Logstash verify connections to genuine Elasticsearch clusters. The transition from Elastic License 1.0 to ELv2 is also underway for other Elastic products, aligning their licensing model to be simpler and more permissive. Elastic has committed to transparency throughout this transition and provides resources for users with questions.

Cerner Corp., a prominent healthcare IT provider, utilizes Elastic machine learning to enhance its infrastructure's health by managing and analyzing the massive volumes of data it processes daily. The company, which earns $5.7 billion annually and operates in 30 countries, aims to empower communities through informed healthcare decisions. Faced with challenges of data quality and scalability, Cerner adopted Elasticsearch as a logging solution, which significantly improved their ability to monitor network storage and virtualization services. The integration of Elastic's technology enables Cerner to manage two data feeds that help detect and respond to performance violations more effectively, reducing their mean time to knowledge (MTTK) by 75%. This approach allows engineers to address anomalies and potential issues proactively, ensuring minimal impact on customer experience. Cerner's strategy involves utilizing Logstash, Elasticsearch, and a custom network crawler to maintain a robust observability platform, as detailed in a presentation featuring Elastic's product managers.

Elasticsearch 7.13 introduces significant performance enhancements, particularly for terms and date_histogram aggregations, by internally rewriting them as filters, leading to substantial speed improvements. One notable optimization involves reading counts directly from index metadata instead of performing searches, achieving a 3.5x speed boost in some cases, as demonstrated by a reduction in request time from 130ms to 37ms on a 33 million document index. These optimizations are especially beneficial for older shards on slower storage mediums, and although they primarily enhance top-level aggregations, they also apply to sub-aggregations like max or avg. While the new version does not yet optimize nested aggregations, it represents a promising step forward, and the potential for future enhancements remains high. Users can access these improvements with Elasticsearch 7.13, available on Elastic Cloud.

Version 7.13.1 of the Elastic Stack has been released, offering several fixes and minor enhancements across its components, including Elasticsearch, Kibana, Beats, Logstash, and Elastic Enterprise Search. This patch addresses specific issues such as rendering problems in Kibana's "By value" panels after an upgrade, a malfunction in the "Math" aggregation for Table view in TSVB, and various Elasticsearch concerns like illegal access during PIT creation for frozen indexes and data loss in snapshot shard scenarios. Users are encouraged to upgrade to this version to benefit from the improvements, and detailed changes can be found in the 7.13.1 release notes.