July 2021 Summaries

Elasticsearch Index Lifecycle Management (ILM) is designed to automate the management of index data across various phases, but users often encounter common configuration issues that require troubleshooting. These include ensuring ILM is running, correctly configuring data deletion phases, understanding the min_age calculation, explicitly attaching policies to indices, and managing existing indices manually. Users should be aware that policy updates only take effect when indices move to a new phase, and ILM operations are performed with permissions of the last editing user. Errors often arise from misconfigured rollover aliases, incorrect index name patterns, and insufficient cluster resources. Effective troubleshooting involves checking policy attachments, resolving alias conflicts, and maintaining good cluster hygiene, with resources such as ILM history and verbose logging available for deeper insights. If issues persist, users are encouraged to seek assistance through Elastic's community and support channels.

Elastic Security's machine learning-based anomaly detection has introduced new unsupervised machine learning jobs designed to detect unusual network activity, with a focus on geographic anomalies. The approach leverages a combination of endpoint, cloud, and network data, similar to the strategic "nuclear triad" concept, to provide a robust detection system that reduces the likelihood of missing threats. A case study demonstrated the effectiveness of network data in identifying anomalies, such as unexpected traffic to Russian IP addresses, which led to the discovery of malware in a lab environment with incomplete endpoint coverage. Geographic analysis of network logs is highlighted as a valuable tool for uncovering unexpected patterns and potential threats, given that malware infrastructure often resides in the developers' home country, creating detectable geographic anomalies. This method is particularly useful in scenarios where endpoint instrumentation is lacking or incomplete, underscoring the importance of a multi-faceted detection strategy.

Yukiya Shimizu explores how to monitor Kubernetes environments using Elastic's Filebeat and Metricbeat, offering a comprehensive alternative to Prometheus and Fluentd for those new to Kubernetes monitoring or looking to leverage Elastic Observability. Filebeat and Metricbeat, part of the free and open Beats platform, serve as lightweight shippers for logs and metrics, respectively, and are well-suited for containerized environments like Kubernetes due to their Autodiscover feature. This feature allows them to dynamically track changes in Kubernetes nodes, pods, and services, automatically correlating metadata with corresponding events and applying settings based on pod annotations. The blog details deploying these tools as DaemonSets on Kubernetes, ingesting data directly into Elasticsearch, and using Kibana for monitoring, with additional insights into collecting Prometheus custom metrics through Elastic APM. The ability to visualize logs and metrics through predefined dashboards and integrate with Elastic's machine learning and alerting features enhances the observability capabilities, offering users an efficient and effective monitoring solution for their Kubernetes environments.

ENGIE, a global energy company, has expanded its security-as-a-service offerings to encompass both operational technology (OT) and traditional IT systems, leveraging Elastic Stack to enhance security across its facilities and networks. The Stuxnet attack on Iran’s uranium enrichment plant serves as a cautionary tale, underscoring the vulnerabilities in the energy sector, particularly as connectivity increases through cloud computing and the Internet of Things (IoT). ENGIE's deployment of Elastic Stack allows for comprehensive data search, anomaly detection, and flexible alert systems, offering real-time and historical visibility of potential vulnerabilities. This robust security infrastructure has not only protected ENGIE’s own systems but also opened new business opportunities, providing customized security services to a diverse range of clients in sectors such as energy, healthcare, and agriculture. The company aims to capitalize on the growing demand for remote monitoring and data gathering, particularly in industries disrupted by the COVID-19 pandemic, as well as in agriculture, where IoT technologies promise to revolutionize crop management. Elastic's proactive approach to software updates and feature requests further positions ENGIE as a competitive differentiator in the realm of security and operational technology services.

Elastic Cloud has expanded its services to the Asia Pacific Northeast 3 (Seoul) region on Google Cloud, enabling users in this area to leverage enterprise search, observability, and security capabilities. This expansion allows customers to efficiently search applications, monitor their environments, and enhance security with endpoint security and SIEM. Elastic Cloud offers flexibility in deployment, allowing users to choose between managed services or self-management across various cloud platforms like Amazon Web Services, Google Cloud, and Microsoft Azure. Users can easily configure their deployments according to specific needs related to compute, memory, or I/O requirements. Additionally, Google Cloud customers can subscribe to Elastic Cloud through the Google Cloud Marketplace, integrating their usage charges with existing Google Cloud bills, and the service is accessible via a simple login to the Elastic Cloud console or a free 14-day trial.

Elastic has introduced a new Support Portal as of July 25, 2021, designed to enhance user experience by integrating seamlessly with Elastic Cloud and improving performance, scale, and relevance. The portal retains existing functionalities such as case creation and management, subscription information, and license downloads, while addressing previous user feedback for a more efficient support system. The updated portal supports larger file uploads, offers a Dark Mode, and allows the use of Markdown for code exchanges. Users are encouraged to ensure they have an Elastic Cloud account to continue accessing support services, while existing data remains intact. Some files may need re-uploading due to backend changes, but overall, the transition aims to provide a more responsive and relevant support experience.

Orange Business Services, a division of Orange Group serving 253 million customers in 26 countries, is modernizing its Security Information and Event Management (SIEM) system using the Elastic Stack to enhance its security monitoring capabilities. By integrating Elastic, the company aims to achieve faster investigation times and simplify data integration, leveraging Elastic's machine learning features to detect anomalies that traditional systems could not. The implementation includes creating a proof of concept within three months, utilizing Elasticsearch, Logstash, Kibana, and Beats across 80 servers to process nearly 100 billion logs. The new system centralizes data, allowing for improved network anomaly detection and efficient data management, and it employs Elastic's Common Schema to facilitate machine learning applications. The company plans to further enhance its security measures by integrating Elastic Security for greater automation and endpoint protection, which is expected to boost productivity and streamline security operations.

The Elastic Stack is a versatile data search and analytics platform used by a diverse range of organizations, including Walmart, NASA, Airbus, Adobe, and the French government, to tackle various challenges by ingesting, searching, analyzing, and visualizing data in real time. Walmart employs Elastic to combat fraud by analyzing metadata to identify fraudulent activities, while Adobe enhances its image search capabilities through Elasticsearch plugins. During the COVID-19 pandemic, Elastic enabled WaKED-CO to efficiently organize and access scientific research, and Airbus utilized the platform to digitize and manage aircraft documentation for thousands of users. NASA's Jet Propulsion Laboratory leverages Elastic to monitor spacecraft metrics, allowing for the swift detection of patterns and anomalies. These examples demonstrate Elastic's adaptability in addressing complex data challenges across different sectors.

Version 7.13.4 of the Elastic Stack has been released, addressing a security vulnerability that allowed authenticated users to submit malformed queries to Elasticsearch clusters, potentially exposing sensitive data like documents or login credentials. This patch also resolves issues in Workplace Search related to the Confluence Cloud connector, which had previously failed to download and extract attachment content or sync permissions due to a backwards-incompatible API change by Atlassian. Users are advised to upgrade to this latest version for improved security and functionality, and further details are available in the security advisory and release notes.

Elastic's partnership with Alibaba Cloud has significantly expanded their user base and community, showcasing Elastic's dedication to forming alliances with trusted cloud infrastructure providers. This collaboration, marked by the receipt of Alibaba Cloud’s Walking Together Award, highlights the integration of Alibaba's self-developed Elasticsearch as part of its Big Data product family, which facilitates ease of access and use for customers in China. Both companies have invested heavily in their partnership, focusing on a commercial offering rather than an open-source one, and aligning their product and engineering teams to enhance the customer experience. The partnership leverages Alibaba Cloud's substantial market presence and community in China, attracting developers and enabling digital transformation through a shared commitment to cloud-native solutions. As Elastic and Alibaba Cloud continue to grow together, they anticipate the launch of new SaaS offerings and an ongoing deepening of their collaborative efforts, underpinned by mutual trust and shared innovation goals.

Elastic has launched free on-demand training courses for its Enterprise Search solutions, specifically Elastic Workplace Search and Elastic App Search, which are designed to offer an immersive learning experience remotely. These self-paced courses are structured with expertly designed materials, demos, and hands-on labs to aid users in creating effective search experiences for workplaces, websites, and apps using the Elastic Stack, known for its speed and scalability. The App Search Fundamentals course focuses on developing search engines with modern, natural search capabilities, while the Workplace Search Fundamentals course teaches how to create a unified search platform across various content sources. Additionally, the App Search Web Crawler Fundamentals course provides guidance on using web crawlers to ingest and search web content. These courses aim to equip users with the necessary skills to deploy robust search experiences using Elastic Cloud, with resources available on the Elastic training page and free trials of Elastic Cloud for practical application.

Free and open software has become integral to modern technology, driving innovation across the application stack, particularly in security. This model enables rapid discovery and resolution of security issues due to its collaborative nature and wide usage. Elastic, a key player in this space, transitioned its security features to a free and open model, enhancing its offerings with capabilities like role-based and attribute-based access control to meet diverse user needs. Elastic's approach not only fosters innovation through its community engagement but also emphasizes transparency and collaboration, aligning with enterprise demands for accessible source code. Elastic's unified architecture supports enterprise search, observability, and security, leveraging the Elastic Stack to accelerate the evolution of its solutions. Despite not adhering to Open Source Initiative-approved licenses, Elastic's "free and open" model is considered sufficient for most enterprise requirements, continuing to resonate with customers and partners.

Elastic has been recognized as a Leader in The Forrester Wave: Cognitive Search, Q3 2021, which underscores its commitment to enhancing search experiences through Elasticsearch. Elastic received the highest possible scores in strategy, operations, and market awareness, reflecting its investment in innovation and integration with platforms like Slack, Box, and Gmail. The company's Elastic Enterprise Search makes it easier for customers to create efficient search experiences by simplifying the complexities of Elasticsearch, providing tools for quick project setups, and offering flexible deployment options. The report praises Elastic's scalability, ease of adoption, and straightforward pricing model, which charges based on compute resource usage and accommodates businesses of any size. Elastic's solutions are designed to support unified search across various content sources, ensuring growth alongside business expansion.

Yamin Tian's article delves into the intricacies of optimizing CPU time performance related to security software, specifically focusing on application startup times. It highlights how security applications, which often sit on the critical path of an application's launch, can significantly impact startup speeds by blocking or allowing execution. Tian suggests using performance testing to measure this impact, establishing a baseline for comparison with and without security software. Once performance issues are identified, the article advises isolating problematic features and using code instrumentation to pinpoint delays. It discusses alternatives to traditional methods, such as using the macOS Endpoint Security framework to obtain code signing information, which can reduce startup time impacts without compromising security. The article underscores the importance of addressing performance early in the development process and suggests using tools like Elastic Observability for ongoing monitoring and improvement.

This tutorial series provides a concise guide on integrating search functionality into an iOS app using Elastic App Search, specifically targeting a movie database. The two-part series utilizes Elasticsearch version 7.12.x and walks users through setting up an Elastic Cloud trial, downloading a movie dataset from Kaggle, and configuring Elastic App Search to index and manage the data. The tutorial includes detailed steps for setting up the environment, configuring schemas, and bulk ingesting data using a Python script that interfaces with App Search through specified API endpoints and credentials. The series culminates in the ingestion of documents into App Search, preparing users to proceed with building the iOS app in the subsequent part of the series.

In the continuation of the blog series on integrating Elastic App Search with an iOS app, this tutorial guides users through creating a movie database search application using SwiftUI and Elasticsearch version 7.12.x. The process begins with setting up a new Xcode project, configuring its attributes, and then building the user interface within the ContentView.swift file using Swift's state management to handle search queries. The tutorial details creating an AppSearch class to manage queries, handling JSON responses, and integrating asynchronous image loading with a third-party solution. It also covers optimizing search results by adjusting relevance tuning in the App Search instance, allowing users to enhance search quality without modifying the code. The guide emphasizes the power and scalability of Elasticsearch, enabling robust search capabilities in mobile applications and encouraging users to explore additional App Search features like synonyms and curations.

On July 7, 2021, Elastic Stack released version 6.8.17, which includes a series of fixes and minor enhancements across the stack, along with important security vulnerability patches. Users are encouraged to upgrade to this latest version to benefit from these improvements. Detailed information about the changes in each component—Elasticsearch, Kibana, Beats, and Logstash—can be found in the release notes, and additional insights into the security fixes are available on the company's security page.

Version 7.13.3 of the Elastic Stack has been released, offering users a series of fixes and minor enhancements, along with important security vulnerability patches. It is recommended that users upgrade to this latest version to benefit from these improvements. The release encompasses updates across various components of the stack, including Elasticsearch, Kibana, Beats, Logstash, and Elastic Enterprise Search, with more detailed information available in the 7.13.3 release notes and the security page.

Elastic Security effectively protected users from the REvil ransomware attack that targeted Kaseya and its customers through a multi-layered approach, preventing 100% of the ransomware samples tested. The system emphasizes the importance of layered defenses, combining SIEM, Endpoint Security, and XDR capabilities to detect and prevent attacks before they cause damage. Elastic's signatureless MalwareScore model, which uses machine learning to preemptively stop unknown attacks, and behavioral ransomware prevention, which monitors potential ransomware activity in real-time, both played critical roles in thwarting the attack. The Kaseya attack, which affected between 800 and 1500 victims, highlighted the vulnerability of small businesses relying on MSPs, as the attackers demanded a $70 million ransom. Elastic Security continues to innovate with additional protections like canary files to detect ransomware behavior early and aims to protect global data through its open and comprehensive threat detection platform.

Elastic Cloud is enhancing its security by deprecating weak ciphers used to connect to the Elasticsearch Service, opting to support only those in the Mozilla intermediate list, a move scheduled to take effect after January 30, 2022. This change, aimed at aligning with security best practices, includes additional support for AES128-GCM-SHA256 and AES256-GCM-SHA384 for Windows 11 compatibility, while retaining flexibility for future updates. Clients using outdated ciphers must update their systems to maintain connectivity with Elasticsearch clusters and associated services like Kibana and APM Server, as failure to do so will prevent successful TLS handshakes and result in downtime. Elastic Cloud is communicating these changes through emails and status updates, emphasizing the importance of client updates to ensure seamless operations across various regions, including AWS and GCP.

Elastic Security has been acknowledged in the 2021 Gartner Magic Quadrant for Security Information and Event Management (SIEM), highlighting its role in enhancing modern Security Operations Centers (SOCs) by providing a flexible and extensible platform. As part of the Elastic Stack, it supports organizations like Uber and Indiana University OmniSOC in transforming their security operations, offering features such as automated threat detection and advanced data handling capabilities. The solution is designed to address security challenges by enabling environment-wide visibility, reducing alert fatigue, and increasing SOC efficiency, while also integrating advanced functionalities like machine learning-based anomaly detection. Elastic Security's high user satisfaction is reflected in its 4.7 out of 5 rating on Gartner Peer Insights, affirming its effectiveness in fulfilling diverse SIEM use cases.

Azure Private Link integration with Elastic Cloud is now generally available, offering a secure way to connect your Virtual Network (VNET) with Elastic Cloud services while avoiding exposure to the public internet. This integration enhances security by routing data through private Azure service endpoints, aligning with security policies and compliance requirements across all subscription tiers. The setup process involves creating a Private Link endpoint with a private IP, configuring a DNS record, and establishing a traffic filter on the Elastic Cloud portal to limit access to specified deployments. By employing this method, only clients using the Private Link route can access the services, ensuring a secure connection to Elasticsearch and related endpoints. Users can start the process by accessing the Azure portal, and those new to Elastic Cloud can explore a 14-day free trial.

Elastic Cloud is migrating its TLS certificates from DigiCert to Let’s Encrypt to enhance security and automate certificate management as their service expands. This transition will begin on October 4, 2021, starting with the Azure southcentralus region, followed by a 14-day client validation period before continuing with other regions from October 18, 2021, to October 29, 2021. Customers using clients incompatible with the Let’s Encrypt ISRG Root X1 certificate need to update their clients or certificate stores to maintain connectivity with their Elasticsearch clusters. Elastic Cloud is providing guidance and support, including testing options and regular updates via email and their status page, to ensure a smooth transition for all users.