February 2021 Summaries

In the 7.11 release of the Elastic Stack, Elastic announced the general availability of their Ruby and Python clients for Elastic Enterprise Search, following their initial beta introduction in version 7.10. These clients, whose source code is accessible on GitHub, facilitate interactions with Elastic Enterprise Search, allowing users to index and search documents with ease. Installation instructions and documentation are provided on elastic.co for both clients. A PHP client is also under development, expected to be released in beta with version 7.12, utilizing PSR-7 as the HTTP standard. Users can explore these new clients through a free 14-day trial of Elastic Enterprise Search on Elastic Cloud or by downloading and managing it independently.

Elastic Security offers a solution for overcoming limitations in traditional SIEM systems by enabling comprehensive data collection without the constraints of per-ingest or per-endpoint costs, thus allowing security teams to fully utilize high-volume data sources like cloud application and DNS logs. By leveraging Elastic Security's capabilities, organizations can enhance their SIEM's visibility, perform thorough historical analyses, and gain security insights through diversified data types, which helps prioritize relevant alerts and reduce adversary dwell times. The platform supports automated detection and integrates with the MITRE ATT&CK framework, providing out-of-the-box detections and utilizing machine learning to identify malicious activities. Elastic's approach allows for seamless integration with existing infrastructure, offering a scalable and fast solution built on Elasticsearch to address top security use cases, enhance operational efficiency, and improve overall security efficacy.

Elastic has introduced the Elastic Certified Observability Engineer certification to acknowledge individuals with advanced skills in system observability, which is essential for monitoring the health of complex technological ecosystems. This certification is aimed at a wide range of IT professionals, including software developers, engineers, data architects, and system administrators, who are responsible for implementing observability in their systems. The certification exam is hands-on and performance-based, requiring candidates to execute real-world tasks using the Elastic Stack, such as ingesting logs, metrics, and traces, and analyzing events with Kibana and machine learning. To prepare for the exam, individuals are encouraged to take the Elastic Observability Engineer training, available in both live virtual and self-paced formats. An upcoming webinar will also provide further guidance on preparing for the certification. By achieving this certification, professionals can demonstrate their expertise in observability, ensuring that company systems are managed effectively and securely.

The article by Kent Brake outlines a comprehensive guide on how to monitor NVIDIA GPU metrics using Elastic Observability, highlighting the growing importance of GPUs in various high-performance computing applications beyond gaming, such as neural network training and data center workloads. It details the process of setting up the necessary NVIDIA tools and Elastic Observability components, including installing NVIDIA Datacenter Manager, NVIDIA's gpu-monitoring-tools, and Metricbeat, with specific instructions for configuring these tools on a cloud deployment. The guide emphasizes the modularity of Metricbeat's configuration, which enables the integration of GPU metrics via Prometheus, and provides troubleshooting tips and configuration checks to ensure successful monitoring. Additionally, it discusses the utility of Elastic Observability in analyzing GPU performance, offering insights into various metrics like GPU temperature, power usage, and clock speeds, and suggests using Elastic alerting and machine learning to automate recommendations and detect anomalies. The article concludes by inviting users to try the steps with a free trial of Elastic Cloud.

Elasticsearch's new cold tier of searchable snapshots, now generally available in version 7.11, offers a significant reduction in storage costs—up to 50% compared to the warm tier—while maintaining the reliability and redundancy of the hot and warm tiers. This tier achieves cost savings by keeping only the primary shards on local storage and relying on snapshots stored in cloud object stores for resiliency. The team conducted thorough validation tests, including full-cluster restarts, rolling restarts, and node crash scenarios, to ensure the cold tier operates efficiently and reliably at scale. These tests demonstrated the effectiveness of the persistent cache introduced in Elasticsearch 7.11, which minimized network traffic and expedited the process of returning the cluster to a healthy state. Users can begin using the cold tier by upgrading to Elasticsearch 7.11 or starting a new cluster on Elastic Cloud, with detailed guidance available in the official documentation.

In 2021, Security Information and Event Management (SIEM) systems have evolved into crucial tools for real-time threat detection and data breach prevention, requiring integrations with big data, advanced analytics, and cloud services. Elastic Security addresses the modern demands of SIEM by offering robust cloud integration, data protection, and seamless integration with security ecosystems, enhancing threat detection and incident response capabilities. The platform incorporates advanced analytics such as anomaly detection and machine learning, which aid in identifying complex threats and reducing detection time. Elastic Security also aligns with standardized cybersecurity frameworks like MITRE ATT&CK and supports flexible deployment models, including on-premises, cloud, and hybrid setups. This versatility ensures adaptability to the ever-changing threat landscape, positioning Elastic as a comprehensive solution for modern security challenges.

Version 7.11.1 of the Elastic Stack was released, featuring important bug fixes and minor enhancements. Notable fixes included addressing a regression in Elasticsearch that affected starting configurations for Active Directory or LDAP realms and resolving an issue in the Elasticsearch repository-azure plugin that prevented large snapshots from succeeding with Azure Storage Service. Users are encouraged to upgrade to this latest version for improved performance and functionality. The release encompasses updates across various components of the Elastic Stack, including Elasticsearch, Kibana, Beats, Logstash, Elastic Enterprise Search, Elastic Observability, and Elastic Cloud. For detailed information on all changes, users are directed to consult the 7.11.1 release notes.

Elastic has introduced a new usage analysis page in Elastic Cloud that allows users to explore and analyze their deployment costs more effectively, helping them understand how their resource usage contributes to their monthly bills. This tool breaks down costs into three main components: deployment capacity, data transfer, and storage, offering insights into each to aid cost management. Deployment capacity, which includes costs related to nodes and node types like Elasticsearch and Kibana, is primarily measured in RAM hours, while data transfer and storage costs account for data volumes and backup storage, respectively. The page allows users to view detailed cost breakdowns by deployment or product, offering a granular view of usage patterns and enabling cost distribution across teams or departments. Users can access real-time billing information, making it easier to adjust spending within a billing cycle, and can drill down into usage records with flexibility in time range selection. The improved interface now includes month-to-date aggregates of data transfer and storage costs, enhancing transparency and aiding in budget management. Users are encouraged to leverage this tool to optimize their Elastic Cloud deployments and consider Elastic's provided resources, such as an ebook, for additional cost-saving strategies.

Elastic Cloud's 7.11 release introduces several features designed to enhance deployment efficiency, scalability, and cost-effectiveness. Users can benefit from searchable snapshots and cold storage tiers to optimize storage costs by utilizing cloud-based storage like Amazon S3, Azure Blob, and Google Cloud Storage. The anticipated autoscaling feature will allow deployments to grow in response to increased resource demands without manual oversight. Cross-cluster search and replication enable data integration across different regions and cloud providers, improving data access and reliability. The update also includes a usage analysis page for better cost management and the Elastic Cloud Terraform provider for infrastructure automation. Additionally, users can upgrade to the Enterprise tier directly from the Elastic Cloud console, offering enhanced features such as searchable snapshots and solutions for Elastic Enterprise Search, Elastic Observability, and Elastic Security. The release also highlights simple sign-up using Microsoft accounts and Elastic Cloud on Kubernetes 1.4's support for Elastic Agent deployment.

Kibana 7.11 introduces several exciting updates, including the general availability of alerting in the Elastic Stack, marking a significant advancement after nine months in beta. This new alerting framework is crafted to meet the needs of mission-critical monitoring, offering integrations with platforms like PagerDuty, Jira, ServiceNow, and Microsoft Teams. Additionally, Kibana now supports tags for better organization of dashboards and visualizations, a feature requested by many users to enhance content management. The release also enhances data visualization capabilities in Kibana Lens with customizable color palettes and chart labels, and introduces the ability to export chart data to CSV. Elastic Maps Server is now in beta, designed for secure environments, while new geo alerts improve location-based monitoring. Kibana also offers anonymous access to share dashboards without requiring credentials, and enhanced audit logging for greater security and compliance. Machine learning jobs have become space-aware, and a new transform type called Latest allows users to track and update the most recent documents. These updates, along with a redesigned data visualizer and a map view for data frame analytics, are now available for trial with Elasticsearch Service on Elastic Cloud.

Elastic has introduced a beta version of its App Search web crawler in version 7.11 of Elastic Enterprise Search, providing a new method for ingesting publicly available web content to make it searchable on websites. This web crawler, now available for both self-managed and Elastic Cloud deployments, retrieves and indexes content from specified web pages, enhancing the searchability of content without requiring coding. Elastic Cloud offers several advantages, such as speed, scale, and simplified management, and supports deployment across major cloud providers like Google Cloud, Microsoft Azure, and AWS. The web crawler extracts key elements such as page titles, meta descriptions, and body content from HTML pages, following links to ensure comprehensive content discovery. Users can create entry points and crawl rules to manage which parts of a website are indexed, providing flexibility in content management. This new feature is positioned as a powerful tool for users to enhance their search capabilities, with a focus on simplicity and efficiency.

Version 6.8.14 of the Elastic Stack has been released, addressing security vulnerabilities and recommending users to upgrade to this latest version. The release includes updates for Elasticsearch, Beats, Logstash, and Kibana, with detailed changes available in the release notes. Users are encouraged to visit the security page for more information about the fixes.

Elasticsearch 7.11.0 introduces significant updates, including the debut of schema on read with beta runtime fields that allow field definitions at query time, enhancing flexibility without reindexing data. This release also includes enhancements to Elastic Enterprise Search, Elastic Observability, and Elastic Security solutions. Notably, Elasticsearch introduces searchable snapshots and a cold tier, offering cost-effective data storage and search capabilities without rehydration. Additionally, the release brings improvements in threat detection with Event Query Language (EQL) and geo_line aggregation for tracking object paths. New security features now log configuration changes and UI enhancements in Elasticsearch streamline user experience, particularly with Painless scripting and index lifecycle management. The release also features a new machine learning transform type, Latest, for tracking recent document updates. Elasticsearch 7.11 is available for deployment on Elastic Cloud, offering a free trial for users to explore its new functionalities.

Elastic Observability 7.11 introduces several features designed to streamline investigative workflows and improve operational efficiency, such as a new APM service health overview page that consolidates key service health data for easier troubleshooting and root cause analysis. The release also enhances the Metrics app with a detailed view of host health, enabling quicker infrastructure monitoring and issue resolution. Additionally, the Elastic Common Schema (ECS) logging libraries are now generally available, facilitating seamless log-to-trace correlation by automatically embedding trace context into logs. Other notable updates include the introduction of runtime fields for flexible data querying and the general availability of searchable snapshots, which optimize data storage and access by enabling direct searches on object stores like S3. These advancements collectively aim to reduce mean time to insight and resolution while providing users with versatile tools for managing complex observability scenarios.

Logstash 7.11.0 has been released, introducing significant enhancements to the centralized pipeline management (CPM) feature to improve user experience. This release addresses longstanding user requests by allowing pipelines to be dynamically added to online Logstash instances without a restart through the use of wildcards, making the configuration process more flexible and dynamic. Additionally, it resolves a persistent issue where deleting pipelines did not clear the pipeline registry, making the recreation of pipelines with the same name problematic. The update invites users to download, test the new version, and provide feedback through various platforms.

Elastic Stack has announced the general availability of its alerting feature with the release of version 7.11, building on a beta version introduced in May 2020. This new alerting framework integrates deeply with Elastic's products and third-party platforms like email, PagerDuty, ServiceNow, and Microsoft Teams, transforming raw customer feedback from its predecessor, Watcher, into a more efficient system. The alerting system is designed to make alerts more meaningful and actionable, facilitating workflow initiation and issue resolution through improved user interfaces, multi-tenancy support, and role-based access controls. The release introduces new third-party connectors and expanded capabilities like alert grouping and state-change triggers, enhancing the alerting experience. Elastic Stack plans to further expand this functionality, including deeper analytics integrations and additional tools to minimize alert noise and improve signal clarity. Users can explore these features by starting a free cloud trial or downloading the 7.11 release for self-managed deployments.

Elasticsearch 7.11 introduces runtime fields, a schema on read feature that complements its existing schema on write mechanism, offering users enhanced flexibility in querying and indexing data. Unlike traditional indexing where all fields are predefined and stored at ingestion, runtime fields allow for fields to be created and evaluated at query time, enabling dynamic data exploration without the need to reindex. This innovation is particularly useful for adapting to changing data formats, correcting indexing errors, and iteratively refining data structures. Runtime fields are integrated within the existing Elasticsearch platform, facilitating seamless use alongside indexed fields without additional procedural overhead. This approach empowers users to balance performance and flexibility, making it easier to adjust data structures as needed and supporting more efficient and cost-effective data management. The feature is currently in beta and further enhancements are planned to expand its capabilities in Kibana, including UI tools for field creation and query customization.

Elastic Enterprise Search 7.11 introduces several enhancements, including a new web crawler for Elastic App Search and the integration of Box as a content source for Elastic Workplace Search. The web crawler, currently in beta, streamlines the indexing of public-facing web content, offering flexible customization options to improve search relevance and efficiency. The addition of Box expands Workplace Search's library of prebuilt integrations, providing a unified and secure search experience across popular tools like Google Drive and Dropbox, with adjustable source prioritization. Enhancements also include document-level permissions for Jira and Confluence to ensure secure access and improved search relevance. Furthermore, the update introduces the Events API to gather user behavior analytics, enabling the creation of detailed Kibana dashboards to analyze search queries and click-through rates, thereby identifying content areas needing updates or removal. Existing Elastic Cloud customers can access these features directly, while new users are encouraged to explore Quick Start guides and free trials.

Elastic has announced the private preview launch of its first-party integration with Microsoft Azure, which allows users to deploy and manage the Elastic Stack, including Elasticsearch and Kibana, directly from the Azure portal as a native service. This integration offers several advantages, such as simplified deployment, streamlined authentication through single sign-on (SSO) between Azure and Elastic consoles, automated log and metric ingestion from Azure services, and secure network traffic via Azure Private Link. Users can also receive consolidated billing for Elastic usage within their Azure bills. The integration facilitates running Elastic in any Azure-supported region, enhancing data locality and reducing latency. Users can provision prebuilt solutions like Elastic Enterprise Search, Observability, and Security, or create their own applications using the Elastic Stack. This native integration simplifies the process of managing Elastic instances, allowing users to monitor their health and status without leaving the Azure console and access advanced configurations with ease.

Elastic 7.11 introduces significant enhancements across its Enterprise Search, Observability, and Security solutions, integrated within the Elastic Stack—comprising Elasticsearch and Kibana. The release highlights the general availability of searchable snapshots and a new cold tier, enabling cost-efficient data storage and flexibility through schema on read, which remains in beta. A new web crawler enhances content searchability in Elastic Enterprise Search, while Elastic Observability improves root cause analysis and troubleshooting with enhanced service health and host detail views. Elastic Security advances detection and remediation capabilities with new machine learning jobs and customizable alert notifications, and it supports extended data retention with the cold tier for scalable security data access. Elastic Cloud users benefit from improved storage management with searchable snapshots, autoscaling, and enhanced cross-cluster replication and search capabilities. Additionally, Elastic 7.11 marks a shift in licensing for Elasticsearch and Kibana to a dual model under the Elastic License and SSPL, aiming to simplify and broaden its licensing terms without impacting most customers or community users.

Elastic Security 7.11 introduces a range of enhancements aimed at improving threat detection, alert management, and data integration capabilities for security teams. Key features include prebuilt detection rules for cloud applications and Windows environments, updated machine learning jobs for anomaly detection across Windows and Linux hosts, and specialized detection rules for SUNBURST-related threats. The update also supports MITRE ATT&CK sub-techniques, offers expanded alert management actions for integration with third-party tools like Jira and ServiceNow, and introduces customizable alert notifications for better context. The usability of the Timeline workspace has been improved with features like tabbed information access and multicolumn sorting, while accessibility enhancements include better keyboard navigation and screen reader support. Elastic Agent now supports additional data sources and can ingest database audit logs, Snyk vulnerability data, and Windows security events by default. Additionally, Elastic 7.11 offers malware prevention options recognized by Windows and streamlined lifecycle management through its Fleet interface. The release also includes the general availability of searchable snapshots, allowing for extended retention and cost-effective access to high-volume security data from various sources.

Elasticsearch has introduced runtime fields, a schema-on-read capability that complements its traditional schema-on-write approach, providing flexibility in altering document schemas post-ingestion and generating fields for search queries. Runtime fields, which are evaluated at query time, do not increase index size and can reduce storage costs and ingestion speed, but they may lead to expensive queries and decreased search speed. These fields can be defined in index mappings or queries and are useful for fixing errors in indexed data by overriding values temporarily. Runtime fields allow experimentation with data structures without impacting resource consumption, and users can eventually convert useful runtime fields into indexed fields for better performance. This approach supports dynamic mapping, reducing the risk of mapping explosions by creating new fields as runtime fields initially, and later indexing them if beneficial. Elasticsearch recommends using runtime fields alongside indexed fields to balance performance and flexibility, and encourages users to explore this feature by upgrading to version 7.11 or using the Elasticsearch Service.

WaKED-CO, an initiative by the French Ministry of Armed Forces, utilizes Elastic Cloud to enhance research on COVID-19 and future pandemics by streamlining access to scientific literature and data. Developed rapidly, the project aims to track the epidemic's evolution and assist stakeholders in decision-making by integrating diverse data sources, including peer-reviewed publications, through an elaborate ETL process. The platform, leveraging Elasticsearch and Kibana, enriches data with natural language processing techniques to improve searchability and accessibility, offering real-time monitoring and notifications for critical updates. Available on France's public COVID-19 platform since April 2020, WaKED-CO supports crisis management and scientific research with an average of 2,500 daily queries, containing data from over 1.2 million publications. The initiative continues to expand, with plans to utilize tools like Canvas to visualize scientific progress, aiming to serve as a comprehensive crisis management tool beyond the current pandemic.

The Elastic Contributor Program, initially launched globally in September 2020, aims to recognize and reward contributions from the Elastic community by fostering knowledge sharing and encouraging friendly competition. The program, which has seen participants from diverse geographical and linguistic backgrounds, is set to begin its 2022 cycle on February 1. During the 2021 cycle, participants highlighted the benefits of gaining knowledge and contributing to personal and career growth, with over 750 submissions received from 105 community members across 57 countries. The program recognizes contributors with Bronze, Silver, and Gold statuses at the Elastic Community Conference and offers rewards such as swag, extended Elastic Cloud trials, and training. New contribution types, like technical answers, have been added, allowing participants to earn points for engaging with Elastic products on platforms like discuss.elastic.co and Stack Overflow. Top contributors from each region will be recognized in March 2022, and guidelines are available for those interested in participating.

Elastic Enterprise Search can be secured using Security Assertion Markup Language (SAML) for single sign-on authentication, demonstrated using Okta as the identity provider. The process involves setting up an application in Okta and configuring SAML settings such as Single Sign-On URL and Audience URI, followed by adjustments in the Elasticsearch.yml configuration file within the Elastic Cloud console. Specific parameters from Okta, including metadata path and entity ID, need to be integrated into the configuration, along with ensuring separate SAML realms if using Kibana. Finally, the setup involves enabling SAML in Enterprise Search through the Elastic Cloud deployment configuration, specifying the auth source and configuration namespace, and restarting the cluster. The guide notes that the SAML configuration detailed is outdated, urging users to refer to the latest documentation, and offers a free 14-day trial for those new to Elastic Cloud.

Cisco developed My Cisco Entitlements (MCE) using Elasticsearch to enhance transparency and management of its IT infrastructure, addressing the complexity that arises as companies scale. By leveraging Elasticsearch, Cisco built a secure and user-friendly platform that offers a holistic view of about 1.5 million service contracts and 2 billion assets, allowing for efficient tracking and optimization of licenses, subscriptions, and other critical assets. The implementation of MCE helps reduce wasteful spending, ensures compliance with license agreements, and provides leverage during negotiations. The platform, supported by Elastic's technology stack including Logstash, enables rapid data ingestion and high-performance indexing, allowing Cisco and its 15,000 partners and internal customers to manage assets effectively, thus minimizing financial, legal, and operational risks.

Aaron Jewitt's blog post explores the creation of a malware analysis sandbox using Elastic Security, highlighting its importance for information security teams in dealing with potential threats such as phishing emails with seemingly benign attachments. The post details the process of setting up a virtual machine (VM) sandbox environment to safely execute and observe malware, leveraging Elastic products to streamline data collection and analysis. It discusses the benefits of dynamic malware analysis, which involves running suspicious software in an isolated environment to gain insights into its behavior, such as process execution, network connections, and file modifications. Jewitt emphasizes the advantages of using Elastic Security for its ease of setup and comprehensive data collection capabilities, including endpoint security and network traffic monitoring. The post also covers practical steps for configuring the sandbox environment and integrating Elastic detection rules to enhance the identification of malicious activities and improve organizational defenses.

Elastic has introduced the Elastic License v2 (ELv2), a simplified and more permissive non-copyleft license for Elasticsearch and Kibana, aiming to minimize disruption and protect against misuse while maintaining openness. The ELv2 allows users to "use, copy, distribute, make available, and prepare derivative works" with three main restrictions: not providing the products as a managed service, not circumventing license key functionality, and not removing licensing or copyright notices. This change reflects Elastic's intent to support its community by addressing concerns and incorporating learnings from other companies like MongoDB and CockroachDB. While the SSPL, a copyleft license created by MongoDB, remains an option for the source code, the license change does not affect Elastic Cloud or self-managed customers. The company emphasizes its commitment to developing new features and supporting its community, aiming for continued success with its products.

Elasticsearch 7.10 introduced enhancements to data lifecycle management by formalizing the configuration of data tiers—hot, warm, and cold—using specific node roles rather than the previous node attributes approach. This change simplifies the process of managing time-series data by allowing data to be automatically migrated between tiers within Index Lifecycle Management (ILM) policies, which utilize the new index.routing.allocation.include._tier_preference setting for specifying tier preferences. Additionally, the update introduced the concept of searchable snapshots, which optimizes storage by allowing one copy of the data to be stored locally while another resides in a snapshot repository, reducing the need for local replicas. The new data_content role caters to non-time-series data, ensuring all types of data are effectively managed within the cluster. The blog post provides practical examples of setting up nodes and configuring ILM policies to leverage these new features, ultimately leading to more efficient data storage and management practices.