September 2023 Summaries

In Filebeat versions 8.10.0 and 7.17.12, a new fingerprint mode has been introduced for Filestream inputs, allowing users to identify files using a SHA256 hash of their content instead of relying on file system metadata. This change addresses issues with unstable file identifiers that can lead to data loss or duplication, particularly in environments where file systems cache and reuse inodes. The fingerprint mode offers a unique file identifier by hashing a configurable byte range of the file, providing a more stable solution compared to using device_id and inode values. While the new mode incurs a performance hit, optimizations made during its implementation have resulted in an overall performance improvement in the new Filebeat release, making Filestream faster even with the fingerprint mode enabled. This advancement highlights the ongoing focus on enhancing Filebeat's performance and reliability in handling file system events and data ingestion.

Elastic has introduced a new Verified Managed Services Provider (MSP) designation to enhance its partner program, emphasizing quality, reliability, and support. This designation is aimed at helping MSP partners differentiate themselves by undergoing a thorough review process that ensures they possess the necessary architecture, best practices, and resources to manage Elastic solutions at high standards. The program offers benefits such as guided support, service co-creation workshops, and go-to-market assistance, alongside a "Verified MSP" badge and priority event sponsorship. These enhancements are designed to provide customers with increased confidence in service delivery and to highlight the most competent MSPs within the Elastic ecosystem. Existing verified partners include Kyndryl, Accenture, Cyberfort Group, Malcrove, ECS, Expedient, Proficio, and NEC Security. Potential partners interested in becoming verified must fulfill criteria including having two Elastic certified engineers, a reference architecture, and a list of third-party vendors.

Elastic Cloud offers a security feature known as traffic filtering, which provides network layer security by restricting access to deployments from specified networks through the use of IP or CIDR-based rules. This is combined with role-based access control (RBAC) to enhance security measures. The service supports integration with major cloud providers such as AWS, Azure, and Google Cloud, allowing for the implementation of traffic filters that leverage AWS PrivateLink for private connectivity. This ensures that traffic remains within the AWS network, avoiding exposure to the public internet. Customers can configure multiple rule sets for a single deployment, associating them with specific IP addresses or AWS VPC endpoints to control access. The AWS PrivateLink traffic filter is available at no additional cost and is implemented by creating a VPC endpoint and DNS records, followed by configuring and applying the traffic filter rules within the Elastic Cloud console. This setup provides a secure way to manage network access to Elastic deployments, and users can monitor and analyze traffic patterns using Elastic VPC and Route53 integrations for enhanced visibility.

Version 8.10.2 of the Elastic Stack has been released, addressing a significant issue with Cross-cluster search (CCS) that affected users after upgrading to versions 8.10.0 or 8.10.1. This problem caused Kibana to incorrectly display a "Received partial response" error in various applications like Discover and Security when CCS requests returned partial results. Users are encouraged to upgrade to this latest version for improved performance and to avoid these errors. For more detailed information on the fixes and changes included in this release, users can refer to the official release notes.

Elastic Security participated in the MITRE Engenuity ATT&CK Evaluations: Enterprise 2023, showcasing its capabilities in detecting and preventing memory-based and kernel-level attacks, particularly those associated with the Turla malware family, a sophisticated Russian advanced persistent threat group. The evaluations, which focus on real-world threat scenarios, allowed Elastic to demonstrate its detection and prevention technologies, including behavior-based alerts powered by machine learning and enhanced visibility across Windows, macOS, and Linux operating systems. Elastic's security solutions were evaluated for their effectiveness in both detection and prevention, with the latter allowing vendors to activate prevention technologies. Elastic effectively stopped attacks in most tests by preventing malware execution at the initial stages, highlighting its integration of core protection technology and pre-built security rules. Since the evaluation, Elastic Security has released new features, including enhanced attack prevention and detection capabilities, a generative AI assistant for investigations, and additional rules and protections to address a wider range of threats.

Elastic's AI Assistant for Observability, powered by the Elasticsearch Relevance Engine, is designed to enhance the analysis and resolution of observability issues by providing context-aware insights and eliminating the need for manual data retrieval across silos. This interactive tool, currently in technical preview, helps Site Reliability Engineers (SREs) better understand application errors, log messages, and alerts, offering suggestions for code efficiency and enabling real-time log spike analysis. The AI Assistant integrates with large language models like OpenAI and Azure OpenAI, allowing users to input private data such as runbooks and incident histories for more personalized support. It features a chat interface that facilitates natural language interactions, enabling users to query and visualize relevant telemetry data, conduct root cause analysis, and execute API functions. This tool is accessible to Elastic Observability 8.10 users with an Enterprise license and aims to improve proactive issue resolution by leveraging machine learning capabilities.

Elastic Observability is transforming the field by integrating AI and generative AI technologies, OpenTelemetry standardization, continuous profiling, and enhanced log analytics to deliver more proactive and intelligent solutions. Leveraging its machine learning expertise, Elastic has introduced the Elastic AI Assistant, now in technical preview, which offers context-aware insights and suggestions for resolving application errors and optimizing code efficiency. The company is strengthening its commitment to OpenTelemetry, providing native support, and contributing to its establishment as an industry standard to facilitate open data ingestion and improve operational efficiency. Additionally, Elastic's Universal Profiling tool aids in optimizing computational efficiency by identifying performance bottlenecks without instrumentation, while enhanced log analytics improve log categorization and analysis through AI. By unifying metrics, logs, traces, and profiling in a single platform, Elastic aims to help Site Reliability Engineering (SRE) teams manage complex systems more effectively, reduce costs, and achieve sustainable growth.

In the context of modern software development, effective log management is crucial for maintaining application performance and reliability. Logs provide detailed information that is essential for debugging, performance analysis, security, and compliance. The logging journey encompasses collecting and ingesting logs, processing and enriching them, and analyzing and rationalizing the data. Structured logs are generally preferred for their ease of use, but unstructured logs can still be managed effectively through full-text search capabilities. Machine learning can enhance log analysis by identifying patterns and anomalies that might not be visible through manual methods. Traces, which provide deeper insights into transactions, are highlighted as a complementary tool to traditional logs, especially in cloud-native environments. To improve operational efficiency, organizations are encouraged to reduce time spent on data transformation, decide between centralized or decentralized log storage, and implement a log deletion policy to manage large volumes of data. This comprehensive approach ensures the reliability, performance, and security of applications and infrastructure in a complex and rapidly evolving software landscape.

Version 8.10.1 of the Elastic Stack has been released, and users are encouraged to upgrade to this latest version over previous patch versions in the 8.10.x series. This update addresses security issues, and users are advised to refer to the security announcements for more detailed information. For a comprehensive list of fixed issues and changes introduced in this version for each product, the release notes provide detailed insights.

Role-based access control (RBAC) is an essential security feature in Elastic Cloud that allows organizations to assign specific roles to users and API keys, enabling fine-grained control over access to cloud resources. This system helps to manage permissions by assigning roles that dictate access levels and privileges, ensuring users can only access the resources necessary for their roles, thereby reducing the risk of unauthorized access. Within Elastic Cloud, roles can be applied at both organization-wide and deployment-specific levels, with various standard roles available such as Organization owner, Billing admin, Admin, Editor, and Viewer, each providing differing levels of access and control. Managing these roles is straightforward, with options to edit user and API key roles directly from the Elastic Cloud console, allowing for immediate implementation of changes.

Elastic Cloud has expanded its availability to Amazon Web Services (AWS) in the Europe North (Stockholm) region, allowing users to deploy Elastic's Search, Observability, and Security solutions with increased flexibility and reduced operational overhead. These solutions enhance search experiences, monitor applications, and secure technology investments by integrating endpoint security with event monitoring. Users can select their preferred cloud provider, whether AWS, Google Cloud, or Microsoft Azure, and tailor deployments according to specific compute, memory, or I/O needs. Elastic Cloud offers a managed service or self-management option with automation tools, and users can subscribe through the AWS Marketplace with the possibility of a 7-day free trial. The announcement underscores Elastic’s commitment to scalable, real-time solutions while also noting that future feature releases are subject to change at the company's discretion.

Elasticsearch 8.10 introduces query rules, a feature that allows for the customization of search results based on query terms or contextual information. This innovation enables users to tailor search outcomes for marketing campaigns, personalization, or segment-specific results by implementing pinned query rules, which prioritize certain documents in search results based on defined criteria. Supported criteria for matching include exact, fuzzy, prefix, and suffix matches, among others. Administrators can create and manage these query rules via the query rules management API, storing them within Elasticsearch and applying them during searches to rewrite queries into pinned queries, which then prioritize specified documents. An example use case involves a global e-commerce site promoting different versions of a wireless charger based on the user's location, ensuring the most relevant product is featured prominently in search results. This capability is part of the broader suite of features in Elastic 8.10, offering users enhanced control over search result customization.

Elastic Cloud has expanded its availability to two new Google Cloud regions, Europe West 9 (Paris) and Asia Pacific Southeast 2 (Jakarta), enhancing its ability to provide scalable solutions for search, observability, and security. These expansions allow customers to deploy Elastic Cloud on Google Cloud, Microsoft Azure, Amazon Web Services, or a combination of these, offering flexibility in deployment preferences. Elastic Cloud offers cloud-native features that help reduce operational overhead by managing maintenance, upgrades, and security patches, while providing options for managed services or self-management through built-in automation tools. Customers can choose specific regions and cloud service providers to optimize their deployments based on their individual requirements. Additionally, Google Cloud customers can subscribe via the Google Cloud Marketplace, integrating Elastic costs into their consolidated billing and drawing from cloud purchase commitments. Elastic retains discretion over the release timing of features, which may be subject to change.

Bhavya Gupta, an Elastic Community member and an Elastic Engineer at Jolera, shares her journey into the IT industry, sparked by her fascination with technology's transformative power since high school. Her interest in Elasticsearch began during college when she explored efficient searching technologies, leading her to become a top contributor on Stack Overflow for Elasticsearch queries. Despite the traditionally male-dominated field, Bhavya draws inspiration from pioneering women in tech and remains undeterred by potential biases. A significant milestone in her career was becoming a Silver Contributor in the Elastic Community, passing the Elastic Certified Engineer Exam on her first attempt, and securing a job offer due to her expertise. Her involvement in the Elastic Contributor Program provided hands-on experience and valuable connections within the community. Bhavya praises the supportive nature of the Elastic Community, highlighting its comprehensive documentation and the opportunities for learning and growth offered by the program.

Elastic Stack 8.10 introduces significant enhancements to cross-cluster and vector search, data ingestion, Kibana, and cloud registration, aiming to improve user experiences and streamline operations. Key updates include faster vector search capabilities with increased parallelization, general availability of kNN vector search for up to 2048 dimensions, and improved cross-cluster search (CCS) functionality with more detailed status information. Kibana now offers features such as data drift detection through the Data Comparison view and log rate analysis for both spikes and dips, alongside enhanced multivalue filtering options for charts. Security improvements include a new model for cross-cluster connections using API-based authentication and better integration capabilities through a client certificate-supported Webhook connector. Additionally, Elastic Agent now supports Kafka for streaming application data, and a new synonyms API simplifies synonym management within clusters. These features, available on the Elastic Cloud, aim to enhance data analysis, integration, and security for Elastic users.

Elastic Observability 8.10 has been released, bringing the general availability of Elastic Universal Profiling™ and enhancements to the Elastic AI Assistant. Elastic Universal Profiling offers a comprehensive, always-on profiling solution that requires no code changes, operating within the Linux kernel to provide detailed visibility across all system components. It helps improve computational efficiency, reduce cloud costs, and support environmental goals by profiling every line of code without disrupting workloads. Meanwhile, the Elastic AI Assistant has been enhanced with a new chat feature that allows users to interact using natural language, recall information, and access a knowledge base for additional context and recommendations. This feature, currently in technical preview, leverages Elastic Learned Sparse EncodeR (ELSER) and external LLMs (OpenAI or Azure OpenAI) to assist with problem analysis and provide insights across various observability aspects. The Elastic AI Assistant is an enterprise-licensed feature, and feedback from users is encouraged to refine its capabilities. Elastic Observability 8.10 is available on Elastic Cloud, with options for self-managed deployment also provided.

Elastic Universal Profiling is a newly launched continuous profiling solution aimed at enhancing software efficiency and sustainability. This tool operates within the Linux kernel space using eBPF, capturing essential data without disrupting operations or requiring code modifications. By providing detailed visibility into every line of code, including application, kernel, and third-party libraries, it enables organizations to optimize CPU utilization, reduce CO2 emissions, and lower cloud costs. Elastic Universal Profiling supports a range of programming languages and is designed for SREs, developers, and FinOps professionals, offering insights into performance optimization, cost management, and environmental sustainability. This approach aligns with the growing need to certify software as energy efficient, addressing the lack of reliable methods for measuring the carbon footprint of software code.

Elastic Search 8.10 introduces advanced personalization capabilities and an expanded connector catalog, enhancing its search functionalities. The new version allows users to customize search results with tailored query rules, offering more control over the display of documents based on defined criteria and contextual information such as geolocation and demographics. The introduction of a Synonyms API simplifies the process of managing synonyms by enabling programmatic updates without interrupting service. Additionally, the connector catalog is expanded to support a variety of knowledge and communication systems, including Google Drive, Salesforce, OneDrive, GitHub, Gmail, and Slack, facilitating direct data ingestion and enhancing unified search experiences. These features, available on Elastic Cloud and for self-managed setups, aim to improve the efficiency and accuracy of search experiences across different platforms and data sources.

Elastic Security 8.10 introduces several enhancements aimed at improving detection and response capabilities, including a MITRE ATT&CK-aligned detection coverage view and richer alert contextualization to accelerate investigations. The release extends cloud security posture management (CSPM) to Google Cloud Platform, offering a comprehensive view across multi-cloud environments, while automating CSPM onboarding for AWS accounts to simplify cloud infrastructure management. The integration of Elastic AI Assistant, now in general availability, assists analysts by synthesizing alert details and recommending investigative actions. Analysts benefit from new visualizations and an expanded Alert sidebar that provides detailed insights, such as user and host risk scoring and threat intelligence correlations, aiding in faster and more effective response to potential threats.

Elastic 8.10 introduces several new features across its platform, enhancing capabilities in observability, security, and search. The release includes the general availability of Elastic AI Assistant for Security, improved cross-cluster search functionality, and the tech preview of Elastic AI Assistant for Observability, which leverages ESRE to provide actionable IT insights. Elastic Security has been upgraded to better unify SIEM, endpoint security, and cloud security, while Elastic Search introduces new APIs and connectors to enhance modern search experiences. The Elasticsearch Platform, the foundation for all Elastic solutions, now supports 2048 dimensions in dense vector search and offers enhanced cross-cluster search features. These updates, available on Elastic Cloud, aim to provide users with comprehensive tools for managing and securing their digital ecosystems.

Apache Lucene has introduced a significant improvement in query performance by combining two existing optimizations, BS1 and MAXSCORE, for evaluating disjunctive queries, which involve terms connected by "OR". Previously, BS1 was used for exhaustive evaluation, while MAXSCORE and WAND were employed to compute top-k hits, but these approaches were not used together. BS1 reduces overhead by processing document IDs in fixed-size windows and iterating through matching IDs, while MAXSCORE skips over unnecessary hits by predicting upper score bounds. By integrating BS1's efficient processing with MAXSCORE's selective evaluation, the new approach outperforms both exhaustive evaluations and existing top-k methods, particularly in scenarios with high-frequency terms or many clauses. This advancement is expected to be included in Lucene 9.8 and subsequently in Elasticsearch, enhancing query performance for complex searches.

Version 7.17.13 of the Elastic Stack has been released, offering important updates and fixes for potential security vulnerabilities. Users are advised to upgrade to this latest version over the previous 7.17.12 and 7.16.x releases to benefit from these improvements. Detailed information about the fixed issues and complete changes for each product in this version can be found in the release notes, and a security advisory is available for more specifics on the addressed vulnerabilities.

Version 8.9.2 of the Elastic Stack has been released, prompting users to upgrade due to its enhanced security measures over previous versions, such as 8.9.1 and 8.8.x. This patch release addresses potential security vulnerabilities, and further details can be found in the associated security advisory and release notes, which provide a comprehensive list of fixed issues and changes in each product included in the update.

Kibana's Discover UI, a core feature of Elastic, enables users to search, filter, and inspect time series data, but it can encounter issues such as long load times, timeouts, and errors due to factors like uncompressed response sizes, mapping explosions, and browser limitations. This troubleshooting guide addresses common problems, such as page errors on load, missing data views, mapping explosions, field conflicts, slow Elasticsearch queries, and rendering issues influenced by mapping explosions. It offers a structured approach to resolving these issues, including checking browser network tabs, using HAR logs, and adjusting settings in Kibana and Elasticsearch. The guide emphasizes the importance of maintaining good data hygiene to prevent performance issues and highlights that the release of new features is at Elastic's discretion.

Asuka Nakajima's journey from aspiring hacker to a security research engineer at Elastic is driven by her early fascination with cybersecurity, inspired by the book Project SEVEN. Her career path included studying programming and security at university, interning with various companies, and participating in security competitions. At Elastic, she focuses on developing the Endpoint Detection and Response (EDR) feature for Windows within Elastic Defend, writing test code, performing code reviews, and keeping abreast of the latest cyber threats. Beyond her professional work, Asuka founded CTF for GIRLS, a group that hosts workshops and lectures designed to address the gender imbalance in the cybersecurity field by providing a supportive community for women to learn and engage in various security disciplines. With more than 1,200 women having attended these workshops, Asuka believes that creating spaces for women in tech is crucial in overcoming the social and psychological barriers that contribute to gender disparity in the industry. Her advice to women interested in tech is to pursue their interests regardless of societal expectations, emphasizing the importance of seeking opportunities rather than focusing on limitations.

Governments worldwide are grappling with a range of technological challenges in the twenty-first century, including aging IT infrastructure, cybersecurity threats, AI integration, data privacy, budgetary constraints, and evolving geopolitical landscapes. Many government systems are burdened by costly and insecure legacy IT systems, and the potential of data remains underutilized due to a lack of unified strategies and cultural shifts. Budget limitations hinder the ability to keep pace with technological advancements and citizen demands for efficient online services. Cybersecurity is a primary concern due to increasing cybercrime, yet a global shortage of cybersecurity professionals complicates defense efforts, highlighting the potential role of AI in addressing this gap. The integration of AI into government functions requires careful management to avoid ethical pitfalls and societal disruption, while protecting citizens' data privacy is crucial in maintaining public trust. These interconnected challenges suggest that collaboration and knowledge sharing among global governments could lead to more effective and secure governance models suitable for modern demands.