November 2023 Summaries

Elastic is focused on enhancing the developer experience for AI application development by simplifying complex workflows and supporting an open ecosystem. Building on its foundation in Apache Lucene, Elastic offers an advanced vector database and search engine with innovations like Elastic's proprietary Learned Sparse EncodeR model (ELSER) for semantic search. The company is addressing challenges in generative AI by partnering with the ecosystem to support transformer and foundation models, reducing costs and processing times for retrieval augmented generation (RAG), and enabling seamless integration with tools like LangChain. Elastic's recent efforts include a streamlined developer experience with simplified commands, automatic mappings, and a serverless architecture, alongside the introduction of new language clients. These advancements aim to make Elasticsearch the leading platform for scalable, cost-effective, and high-performing search and generative AI applications, while maintaining openness and flexibility for developers to adapt to the rapidly evolving AI landscape.

ES Rally is a performance testing tool for Elasticsearch that allows users to run and record comparative tests on different versions or hardware configurations of Elasticsearch clusters. The guide explains how to create custom ES Rally tracks using one's own datasets to evaluate specific scenarios or test changes in data models. It emphasizes the importance of choosing an appropriately sized dataset for accurate metrics and details the process of setting up a custom track, including the installation of ES Rally via container distribution and the use of Elasticsearch Reindex API to prepare data. Once a custom track is created, users can compare different executions to assess the impact of changes, such as data model optimizations, on performance and resource usage. The guide also highlights the potential of ES Rally to assist in decision-making by providing insights into cluster performance and optimization opportunities, with a suggestion to explore further documentation or consult experts for additional use cases.

Generative AI is significantly transforming business operations by enhancing operational efficiency, mitigating security risks, and improving customer experiences. Insights from Elastic's IT leaders—CIO Matt Minetola, CISO Mandy Andress, and Chief Customer Officer Rick Laner—demonstrate how integrating generative AI with proprietary data can bolster operational resilience and AIOps capabilities, streamline security processes, and personalize customer interactions. Minetola emphasizes the importance of a unified data platform for actionable outcomes, while Andress highlights the need for secure AI tools to detect vulnerabilities swiftly. Laner focuses on using AI to meet customer demands for tailored, timely experiences, cautioning against the potential for misinformation and stressing the value of feeding specific data into AI systems. The overarching strategy involves leveraging AI to manage workflows, enhance security, and deliver precise information, thereby fostering trust and driving sales.

Kibana, a UI product for accessing the Elasticsearch database, relies on several key features and plugins such as Telemetry, Rules, Reporting, and Sessions, which can affect its overall performance. The article discusses troubleshooting Kibana's health, particularly focusing on its dependencies, including connectivity to Elasticsearch, access to system indices, and the initialization of the Task Manager and other plugins. The Task Manager, crucial for Kibana's performance, has its own Health API, and issues with it can cascade into other plugins like security and reporting. The text emphasizes the importance of understanding and maintaining Kibana's health by checking its status via Kibana APIs and using JQ commands to parse JSON data and identify problematic plugins or rules. It also highlights the significance of managing Alerting Rules and their potential impact on Task Manager performance, suggesting methods to diagnose and address issues related to rule quantity and quality. The article encourages users to monitor Kibana's health actively and disable unnecessary rules to optimize performance.

Elastic has introduced a new feature in its Elastic Agent that allows for native output to Kafka, enhancing data routing capabilities and offering users increased scalability and flexibility in data streaming and processing. This integration is particularly beneficial in environments with complex data needs, such as IoT, automotive, and manufacturing, where real-time data collection and processing are critical. With Kafka's robust streaming capabilities, users can efficiently manage high-volume data, perform real-time data ingestion and processing, and leverage Elasticsearch for advanced search, analytics, and visualization. The Elastic Agent's integration with Kafka simplifies deployment and management, offering centralized control through Kibana's Fleet, and eliminating the need for multiple Beats and separate configuration files. Additionally, Kafka acts as an intermediary, providing data aggregation, transformation, and optimized Elasticsearch indexing, making it a powerful tool for creating scalable and resilient data pipelines. This integration not only enhances predictive maintenance and operational insights but also bolsters security through Elastic's SIEM and endpoint protection solutions. Elastic and Kafka together offer a unified data integration system that allows for seamless data collection and movement, enabling businesses to transform raw data into actionable insights across various applications and IT architectures.

Elastic and the University of Canberra (UC) have partnered to address the IT industry's skill shortages by offering final-year students Capstone projects that enhance their technical and soft skills using Elastic's software solutions. Since 2022, Elastic has been a technical sponsor, providing access to its Elasticsearch product, which is highly valued in the data-related job market. These projects allow students to tackle real-world problems in fields such as e-commerce, monitoring, and cybersecurity, without the burden of maintaining their own infrastructure, thanks to Elastic Cloud's Software as a Service platform. The collaboration also includes mentorship from Elastic professionals who assist students in developing project management, stakeholder engagement, and technical competencies, ultimately preparing them for successful careers in IT. This partnership not only aims to deliver work-ready graduates but also seeks to cultivate a skilled and diverse community of Elastic users, addressing the growing demands of the industry.

Kathleen DeRusso transitioned from a business major to a computer science major after discovering a passion for solving complex problems in a Java developer course, which led her to a successful career in search technology. She initially worked in a B2B company, developing search solutions for electronic parts using Lucene and SOLR before leading a migration to Elasticsearch, and later shared her work at an ElasticOn conference. In 2020, she joined a SaaS company, leading a predominantly female agile team and mentoring junior members, fostering career growth and team cohesion. Now at Elastic, Kathleen holds a position as a Principal Software Engineer, where she contributes to enhancing search solutions for e-commerce applications, focusing on improving developer experience and implementing AI-enabled search functionalities. Kathleen appreciates Elastic's commitment to diversity and finds value in the strong female representation within her team, encouraging women in tech to overcome imposter syndrome and advocate for their needs.

The integration between Elastic Security and Amazon Security Lake aims to enhance security information and event management by combining Elastic's robust threat detection and response capabilities with Amazon Security Lake's centralized data collection. Elastic is updating its integration to support the Open Cybersecurity Schema Framework (OCSF) versions 1.1 and 1.2, ensuring all security data is actionable within Elastic Security. Amazon Security Lake acts as a centralized repository for security data, simplifying data collection from AWS services and third-party sources, and feeds this into external SIEM solutions like Elastic with minimal operational overhead. This integration enables organizations to analyze their security posture comprehensively, streamline security operations, and automate responses to incidents. While Amazon Security Lake focuses on log storage and basic analysis, Elastic Security provides a more comprehensive security solution, leveraging machine learning and threat intelligence to fortify defenses. The partnership between Elastic and AWS is set to redefine security data management, offering a seamless transition to actionable insights through data integration.

Elastic has announced its acquisition of Opster, a company known for its tools that enhance the monitoring and management of Elasticsearch operations, signaling a strategic move to improve Elastic's cloud offerings. By integrating Opster's AutoOps platform, Elastic aims to provide deeper insights, prevent and resolve issues, improve search performance, and reduce hardware costs for Elasticsearch users. This acquisition will help Elastic deliver a more fully managed and user-friendly cloud service, while also continuing to offer some of Opster's free products to millions of users. Opster's CEO, Ziv Segal, expressed enthusiasm about joining Elastic, emphasizing the potential to advance their shared vision of streamlining search operations and maximizing the value of Elasticsearch deployments. This collaboration is anticipated to enhance the capabilities available to DevOps and engineering teams, fostering more efficient and successful search operations.

Elastic has introduced a new serverless architecture for Elasticsearch, designed to decouple compute from storage, simplify operations, and enhance scalability and performance. This architecture employs inexpensive object storage as the system of record, with distinct search and indexing tiers that allow for independent scaling and optimization. The serverless model eliminates the need for managing clusters and infrastructure, enabling users to focus on data and queries while the platform handles operational tasks. This approach enhances search, observability, and security, offering purpose-built products that streamline user experiences with features such as simplified APIs, machine learning for anomaly detection, and continuous onboarding. The re-engineered architecture supports fast, low-latency queries even with large-scale data, leveraging advanced caching and parallelization techniques to maintain performance. Elastic's serverless offerings promise to simplify complex data workloads, making it easier for users to achieve high-speed searches and efficient data management without operational burdens.

Recent advancements in generative AI have significantly reshaped search capabilities, leading Elastic to integrate its Workplace Search functionality with Elasticsearch for enhanced internal knowledge search experiences. While the standalone Workplace Search product will remain supported, new features and security upgrades will be added to Elasticsearch, which is now recommended for developing flexible and transparent search experiences. This integration leverages Elastic connectors to optimize data ingestion and search management, allowing for more intuitive and customizable search experiences that can incorporate semantic, vector, and AI-driven search methodologies. Elastic's approach emphasizes the use of native connectors and self-managed connector clients, which facilitate seamless integration and indexing of private data into search-optimized Elasticsearch indices, offering a robust framework for building sophisticated search applications.

Kibana has introduced a new feature called the Links panel, released in technical preview with Kibana 8.11, designed to enhance the organization and navigation of dashboards. This feature allows users to create thematically related dashboards and link them together efficiently, addressing previous issues with the Markdown text panel, such as the inability to carry over context between dashboards and broken links when importing or copying dashboards to different spaces. The Links panel enables users to create a customizable navigation experience by allowing the addition of links to dashboards or other URLs with options to list links vertically or horizontally, customize link text, and decide whether to carry over filters and queries. The links can be saved to a library for reuse across multiple dashboards, ensuring a consistent and streamlined user experience. Additionally, the feature supports opening links in new tabs and adding emojis for differentiation and engagement. The Links panel aims to provide a more effective and seamless way to navigate between dashboards, improving user interaction and performance.

Elastic Stack version 7.17.15 has been released, and users are encouraged to upgrade to this latest iteration due to its improvements over previous patch versions within the 7.17.x series. This release addresses a potential security vulnerability, and detailed information about the fixed issues and changes included in this version can be found in the release notes. Users are advised to consult the security advisory for further details on the vulnerability fix.

Elastic Stack version 8.11.1 was released on November 13, 2023, and users are advised to upgrade to this version due to its inclusion of a fix for a potential security vulnerability. The release is recommended over previous patch versions within the 8.11.x series. For more comprehensive information on the specific issues addressed and a detailed list of changes in each product included in this update, users are encouraged to consult the release notes and security advisory.

The Tribal Cybersecurity Grant Program (TCGP), launched by FEMA on September 27, 2023, is designed to bolster the cybersecurity defenses of US tribal governments, who face increasing cyber threats. With a fund of $18.2 million, the program emphasizes tribal sovereignty and resilience, promoting technological defenses aligned with tribal values through the collaboration of CISA and FEMA. Tribal governments must develop robust cyber governance, adapt their systems to emerging threats, and prioritize advanced logging tools to qualify. Applications, due by January 10, 2024, require a Cybersecurity Plan and TCGP Investment Justification form. The program encourages pooling resources for collective projects, while Elastic, a leader in log monitoring, offers comprehensive solutions to manage log data effectively, reducing data breach and ransomware risks significantly.

Nick Keats, a senior technical consultant at Elastic, transitioned from a military career as a combat engineer and paratrooper to a civilian role in the tech industry, with support from the Army's cyber protection training and the Skillbridge program, which aids military personnel in securing civilian jobs. Initially trained in cyber defense, Nick gained experience with Elastic tools like Kibana during his service, which facilitated his move into the tech sector. Despite the challenges of finding a new purpose after leaving the military, Nick found support through Elastic's Mil-asticians ERG, a group for veterans that helped him adapt to civilian life by providing a sense of community and shared experiences. Now, as a full-time technical consultant, Nick uses his skills to engage with clients, solve problems, and teach best practices, drawing on his military experience of teamwork and knowledge sharing.

Nicholas Penning, a seasoned tech industry professional with a strong background in cybersecurity, shares insights into his career and experiences within the Elastic community. He highlights his initial exposure to Elastic's ELK Stack in 2018, which facilitated data ingestion and search capabilities, emphasizing its open-source appeal. A pivotal career moment involved using Elastic to automate a lengthy process, significantly enhancing efficiency and data visualization for threat detection. Nicholas sees potential in generative AI for streamlining cybersecurity reporting, suggesting it could enhance incident response documentation. He praises the Elastic community for its collaborative and supportive nature, where diverse organizations and individuals contribute to problem-solving and feature understanding. For those aspiring to a cybersecurity career, he recommends mastering SIEM systems, scripting, regex, and APIs, which are vital for effective threat management. He encourages involvement in cybersecurity communities to learn and contribute, reflecting on the value such networks could have provided earlier in his career.

Elastic's InfoSec Threat Detection team employs User Entity Behavior Analytics (UEBA) detection packages to enhance the identification of compromised accounts across their systems. These packages consist of detection rules that collectively generate high-fidelity alerts for anomalous user behavior, minimizing false positives. UEBA models leverage machine learning to discern deviations from established norms in user activity, which can be complex and costly but are effective in detecting unusual actions in environments like Slack, GitHub, and other business systems. By implementing building block rules that monitor new terms or activities, Elastic creates threshold alerts to signal significant deviations, thus improving the accuracy of threat detection without overwhelming security analysts with false alarms. The process involves careful planning and tuning to adapt to each system's unique environment, and the integration of these packages into SIEM enables more efficient monitoring and response to potential security breaches.

During a recent team offsite, the Engineering team focused on improving Filebeat's performance by analyzing memory allocation using pprof profiles, particularly during specific logging scenarios. The investigation centered on the event hot-path to identify memory inefficiencies, which revealed that a significant amount of memory was allocated during event processing, specifically due to the logp.newLogger function that instantiated a new logger for each log event, consuming 9.2GB of memory. By removing the unnecessary logger, the team successfully reduced memory usage from 109GB to 99GB for their benchmark, demonstrating a notable optimization. This improvement is part of ongoing efforts by Elastic's Platform Ingest team to enhance performance, with more updates expected in the future.

Elastic Security 8.11 introduces several new features, including piped queries with Elasticsearch Query Language (ES|QL), an Elastic AI Assistant integration with Amazon Bedrock, and enhanced data integrations with Okta, Microsoft Entra ID, Wiz, and Palo Alto Prisma Cloud. These innovations aim to improve the efficiency and effectiveness of threat detection and investigation by allowing security professionals to quickly search, enrich, and transform data. ES|QL enhances the detection engine by reducing alarm fatigue and improving alert relevance, while the AI Assistant now allows users to create queries from natural language prompts. The new integrations with identity management and cloud security technologies provide enriched context and visibility across cloud environments, making Elastic Security 8.11 a comprehensive solution for modern security challenges. Existing Elastic Cloud customers can access these features directly, and those new to Elastic can start with a free trial to explore these advancements.

AI-powered observability solutions are becoming critical for public sector and educational organizations due to the complexity of technology infrastructure, which includes private, public cloud, and air-gapped environments. A Forrester Consulting study, commissioned by Elastic, revealed that Elastic Observability offers significant economic benefits, including a 243% return on investment and $1.2 million in savings over three years, as well as reducing incident resolution time by 85% and increasing data analyst efficiency by 90%. These solutions help overcome challenges such as data silos, inaccessibility, and legacy IT issues by providing consolidated and actionable data insights, improving application performance, and fostering better collaboration and security. For example, organizations have reduced data storage costs by 75% and saved $250,000 annually by transitioning away from legacy systems. As public sector entities face pressure to maximize output with limited resources, AI-powered observability provides a pathway to modernize, enhance, and transform operations while achieving cost savings.

Elasticsearch and LangChain have partnered to develop production-ready RAG (retrieval-augmented generation) templates, enhancing the capabilities of generative AI applications. This collaboration leverages LangChain's LangServe and LangSmith tools, along with Elasticsearch's vector database capabilities, to support companies like Labelbox and Cisco in building scalable AI products. The initiative encourages developers to explore the Elasticsearch RAG template for LangChain and offers resources such as code notebooks and engineering blogs through Search Labs. However, users are advised to exercise caution when using generative AI tools, particularly with sensitive information, as the security and confidentiality of submitted data are not guaranteed.

Elastic 8.11 introduces several key updates across its suite of solutions, notably featuring the technical preview of Elasticsearch Query Language (ES|QL) as a foundational innovation. The release marks the general availability of the Elastic Learned Sparse EncodeR (ELSER) model and includes new connectors for data synchronization, as well as a beta for binary content extraction on edge. In Elastic Observability, enhancements provide unified insights across metrics, logs, and traces, with improvements such as Elastic Universal Profiling™ and updated SLOs. Elastic Security sees the integration of the technical preview of ES|QL, new AI Assistant support for Amazon Bedrock, and beta integrations with Okta and Microsoft Entra ID, among others. The platform also boasts enhancements like the Elasticsearch Relevance Engine™ scaling, simplified data ingestion, and new features in Kibana, available through Elastic Cloud. The availability and timing of these features are subject to Elastic's discretion.

Elastic Observability 8.11 introduces several new features, including ES|QL for Observability, Universal Profiling integration with Elastic APM, and enhanced Service Level Objectives (SLOs). ES|QL, now in technical preview, simplifies data investigations and integrates with the Elastic AI Assistant to create alerts and insights using natural language queries. The integration of Universal Profiling with Elastic APM allows users to correlate application performance issues with underlying system functions without switching contexts, improving visibility and reducing resolution time. Enhanced SLOs feature faster summary calculations, new error rate charts, and the ability to add SLO overviews to dashboards, enabling users to better monitor and manage operational performance. These advancements are available on Elastic Cloud, providing users with a comprehensive suite of tools for improved observability and performance management.

Elastic has introduced the Elasticsearch Query Language (ES|QL), a new piped query language designed to simplify and enhance data investigations by providing advanced search capabilities and concurrent processing, regardless of data source or structure. ES|QL offers a single, unified interface for transforming, enriching, aggregating, and visualizing data, thereby streamlining workflows for security, observability, and search tasks. The language is built from the ground up to be accessible for beginners while powerful for experts, improving speed and efficiency without the need for external systems for specialized processing. By integrating features such as autocomplete syntax, enriched context, and concurrent query handling, ES|QL facilitates quick issue resolution and detailed data analysis, benefiting Elastic's solutions like Security, Observability, and Search. The technical preview of ES|QL marks a significant evolution in Elasticsearch's capabilities, aiming to revolutionize data exploration and manipulation through a simplified, yet powerful, approach.

In a study commissioned by Elastic and conducted by Forrester, organizations utilizing Elastic's AI-powered observability solution reported a 243% return on investment and $1.2 million in savings over three years. The research, which involved interviews with representatives from five different organizations, highlighted significant benefits such as an 85% reduction in time spent on monitoring and incident resolution, a 90% increase in data analyst efficiency, and $2 million in additional profit due to improved customer retention. The study constructed a composite model of an online services organization to demonstrate these impacts, showcasing how adopting a unified full-stack observability platform can enhance visibility, reduce costs, and improve collaboration and customer experience across an organization's tech ecosystem.

Elastic Search 8.11 introduces the general availability of the Elastic Learned Sparse EncodeR Model (ELSER), enabling semantic search for more relevant results based on user intent rather than exact keyword matches. The release also enhances Elasticsearch's capabilities as a vector database with updates to generative AI, improved dense vector mappings, and a more intuitive ingestion process. The platform expands its connector catalog for private data sources like Outlook, Teams, Zoom, and Box, supporting enhanced document-level security and binary content extraction for large files. Additionally, the standalone Workplace Search product enters maintenance mode, with Elasticsearch recommended for internal knowledge search use cases. These updates are available on Elastic Cloud, offering both hosted and self-managed deployment options.

Elastic Stack 8.11 introduces a range of new features, including the Elasticsearch Query Language (ES|QL), which simplifies data exploration and enhances query building with auto-complete and in-app documentation. The release also offers improved data management capabilities, such as the data stream lifecycle for retention and downsampling, and a cross-cluster search response for better search result accuracy. Users can now utilize the ELSER model-2 for AI search, which provides improved relevance and performance. Additionally, the update supports a unified machine learning inference API and offers enhancements in data visualization with color mapping and Lens in-line editing. Other notable features include the general availability of the reroute processor, support for dense and sparse vector searches, and the addition of a Persian language stemmer. These advancements aim to streamline data processing, enhance visualization, and improve search capabilities within the Elastic ecosystem.

ES|QL, or Elasticsearch Query Language, is Elastic’s newly developed piped query language designed to enhance data analysis and investigation by providing powerful computing and aggregation capabilities. It simplifies the process of searching, aggregating, and visualizing large datasets, offering features like lookups and real-time processing from a single interface in Discover. ES|QL's new query engine delivers advanced search capabilities with concurrent processing, enhancing speed and efficiency across various data sources. It caters to diverse users, including site reliability engineers, DevOps, and threat hunters, by enabling efficient system monitoring, deployment assessment, and security threat detection. The language's intuitive interface supports both beginners and experts, with features like auto-complete and in-app documentation that streamline crafting advanced queries into straightforward workflows. ES|QL's integration with Dashboards and Alerting functionalities offers a cohesive data exploration experience, allowing users to create visualizations and alerts directly from their queries. This innovation reflects Elastic’s commitment to enhancing user experiences in data exploration and addressing the evolving needs of its community.

Handling duplicate log entries from noisy applications is a significant challenge for SREs, as these duplicates overwhelm centralized logging platforms, consume storage, and induce alert fatigue. To address this, the blog discusses techniques for log deduplication using various Elastic Stack tools, including Logstash, Beats, and Elastic Agent. Elasticsearch automatically generates a unique ID for each document unless specified otherwise, but duplicate entries can still occur due to retry mechanisms or misconfigurations. The article explores different strategies, such as using unique IDs, hashing with fingerprint processors, and event aggregation, to manage duplicates effectively. Each method has its trade-offs, such as processing overhead or potential data obfuscation, and the choice depends on specific use cases and performance considerations. The blog also highlights the importance of carefully selecting attributes for deduplication to balance between reducing noise and retaining critical alerts.

The US Navy's Capstone Design Concept for Information Superiority aims to enhance digital modernization by securely transferring information to achieve operational resilience and improved customer experience. The recommended approach to achieving these goals involves adopting a Global Data Mesh strategy, which allows data to be accessed and searched without moving it, thus minimizing risks and costs associated with data duplication and transfer. This distributed method, facilitated by the Elasticsearch platform, supports the Navy's objectives by enabling more efficient data management in Denied, Degraded, Intermittent, and Limited (DDIL) environments, fostering data ownership, democratization, and accuracy. Additionally, the use of Elasticsearch's Cross-Cluster Search and Cross-Cluster Replication capabilities bolsters operational resilience by distributing workloads and maintaining data access during system failures. By using a unified data platform, the Navy can reduce silos, enhance collaboration, and better utilize data for strategic mission tasks, ultimately transforming data into a powerful asset in an information-driven landscape.

Elastic Cloud offers a 14-day trial that enables customers to deploy, manage, and scale Elasticsearch clusters and other components of the Elastic Stack across AWS, GCP, and Azure without handling the underlying infrastructure. The trial allows customers to explore Elastic's Search, Observability, and Security solutions, supported by resources such as set-up guides, sample data, and a live demo environment. Customers can also access free training and support from the Customer Engineering team, which assists with technical onboarding and deployment best practices. An extension for the trial can be requested if more time is needed for testing, and the trial can be converted to a paid account for additional features. Elastic Community forums and training videos further support users in utilizing the platform effectively.