August 2024 Summaries

Perry Seale, an Alliance Lead at Elastic, exemplifies the productivity potential of remote work, leveraging technology and a flexible company culture to work effectively from various global locations. Since joining Elastic in May 2024, Perry has continued his decade-long practice of working remotely, prioritizing connectivity and adapting his schedule to align with his clients' time zones. He emphasizes the importance of tools like a second monitor and an AI transcription service to maintain efficiency while on the move. Perry's role involves managing partnerships and necessitates frequent travel, which he combines with a personal goal of experiencing new places each month. Elastic's supportive environment allows Perry to thrive in diverse settings, fostering a sense of community among remote colleagues and promoting cultural appreciation. This approach not only enhances his professional relationships but also enriches his personal life, demonstrating the benefits of a distributed work model that encourages employees to balance work with life experiences.

The Digital Operational Resilience Act (DORA) is a groundbreaking EU regulation set to take effect in January 2025, aiming to transform cybersecurity and operational risk management in the financial sector. Its comprehensive and obligatory approach makes it a benchmark for global cybersecurity standards, extending beyond EU borders to impact international third-party ICT vendors, such as cloud platforms and SaaS providers. DORA integrates cybersecurity requirements across financial institutions and their supply chains, equating cybersecurity risks with financial risks, which contrasts with the more flexible, voluntary nature of many US regulations. The regulation emphasizes operational resilience, a priority highlighted by the COVID-19 pandemic, as financial entities worldwide faced digital operational challenges. Compliance with DORA demands holistic visibility and collaboration among security, compliance, operations, and IT teams to ensure seamless operations and adherence to security standards. This necessitates advanced tools like the Elastic AI Assistant, which aids in responding to security events and enhancing productivity, underscoring the need for financial institutions to innovate and meet evolving customer and regulatory expectations.

Elasticsearch and Kibana are once again being classified as open-source projects with the addition of the AGPL license, alongside existing ELv2 and SSPL licenses, marking a return to the open-source community ethos that Elastic founder Shay Banon emphasizes is integral to the company's identity. The decision to change the license three years ago was driven by issues with Amazon Web Services (AWS) and market confusion, leading to a fork of Elasticsearch. Now, with a stronger partnership with AWS and reduced market confusion, Elastic is reintroducing an open-source license to provide more options to users who value different licensing models. Banon highlights that this move is part of a long-term strategic vision to foster more open-source collaboration globally, and reassures users that while the licensing has expanded, the core offerings and innovations of Elasticsearch, such as advancements in GenAI and security features, continue to thrive.

The blog post outlines a detailed guide on setting up Okta SAML login for Kibana on Elastic Cloud, focusing on integrating Okta as an Identity Provider for single sign-on (SSO) authentication using SAML 2.0. It begins with preparing an Elastic Cloud deployment by creating an instance with Elasticsearch and Kibana, followed by configuring the SAML settings with placeholders to be filled during the Okta setup. The process proceeds with creating a SAML application in Okta, where specific settings from the Elastic deployment are used to configure the application, including audience URI and single sign-on URL. The guide explains how to connect Okta to Elastic by replacing placeholders in the SAML settings with Okta-generated values and mapping Okta groups to Elastic roles to manage user permissions in Kibana. It concludes with testing the SSO integration to ensure successful login via Okta and provides additional resources for troubleshooting and further learning.

Elastic has successfully achieved the AWS Financial Services Competency, marking its fourth AWS competency, after undergoing a rigorous validation process that included a technical audit and review of customer case studies. This recognition highlights Elastic's ability to provide cloud solutions that enhance efficiency, productivity, and innovation for financial sector companies. With the growing data volume in financial services, Elastic's Search AI Platform offers solutions to transform data into strategic opportunities, aiding organizations in customer retention, productivity enhancement, and risk minimization. Elastic's expertise and technical capabilities are validated by AWS, supporting financial institutions in their digital transformation journeys while ensuring compliance with regulations and providing scalable, secure solutions. This achievement complements Elastic's prior AWS competencies in Generative AI, Security, and Data and Analytics, further establishing its leadership in aiding financial firms to innovate and unlock business value.

Craig Abbott, a design manager at Elastic, discusses his experiences with ADHD and the importance of accessibility in design. Diagnosed three years ago, he realized that his existing coping strategies, such as using a paper notebook, fidget toys, and the Pomodoro technique, were essential for maintaining focus. Craig emphasizes that accessibility benefits everyone, not just those with disabilities, by improving overall user experiences. He advocates for iterative accessibility practices, highlighting that efforts to include marginalized groups enhance design for all users. Craig also underscores the importance of flexible working conditions and structured communication to accommodate neurodivergent individuals. His ongoing involvement in accessibility, despite no longer holding a full-time role in the field, reflects his commitment to fostering a culture of inclusivity and continuous improvement.

Elastic AI Assistant and Attack Discovery have integrated with Google Vertex AI to enhance AI-driven security analytics, which aim to alleviate the manual workload on security analysts. These innovations utilize generative AI to streamline the identification and prioritization of security alerts and threats, reducing analyst burnout by enabling faster and more accurate threat detection and remediation. The collaboration with Google Vertex AI's Gemini 1.5 models allows Elastic's tools to deliver advanced reasoning and contextual understanding, significantly improving threat analysis and response. This integration empowers security professionals to concentrate on threat assessment by automating the triage and interpretation of complex attack patterns. Elastic continues to advance its AI capabilities through ongoing partnerships and innovations, ensuring its security solutions remain agile and effective against evolving cyber threats.

In the August 2024 edition of Elastic's DevRel newsletter, the Elastic DevRel team highlights several updates and features in Elasticsearch and the Elastic Stack 8.15, including new capabilities like semantic text processing, enhanced logging data streams, and improved memory efficiency through int4 quantization. The newsletter also introduces the general availability of Elasticsearch's native Learning to Rank feature, which enhances search relevance using machine learning models. Additionally, enhancements in vector databases, such as bit vector support and faster multi-segment HNSW graph search, are detailed alongside the introduction of field statistics in ES|QL for better data insights. The newsletter also outlines upcoming Elastic meetups and events across various regions, including virtual sessions, ElasticON Tour conferences, and calls for local community participation. Furthermore, it covers advancements in stateless Elasticsearch architecture, autoscaling capabilities, and data storage optimizations, as well as collaborations with third-party AI model providers like OpenAI and Google AI Studio.

Balancing the management framework's three pillars—people, process, and technology—is essential for successful security operations, with the human element being crucial to defending against cyber threats. Despite advancements in AI-driven security analytics that enhance the efficiency and stress management of cybersecurity practitioners, the demanding nature of their work often results in burnout, which can compromise both individual well-being and organizational security. To address this, security leaders are encouraged to utilize tools like the Security Analyst Burnout Quiz to assess their team's well-being and take proactive measures such as adjusting workloads and implementing wellness programs. By ensuring the well-being of security teams, organizations can improve morale, retain top talent, and maintain a strong security posture. AI-driven security analytics further supports these efforts by automating tasks, enhancing threat detection, and improving alert accuracy, thus contributing to a more sustainable work environment.

Elastic Cloud's integration with Azure Key Vault allows users to enhance the security of their cloud deployments by implementing encryption at rest through a "bring your own key" (BYOK) approach. This method involves creating an RSA asymmetric key within Azure Key Vault and configuring it alongside Elastic Cloud deployments to encrypt both data and snapshots. Key management features like rotation and revocation are supported, reducing the risk of data breaches and ensuring secure access control. The process requires an Enterprise subscription and involves setting up proper Azure IAM policies to grant Elastic Cloud access to the encryption keys. Additionally, users can monitor the encryption status via the Elastic Cloud Console and perform key rotations or revocations as needed to maintain security. The blog provides a detailed walkthrough of the setup, including key creation, deployment integration, and verification processes, encouraging users to try the feature with a free trial and mentioning future plans to cover encryption with GCP KMS keys.

In the context of vector search, the algorithms approximate nearest neighbor (aNN) and K-nearest neighbor (kNN) play pivotal roles in enhancing data search and retrieval capabilities by converting complex data into vectors, allowing for nuanced and context-aware searches. While aNN is recognized for its speed and efficiency in handling large data sets by approximating results, kNN is noted for its accuracy by identifying the 'k' closest neighbors, essential for precise applications such as medical diagnoses or financial forecasting. Both algorithms are integral to various real-world applications, such as search engines, recommendation systems, and visual search functionalities, each offering unique advantages; aNN excels in situations requiring rapid response and scalability, whereas kNN is more suited for tasks where precision is critical. Elastic's platform leverages these algorithms to provide advanced search solutions, facilitating efficient management and retrieval of large data sets, thereby enhancing user experience and system performance across diverse applications.

Philipp Kahr and Grzegorz Banasiak discuss the use of Rally for benchmarking Elasticsearch, emphasizing the importance of defining clear goals before conducting tests. Rally should not be run on production clusters due to its potential for destructive actions and data loss. The authors provide a detailed analysis of an ingest-only benchmark test run on three Elasticsearch nodes, outlining various metrics such as store size, throughput, and percentile latencies. They highlight the importance of focusing on ingest-related metrics, caution against using maximum throughput as a reliable measure, and underscore the distinction between latency and service time. The blog also delves into the interpretation of telemetry data and the relevance of different metrics depending on the use case, while advising on optimizations and considerations for accurate results. The post concludes with a look at the potential implications for cluster performance and the importance of examining CPU and other system usages in future analyses.

Tencent has contributed to optimizing Apache Lucene's caching system by addressing lock contention issues through the implementation of read-write locks (RWLock). This enhancement allows multiple threads to read from the cache simultaneously while maintaining data integrity during write operations, significantly reducing lock contention and improving performance by 50% to 200% depending on query complexity. This improvement, now integrated into Lucene's main branch, is expected to enhance query performance across various applications, including vector retrieval, BM25 scoring, and range queries, particularly benefiting real-time applications. Tencent's contribution exemplifies the collaborative spirit of the Elastic community and underscores the growing role of Elasticsearch within the Chinese development ecosystem, particularly in scenarios involving generative AI.

IBM has partnered with Elasticsearch to enhance its watsonx Assistant with a new Conversational Search feature by integrating retrieval augmented generation (RAG) capabilities. This integration allows businesses to build conversational AI assistants that leverage their proprietary data, offering enriched chat experiences with business context. The collaboration also involves the use of IBM watsonx Discovery, which integrates with the Elasticsearch vector database to provide semantic, federated, and vector search across various data types and enterprise sources. This partnership enables developers to implement advanced search techniques, such as k-nearest neighbors (kNN) and approximate nearest neighbor (ANN) searches, using popular natural language processing models. Furthermore, Elasticsearch's optimizations have significantly improved efficiency, making it a valuable tool for AI-enabled applications.

Artificial intelligence (AI) is increasingly being implemented in the public sector to enhance operational efficiency, reduce costs, and improve service delivery to citizens, with notable examples from the UK and US where AI is employed to tackle issues like fraud reduction and asylum processing. While AI holds immense potential for productivity gains and economic benefits, such as the projected £200 billion annual savings in the UK and $519 billion productivity boost in the US by 2033, its adoption faces hurdles due to privacy concerns and outdated IT infrastructure. To overcome these barriers, strategies like the "data mesh" approach and retrieval augmented generation (RAG) workflows are being utilized to integrate proprietary data with generative AI securely, enhancing the accuracy of AI outputs without compromising data privacy. Successful implementations, such as Georgia State University's use of generative AI to streamline financial aid processes, highlight the potential of AI to transform public services, though it requires a strong data foundation and a privacy-first approach to ensure responsible use.

Elastic has been recognized as a Leader in the 2024 Gartner Magic Quadrant for Observability Platforms, highlighting its capabilities in managing the increasing complexity and data growth that organizations face. Elastic Observability, powered by Search AI, offers a comprehensive solution to prevent outages and enhance operational efficiency by enabling the retention and analysis of extensive telemetry data, thus facilitating faster root cause analysis via machine learning and generative AI. The platform supports open and extensible observability through its integration with OpenTelemetry, allowing organizations to avoid vendor lock-in while ensuring data privacy and cost efficiency. Elastic's approach includes a unified data store and high-performance data tiers that provide a single pane of glass experience, aiming to eliminate monitoring blind spots and reduce costs associated with data retention and custom metadata. As the observability landscape evolves with AI advancements, Elastic positions itself to meet future demands by offering a flexible cloud or on-premise deployment model while continually enhancing its platform's features.

Sunayana Vatassery returned to Elastic after a four-year absence, transitioning from a successful sales career to a role in product marketing, which she describes as a move back to a company that feels like home. Initially joining Elastic as a sales director, she left to pursue opportunities with smaller private companies, eventually returning in 2024 to a redefined senior product marketing manager role in go-to-market (GTM) Search. This role leverages her extensive sales experience and focuses on promoting Elastic's capabilities in generative AI and vector databases. Sunayana's work involves simplifying complex information for various stakeholders and ensuring the field team is equipped to enhance customer experiences. She emphasizes the importance of continuous learning and collaboration, particularly in navigating the transition from a sales-driven role to one centered on collaboration and strategic marketing. Her experience highlights the adaptability required in career shifts and the support she found within Elastic's consistent company culture, making it a conducive environment for both personal and professional growth.

Elastic Security 8.15 introduces several new features aimed at modernizing security operations through AI-driven analytics, including Automatic Import for streamlining data ingestion and onboarding, and support for Google’s Gemini 1.5 Pro and Flash large language models (LLMs). The update also includes new APIs for the Elastic AI Assistant, enabling sophisticated automation and orchestration strategies, and an on-demand file scanning feature with Elastic Defend integration for enhanced security and compliance. Additionally, the release offers improved investigative context pivoting, allowing users to navigate between different analyses without disrupting their workflow. Available on Elastic Cloud, this version supports a seamless user experience, with options for both cloud and self-managed deployments, providing tools for more efficient threat detection and data management.

Elastic Security 8.15 introduces new chat and management APIs for Elastic AI Assistant, enhancing security operations by automating interactions, managing conversations, and supporting data privacy through anonymization. These APIs allow security operations centers (SOCs) to streamline workflows by automating threat identification and response, accessing Elastic AI Assistant’s knowledge base, and integrating with security orchestration, automation, and response (SOAR) tools. The release also features integration with Google’s Gemini models and improvements to the Attack Discovery functionality, enabling simultaneous comparison of large language models (LLMs) and improved accuracy of lower-cost models. Elastic emphasizes the importance of user caution when handling sensitive data with AI tools and clarifies that the release timing of features is at their discretion, with no guarantee on delivery.

Elasticsearch 8.15 introduces several enhancements aimed at improving search functionality, including semantic reranking and additional tools for vector search, which enhance natural language search capabilities. The update makes Elasticsearch more performant, offering features like automated chunking for semantic text and introducing new third-party model providers such as Google AI Studio and Amazon Bedrock for greater model flexibility. The release also promotes the native Learning to Rank feature to generally available status, allowing for more refined search relevance and the ability to rescore collapsed results. Additionally, Elasticsearch 8.15 provides new options for vector search, including scalar quantization improvements and a new sparse vector query type, while emphasizing the importance of understanding the privacy implications of using third-party AI tools.

Elastic Observability 8.15 introduces several notable features, including enhanced OpenTelemetry capabilities and AI Assistant improvements. The release includes the Elastic Distribution for OpenTelemetry Collector, enabling users to efficiently ship logs and host metrics to Elastic with minimal configuration, and support for OTLP Universal Profiling data in Elastic Cloud. The Elastic AI Assistant now integrates with Google Vertex’s Gemini 1.5 Pro model and allows users to configure custom index support for its knowledge base, providing flexibility and enhanced LLM observability, especially for Azure OpenAI services. Additionally, a new Data Set Quality page helps users identify and resolve data ingestion issues, ensuring accurate and complete data sets. Service Level Objectives (SLO) management is also improved with new history and status reporting features. Elastic Observability 8.15 is available on Elastic Cloud and can be experienced through a free trial for new users.

Elastic 8.15 introduces a host of new features aimed at enhancing semantic search, integrating OpenTelemetry (OTel), and modernizing security analytics. This release includes tools for fine-tuning search relevance, flexible model options, and improved vector search, as well as AI-driven advancements that streamline custom SIEM data onboarding. Following the contribution of the Elastic Common Schema to the OTel project, Elastic 8.15 marks a significant step toward an OTel-first architecture with the distribution of an OTel collector. The update allows developers to implement AI search applications, utilize semantic text and reranking, and offers general availability of Learning to Rank and query rules. Elastic Observability and Security have been enhanced with new capabilities such as Google Vertex integration, LLM observability, and automated custom data integration using generative AI. The Elastic Search AI Platform now includes cross-cluster search enhancements, increased functionality in the Elasticsearch Query Language, and new machine learning features. Elastic 8.15 is available on Elastic Cloud, with the release and timing of features subject to Elastic's discretion.

Elastic Platform 8.15 introduces several enhancements to its Elasticsearch native vector database, including support for bit vectors, SIMD acceleration, and int4 quantization, which collectively improve performance and efficiency in vector search applications. The release features a new semantic_text field type that simplifies the configuration of semantic search by automatically generating embeddings and handling long texts through chunking. Additionally, the Elasticsearch Query Language (ES|QL) has been expanded with new query and aggregation capabilities, including geospatial search functions that align with OGC standards. The platform also introduces LogsDB, a new index mode designed to optimize log storage, and enhances user experience with features like named variables in ES|QL queries, improved cross-cluster search functionality, and enhanced Kibana dashboards. These updates aim to streamline data processing and search operations, making them more accessible and efficient for users without deep machine learning expertise.

Elastic Security has introduced three AI-driven capabilities—Automatic Import, Attack Discovery, and Elastic AI Assistant—to enhance security operations by automating labor-intensive tasks using Elastic's Search AI Platform and LangChain's generative AI orchestration. These features integrate LangChain components like LangGraph and LangSmith, enabling context-aware reasoning and efficient debugging. The Elastic AI Assistant simplifies the generation of ES|QL queries from natural language, using retrieval augmented generation to enrich context for large language models. Elastic's partnership with LangChain is expanding user options for integrating generative AI with their preferred language models, and Elastic Observability further complements this by offering comprehensive tracing and analysis of applications. While these innovations promise significant advancements in security workflows, Elastic emphasizes caution when using AI tools with sensitive data and notes that future feature releases are at Elastic's discretion.

Elastic offers an AI-powered log analytics solution designed to help operations teams manage complex, distributed environments by providing accurate and contextual observability insights. Elastic Observability, built on Search AI, integrates features like the RAG-based Elastic AI Assistant and AIOps, facilitating quicker issue detection and resolution with unlimited data retention and reduced costs. To encourage the transition from legacy systems like Splunk, Elastic's Express Migration program offers consumption and service credits, easing the migration process. By utilizing Elastic's unified platform and scalable data store, customers can experience significant reductions in costs and improvements in business performance through advanced analytics, such as ML-driven insights and anomaly detection. The migration to Elastic Observability promises enhancements in customer experience, operational efficiency, and decision-making capabilities, supported by over 400 integrations and industry-standardized semantics like OpenTelemetry.

Elastic has introduced the Elastic Express Migration program to facilitate the transition of Splunk customers to its AI-driven security analytics platform, Elastic Security. Leveraging the Search AI Platform, Elastic combines comprehensive search capabilities with retrieval augmented generation to automate and enhance security operations center (SOC) workflows. The program aims to alleviate migration challenges by offering incentives like migration credits and service credits to offset costs associated with transitioning from traditional solutions. Elastic's approach includes features such as Automatic Import for data onboarding and a suite of prebuilt integrations, which streamline the migration process. The initiative reflects a broader trend of moving towards AI-driven security analytics, promising improved data management, speed, and automation in threat detection and response.

Elastic has introduced Automatic Import, a feature designed to streamline the onboarding of data for Security Information and Event Management (SIEM) systems by utilizing AI-driven security analytics. This innovation automates the creation of custom data integrations, significantly reducing setup time from days to mere minutes, and is powered by Elastic's Search AI Platform, which leverages large language models for enhanced data processing. Automatic Import addresses the complexities and costs associated with migrating to modern SIEM solutions by facilitating broader visibility and simplifying the integration process for a range of structured and unstructured data formats. This development arrives as organizations seek alternatives to legacy SIEM tools, aiming to enhance security operations and expedite labor-intensive tasks. Elastic complements this capability with a robust library of prebuilt data integrations and a commercial incentive program, Elastic Express Migration, to encourage quicker adoption and mitigate migration inertia.

Elastic Cloud Serverless has expanded its availability to three additional AWS regions: eu-west-1 (Ireland), ap-southeast-1 (Singapore), and us-west-2 (Oregon, USA), offering users enhanced options for deploying Elastic's search, observability, and security capabilities with minimal operational overhead. Built on Elastic's Search AI Lake architecture, this serverless solution enables quick scaling and interactive data searches without the need for managing infrastructure. The platform provides a choice between simplicity with serverless projects and greater control with existing Elastic Cloud Hosted options. Users can quickly create serverless projects by selecting the desired project type and AWS region, with provisioning completed in minutes, thus facilitating rapid deployment and scalability. The expansion allows organizations to leverage AI search features, streamline observability workflows, and enhance security operations while minimizing latency and complexity.

Elastic has announced its plans to undergo an assessment through the Information Security Registered Assessors Program (IRAP) at the Protected Level to enhance its Elastic Cloud compliance portfolio, which underscores its commitment to security and integrity. IRAP is a framework used by Australian governments and security-conscious organizations to ensure cloud service providers have appropriate security controls in place, aligning with the Australian Government Information Security Manual. To achieve this certification, Elastic has partnered with CyberCX, an accredited IRAP assessor, to guide them through the meticulous certification process. Elastic is committed to keeping its customers informed about the progress of this initiative, which is aimed at reinforcing trust in its cloud services, although the timing of any new features remains at the company’s discretion.

Elastic Cloud's "bring your own key" (BYOK) feature allows users to manage encryption at rest by integrating AWS Key Management Service (KMS) keys, enhancing data security and compliance. The blog outlines the process to implement this feature, starting with the creation of an AWS KMS key, setting the necessary policy settings, and subsequently using this key for encrypting data in Elastic Cloud deployments. Users can manage AWS KMS key rotation and revocation, ensuring the protection of data against potential key compromises. The detailed steps include validating the setup, managing access control through AWS Identity and Access Management (IAM) policies, and using customer-managed encryption keys. Additionally, the blog hints at future content that will cover similar encryption processes using Azure Key Vault for Elastic Cloud deployments on Microsoft Azure.