August 2023 Summaries

Elastic has introduced enhanced support for the Japanese language in its Elasticsearch 8.9 release, allowing for semantic searches and natural language processing (NLP) tasks like sentiment analysis using machine learning models. This enhancement includes integration with Japanese NLP models such as BERT, which are crucial for pre-tokenizing text in languages like Japanese. Users can import models from platforms like Hugging Face into Elasticsearch using Eland, a Python library, to add necessary machine learning functionalities. The blog provides detailed instructions on setting up semantic search through vector embeddings, highlighting the importance of indexing vectorized text for effective searches. Additionally, it shows how sentiment analysis can classify text into positive, neutral, or negative sentiments, further expanding Elasticsearch's capabilities. Feedback on this technical preview is encouraged to improve support for Japanese and other non-English languages. Elastic emphasizes the importance of cautious use of third-party AI tools and acknowledges that most NLP functionalities are first developed for English, with support for other languages being gradually integrated.

Generative AI is poised to significantly impact higher education by transforming both pedagogical and IT landscapes. According to an EDUCAUSE survey, a majority of respondents believe that generative AI will drastically change higher education within the next few years. This technology allows universities to enhance productivity, personalize learning, and improve student outcomes by integrating generative AI with their internal data. Platforms like Elasticsearch provide a secure bridge between large language models (LLMs) and private university data, enabling accurate and context-rich outputs. Key benefits include automating IT workflows, personalizing digital services, strategically tracking resources, and enhancing cybersecurity. To effectively implement generative AI, institutions should prepare their data, ensure system scalability, and consider vector search technologies. McKinsey suggests that tech leaders should categorize and organize data, develop scalable systems, and prioritize data pipelines for contextual understanding. Although the potential of generative AI is immense, universities are advised to exercise caution, especially regarding privacy and data security.

Mandy Andress reflects on her journey as a security leader since the early days of the internet and highlights the lessons learned and the challenges faced in the evolving role of Chief Information Security Officer (CISO). The position is characterized by a complex blend of technical skills, business acumen, and interpersonal abilities, requiring CISOs to engage in both strategic and operational tasks. Beyond the day-to-day responsibilities, successful CISOs emphasize the importance of internal communication and collaboration to align security initiatives with business objectives, which involves engaging with executives, department heads, and stakeholders. By fostering a collaborative security culture and aligning with organizational goals, CISOs can significantly enhance their organization's cybersecurity resilience and their own effectiveness. Andress draws on insights from industry experts and fellow CISOs to outline the principles that distinguish exceptional security leaders.

Elastic has been awarded the 2023 Google Cloud Global Technology Partner of the Year Award, highlighting its successful collaboration with Google Cloud in driving digital innovation and addressing global challenges. This partnership has flourished due to shared goals of enhancing customer value and investing in joint product development and marketing strategies, resulting in over 2,600 customers utilizing Elastic on Google Cloud for accelerated insights. Elastic's integration with Google's technologies, such as AI, Elasticsearch Relevance Engine, and Vertex AI, along with support for Google's Cloud Security Command Center and GKE Autopilot, underscores their focus on innovation and customer success. The partnership has enabled enterprises, like Cisco, to implement efficient search platforms, showcasing the combined power of Elastic's speed and accuracy with Google Cloud's scalability to deliver cost-effective and user-friendly solutions.

Elastic and Google Cloud have expanded their partnership to enhance generative AI and security solutions, leveraging Elastic's Elasticsearch Relevance Engine and Google Cloud's Vertex AI platform to offer scalable, privacy-first AI tools. This collaboration aims to accelerate the adoption of generative AI, with advanced semantic search capabilities and industry-leading security measures to safeguard proprietary data. The partnership also introduces Elastic Cloud on Kubernetes within Google Distributed Cloud for better compliance with public sector requirements, alongside new integrations with Google Cloud's Security Command Center for improved cloud security posture management. Additionally, the two companies are investing in joint go-to-market initiatives, including co-selling support and global marketing efforts, to drive innovation and solve complex data challenges in the cloud.

The article from Elastic Culture discusses effective leadership strategies for managing distributed teams, emphasizing the importance of fostering career growth, celebrating differences, and maintaining open communication. Elastic, a company operating globally, highlights the necessity for managers to support and guide employees in their career development while encouraging individuals to take initiative. Leaders like Diana Jourdan and Stacey King Poling stress the value of mentorship and proactive engagement in one's career path. Celebrating cultural and communicative differences is also crucial, with Stacey King Poling and Madhura Chopda advocating for open dialogue and understanding among team members. The article underscores the need for effective communication across time zones, using both asynchronous and synchronous methods, and suggests that empathy and transparency can help overcome the challenges of a remote work environment. Elastic demonstrates how a balance of work and social interaction can be maintained through creative use of platforms like Slack for both professional and personal engagement.

Modalova, a French ecommerce marketplace, successfully doubled its revenue by enhancing search accuracy using Elastic's Elasticsearch via Elastic Cloud on AWS. Specializing in fashion, Modalova offers a curated selection of over 10,000 brands and two million products, drawing 150,000 monthly visitors. The implementation of Elasticsearch allowed Modalova to refine its search algorithms, delivering personalized and relevant results, which boosted conversion rates from 30% to 50%. This technological advancement not only improved the shopping experience but also attracted more brands to the platform, expanding its offerings. Modalova is exploring further enhancements with Elastic, including advanced natural language processing and vector search, to enable features like picture recognition and descriptive queries, aiming to elevate user experience and maintain competitive edge in the market.

Companies are increasingly inundated with data from various sources, creating challenges in managing, securing, and utilizing this information effectively. A survey by Elastic and ThoughtLab highlights that many organizations struggle with data management, particularly in integrating siloed data and using it to enhance customer experiences. The survey identifies "data leaders" as companies that have successfully developed modern data infrastructures and cultures, enabling them to extract significant value from their data, which results in improved revenue, innovation, and decision-making. These leaders employ strategies such as building unified IT platforms, leveraging advanced technologies like AI and machine learning, and fostering a data-driven culture. Examples include Danone's cloud-based data lake, AT&T's use of search technologies, and Putnam Investments' machine learning models, all of which illustrate how data can be operationalized to drive business growth and efficiency.

Black Hat USA 2023 highlighted significant advancements in cybersecurity, with a strong focus on generative AI and its potential to transform the industry. Key discussions included AI's role in democratizing cybersecurity and the DARPA AI Cyber Challenge, which aims to develop AI-powered security tools. Vendors demonstrated the integration of AI into current products, such as Elastic's AI Assistant, which aids in triage and investigation workflows. The evolving threat landscape was addressed with a focus on behavioral analytics to counteract identity-based attacks and insider threats. Elastic's advancements in data management were showcased, emphasizing speed, scalability, and cost efficiency by allowing rapid searches across large datasets without rehydrating them. The conference also highlighted the importance of open data standards, with projects like the Elastic Common Schema and the Open Cybersecurity Schema Framework gaining traction to enable better data analysis and integration. Overall, Black Hat 2023 emphasized the future of cybersecurity with AI, advanced analytics, and open integration frameworks, positioning Elastic and its community as leaders in modernizing security operations.

Generative AI is poised to revolutionize the healthcare industry by enabling advancements such as disease prediction, clinical trial optimization, new drug discoveries, enhanced patient support, personalized medication, and improved medical training. Utilizing private data alongside technologies like Elasticsearch, generative AI can analyze vast amounts of structured and unstructured medical data, facilitating early detection of diseases, efficient participant recruitment for clinical trials, and expedited drug research. AI-powered chatbots offer 24/7 patient support, while personalized medication benefits from data-driven insights into individual genetic profiles. Medical training is enhanced through realistic simulations, allowing trainees to practice in safe environments. However, the adoption of generative AI in healthcare necessitates robust privacy measures to protect patient data, including compliance with regulations like HIPAA. Elastic’s platform supports these advancements with its data management and real-time monitoring capabilities, although caution is advised when using third-party AI tools to ensure data security and confidentiality.

Version 8.9.1 of the Elastic Stack has been released, and it is recommended that users upgrade to this latest version over previous patches in the 8.9.x series. The release notes contain detailed information about the issues that have been fixed and the changes made to each product in this version.

Manuel Alba, a civil engineer specializing in computer science and telecommunications, is the organizer of the Elastic community in Chile and holds a prominent role at the University of Chile as a head of software development and architecture. His journey with Elastic began at Entel when the company transitioned from Splunk, which sparked his appreciation for the tool's observability features. This initial exposure led him to recognize Elastic's potential and incorporate it into his teaching. Alba's enthusiasm for Elastic technology culminated in organizing a virtual meetup during the pandemic, attracting 122 students and inspiring him to establish a series of knowledge-sharing events across Latin America. His commitment to the community earned him the Bronze Contributor award in 2022, and he emphasizes the significance of creating a supportive, sales-free environment where developers can learn and share insights. Alba credits the Latam Community team for their unwavering support, which has been crucial in fostering an inclusive and enriching ecosystem centered around Elastic's core pillars of search, observability, and security.

Public sector organizations are facing a "perfect storm" in cybersecurity, akin to a complex weather system, due to the convergence of increasing digital footprints, the rise of sophisticated cybercrime, regulatory pressures, and a significant cyber skills gap. The growth of digital interactions and technologies like IoT, 5G, machine learning, and AI is expanding data sources and complexity, while cybercrime, including state-sponsored attacks and ransomware, is becoming more organized and prevalent. Governments are introducing more regulations to protect citizen data, adding pressure on public sector teams. Compounding these challenges is the shortage of qualified cybersecurity professionals, making it difficult for the public sector to develop effective strategies. To navigate these challenges, public sector entities need forward-thinking strategies that evolve with technological advances and threat actor tactics, leveraging open security systems and engaging with experts and partners like Elastic to address skills gaps and innovate in defense strategies.

Stephanie Harris transitioned from a music-focused career to become a Senior Software Engineer at Elastic, drawing on her longstanding interest in computers and her knack for demystifying complex concepts. Initially pursuing music, Stephanie studied Classical Guitar and Choral Conducting but struggled to find fulfilling employment in that field, prompting her to revisit her childhood passion for technology, exemplified by the Spice Girls-themed website she created in fifth grade. She shifted gears and joined a coding bootcamp, which paved the way for her software engineering career. At Elastic, she contributes to the Platform Billing team, responsible for managing customer billing data, and appreciates the increasing diversity within her team. Stephanie emphasizes the importance of diverse perspectives in addressing a wide range of edge cases and encourages women aspiring to enter tech to embrace failure and seek supportive mentors.

David Pilato discusses various methods to preload Elasticsearch with a dataset, emphasizing that modifying the official Docker image is not ideal due to maintenance challenges. Instead, he suggests alternative approaches, including using a shell script, Logstash, Docker Compose, snapshot and restore, mounted directories, and Elastic Cloud. Each method has its advantages and limitations, such as flexibility, speed, and ease of setup. Pilato provides detailed steps for implementing these solutions, highlighting that the choice depends on user preferences and constraints. He encourages community input and reminds readers that the availability of features is subject to Elastic's discretion.

Philipp Kahr's blog post, dated August 10, 2023, delves into the intricacies of identifying slow queries in generative AI search experiences using Elasticsearch. It highlights the importance of instrumentation within Elasticsearch to gain insights into underlying operations, specifically when utilizing the Elastic Learned Sparse EncodeR (ELSER) model for semantic search. The post outlines the setup process, from activating tracing in Elasticsearch to creating indices and ingest pipelines, and demonstrates how to measure and analyze transaction durations and query times. Using a dataset from OpenWebText, which contains roughly 40GB of text, the blog post illustrates how to index documents and analyze the time taken for machine learning model inferences. It emphasizes leveraging Kibana for visualization and metrics tracking, such as document size and query durations, to optimize search performance. The post concludes by acknowledging the current issue with Elasticsearch spans, which will be addressed in a future release, and encourages users to utilize transaction duration data for anomaly detection and A/B testing.

Elastic and Kyndryl have collaborated to enhance mainframe observability, targeting industries heavily reliant on mainframes, such as financial services, healthcare, retail, and manufacturing. Mainframes, known for their reliability and capability to manage vast data volumes, pose challenges in monitoring and performance analysis. The Elastic-Kyndryl solution addresses these challenges by utilizing Elastic technology to ingest System Management Facility (SMF30) and Syslog data, providing insights through customizable dashboards. This approach enables organizations to identify resource-intensive programs, detect anomalies, and optimize performance, ultimately reducing operational costs and improving service continuity. Additionally, the solution's batch job analytics tool offers predictive capabilities and visualization of performance metrics, ensuring system reliability and compliance, particularly in sectors where data security is critical. The implementation of this solution promises significant business benefits, such as cost savings and enhanced operational efficiency, while maintaining flexibility and scalability across large environments.

Elasticsearch has introduced a new specialized query language, Elasticsearch Query Language (ES|QL), which has been integrated into the Elasticsearch main branch after a year of development. Designed for expressiveness, composability, and speed, ES|QL serves as a unified querying interface that simplifies interactions with Elasticsearch by bringing compute capabilities close to the data, reducing the need for data transfers to external systems. Unlike existing query languages like QueryDSL, ES|QL includes a dedicated query and compute engine that performs comprehensive tasks such as filtering, processing, and sorting within the Elasticsearch environment, and it operates on data blocks rather than individual rows to enhance performance. The language supports complex queries involving multiple chained operations and is designed to exploit local data characteristics through specialized execution strategies that emphasize vectorization and multi-threading, offering significant performance improvements over traditional methods. Now available as a tech preview in Elasticsearch's main branch, ES|QL is a free tier feature that encourages user feedback as it continues to evolve, with future updates and features remaining at the discretion of Elastic.

Elastic has announced the technical preview of the Logstash Operator for Elastic Cloud on Kubernetes (ECK), which simplifies Logstash management by integrating it with the Elastic Stack on ECK and removing the need for manual resource management. Users can configure Logstash pods with just a few lines of YAML, utilizing the Logstash Domain Specific Language (DSL) for pipeline configuration, either inline in the Custom Resource Definition (CRD) or in mounted volumes. The operator supports multiple pipelines and automatically reloads updates without needing to restart pods. The preview includes enhanced Elasticsearch integration with plugins facilitated by elasticsearchRef and pre-populated environment variables, as well as simplified monitoring through stack support. The Logstash setup can include creating Logstash pods that send data to Elasticsearch, integrating Elastic Agent, and enabling stack monitoring. Further development promises even more streamlined integration within the Elastic Stack, with the feature set currently in technical preview.

Elastic has introduced advanced entity analytics to enhance security operations by integrating machine learning-based behavioral analytics, risk-based context, and generative AI into their workflows. This innovation is part of Elastic's strategy to modernize security information and event management (SIEM) by unifying SIEM, endpoint security, and cloud security into a single product, making it a leading platform in security analytics. The approach, which includes unsupervised and supervised machine learning capabilities, aims to provide security analysts with the tools needed for efficient threat detection, investigation, and response (TDIR). Elastic's system is integrated with an AI Assistant, offering analysts contextual guidance and support, and aims to democratize security by reducing the complexity and cost of data management. Elastic's advanced analytics are designed to be a native component of their platform, addressing traditional user and entity behavior analytics (UEBA) use cases and offering a streamlined, holistic approach to threat management.

In a comprehensive comparison of Elasticsearch 8.7 and OpenSearch 2.7, Elasticsearch consistently outperformed OpenSearch across various performance metrics, demonstrating a speed improvement of 40%–140% in key areas such as text querying, sorting, date histogram, range, and terms. The study highlighted Elasticsearch's superior resource efficiency, using less disk space and optimizing data compression better than OpenSearch. The testing involved creating equivalent clusters for each platform and running extensive queries to ensure statistical accuracy, validated by a third-party enterprise strategy group. The findings suggest that Elasticsearch offers a compelling choice for organizations requiring fast, efficient, and accurate search capabilities, making it particularly suited for use cases in ecommerce, security analysis, and large-scale observability.

Elastic has contributed the Elastic Common Schema (ECS) to the OpenTelemetry (OTel) project to standardize security and observability data within the OTel Semantic Conventions framework, which is gaining traction due to its vendor-agnostic nature. This initiative, embraced by various security vendors and partners, aims to simplify data querying, visualization, and analysis, thereby enhancing machine learning applications and reducing the complexity associated with analyzing software, performance, and security issues. The collaboration has received positive feedback from companies like Cribl, Mimecast, and Tines, who appreciate the increased flexibility, interoperability, and reduced integration challenges that these open standards offer. Open source conventions are seen as crucial for improving productivity and modernizing security operations, with the movement encouraging further community contributions and real-world testing to strengthen technology solutions.

A recent IDC Market Perspective report highlights the Elastic AI Assistant, emphasizing its functionality and value in the realm of generative AI, particularly due to its accessibility to the broader practitioner ecosystem. Elastic has leveraged its experience in machine learning and its Elasticsearch Relevance Engine™ to bring this technology to market swiftly, offering a freemium model and monitoring OpenAI usage to enhance user experience. The report suggests that Elastic's approach, which treats security as a data problem, offers significant advantages in modern security operations by allowing organizations to adopt generative AI effectively. IDC recognizes the potential for expanding Elastic AI Assistant's capabilities beyond IT and cybersecurity to boost customer satisfaction further. Elastic's open platform enables transparency and ease of access to its code, supporting the training and development of AI models. However, the report advises caution when using AI tools with sensitive information and emphasizes understanding privacy practices.

Philipp Kahr's article focuses on troubleshooting slow Elasticsearch queries to enhance user experience, emphasizing the importance of identifying and addressing slow queries in various applications such as e-commerce and workplace search solutions. The article discusses the use of Elasticsearch's slow log and tracing features to capture query performance metrics, highlighting the advantages of application performance monitoring (APM) for more detailed insights. It describes how to activate tracing via configuration settings and dynamic adjustments, as well as the integration of APM to capture REST API endpoints and enrich search applications with metadata. The article includes a practical example using a Flask application to demonstrate query performance analysis and optimization, ultimately aiming to improve search speed and system throughput. Additionally, Kahr offers tips on creating dashboards and visualizations to track performance trends and mentions a known issue with transaction duration metrics in Elasticsearch, which is expected to be addressed in a future release.

Kibana, built on the Node.js framework, is transitioning to Node.js 18 following its elevation to Long Term Support (LTS), which necessitated a custom build due to Node.js 18's incompatibility with certain older Linux distributions like Ubuntu 16, Red Hat Enterprise Linux 7, and CentOS 7. This transition was delayed as Node.js 18 requires a newer version of the GNU C Library (glibc) than these distributions support, prompting Kibana to develop a version compatible with glibc 2.17. With Node.js 16's End-of-Life date brought forward to September 2023, the upcoming releases of Kibana 8.10.0 and 7.17.13 will include Node.js 18, ensuring continued stability without disruption to users. Kibana's custom Node.js build will also be available for other projects via GitHub. Users on outdated Linux distributions are advised to upgrade or consider alternatives like Elastic Cloud or Docker images to maintain compatibility. Elastic highlights that while changes are at its discretion, keeping the Elastic cluster updated ensures users benefit from the latest security patches and bug fixes.

Elastic challenges the prevalent notion that organizations must use cumbersome data lakes for long-term data retention, advocating instead for a more efficient approach using their own platform. Elastic's solution leverages a unified data layer that allows fast, affordable, and accessible analysis of large datasets without the need to rehydrate entire contents from storage. This approach is especially beneficial in scenarios like the SolarWinds attack, where rapid access to historical data is crucial. Elastic's system uses an index-on-write methodology, enabling high-speed searches over inexpensive object stores and across multiple systems via cross-cluster search. This capability not only enhances threat detection and incident response but also supports advanced analytics and visualization tools. Elastic claims their solution is not only more powerful but also more cost-effective than traditional data lakes, as demonstrated by users like the U.S. Cybersecurity and Infrastructure Security Agency.

NSM-8, introduced in January 2022, directs the US Department of Defense and the Intelligence Community to enhance cybersecurity through unified standards and reporting, aligning with the objectives of Executive Order 14028 for civilian agencies. Elastic has supported these entities by providing a cost-efficient, unified approach to cybersecurity, logging compliance, and data analysis. Key aspects of NSM-8 include accelerating the shift to secure cloud services, implementing Zero Trust Architecture, deploying Endpoint Detection and Response (EDR) capabilities, and maintaining comprehensive event logging. Elastic's platform facilitates real-time data search, correlation, and machine learning across multi-cloud and multi-cluster environments, helping agencies reduce attacker dwell time and improve incident response. By using frozen tier storage for older data, Elastic offers affordable, easily searchable data retention, significantly lowering storage costs. Elastic's extended detection and response (XDR) solution enhances protection by integrating data from various sources, applying detection layers, and automating policy enforcement. NSM-8 emphasizes the extended retention of logs for effective investigation and remediation, a directive Elastic supports with its comprehensive logging capabilities.

New SEC rules require companies listed on the US Stock Exchange to disclose material cybersecurity incidents and detail their cybersecurity risk management strategies annually, aiming to enhance transparency for shareholders and investors. The article explores how these filings can be efficiently collected and analyzed using Elastic and sec-api.io, which pre-processes data to allow for easier access through its APIs. By leveraging Elasticsearch, the author demonstrates a proof of concept for automating the retrieval and analysis of SEC filings, particularly focusing on Form 8-K, which includes new item 1.05 for cybersecurity incidents. Although item 1.05 is not mandated until December 2023, the author uses item 1.04 related to mining safety as a substitute to test the system, successfully setting up automation for data ingestion and visualization in Elastic Cloud. The process provides valuable insights for CISOs and business leaders to understand the types of security incidents being reported and enables alerting when relevant incidents occur. This project highlights the potential of Elastic Cloud to analyze public data for actionable insights, with the caveat that certain features may change at Elastic's discretion.

Large language models (LLMs) often produce inconsistent responses, which complicates their integration into workflows that require specifically formatted outputs, such as queries. To address this, the concept of a "prompt sandwich" is introduced, which structures prompts into three layers: system prompt, context, and user prompt. This method guides the conversation's direction, provides necessary context, and formulates specific questions or requests to enhance consistency and quality. The Elastic AI Assistant exemplifies this technique by integrating data like security alerts and pseudonymizing sensitive information to maintain privacy. Users can customize prompts and manage data anonymization settings, ensuring only selected fields are shared with LLMs. This approach not only improves the reliability of responses but also emphasizes the importance of data privacy when utilizing generative AI.