July 2023 Summaries

Elastic Security has introduced a new advanced detection analytics package aimed at identifying malicious Remote Desktop Protocol (RDP) connections to prevent lateral movements by attackers. This package enhances the ability to detect fileless malware and malicious processes exploiting remote services and sessions, specifically targeting the commonly abused Windows RDP feature. The 8.9 release includes additional anomaly detection jobs and rules that establish a baseline for expected RDP session behavior, flagging deviations indicative of potential threats. The package, which is now generally available to Elastic users, leverages both anomaly detection and pre-built detections to enhance security measures against zero-day attacks. It provides tools for monitoring RDP session activities, such as tracking session duration and process execution, and includes the Living off the Land (LotL) Attack Detection package to identify malicious processes within RDP sessions. The setup process involves installing the package, running pivot transforms, and executing anomaly detection jobs, alongside enabling security detection rules to alert users of suspicious activities. Elastic Security continues to expand these capabilities, integrating user feedback to improve detection features.

The US Office of the Director of National Intelligence (ODNI) has introduced a data strategy for 2023–2025, emphasizing the need for the Intelligence Community (IC) to adopt common services, enhance data interoperability, and prepare for artificial intelligence (AI) integration. The strategy, which follows the 2017 data plan, aims to address challenges such as increasing data volumes and the need for advanced data capabilities. Key focus areas include end-to-end data management, data interoperability, digital innovation partnerships, and transforming the IC workforce to be data-driven. Elastic is highlighted as a tool to support these goals by providing a unified data platform that facilitates secure data discovery, access, and analytics, while enabling agencies to manage and leverage data more effectively. The strategy underscores the importance of viewing data as a strategic asset in achieving IC mission success and emphasizes the role of AI in enhancing intelligence capabilities.

Elasticsearch 8.9 has introduced an optimization for cardinality aggregations through dynamic pruning, significantly improving query performance under certain conditions. This optimization can dramatically speed up cardinality computations, sometimes up to 1,000 times faster, by using index structures to dynamically reduce the set of matches needing evaluation. For example, when determining the number of unique Kubernetes deployments monitored by Elastic Kubernetes integration, the optimization efficiently filters out previously seen values, reducing unnecessary evaluations. The process involves introducing a filter on a disjunctive query to match only new values, which dynamically updates as new values are detected, thus skipping redundant document evaluations. The most significant speedups occur when all unique values of a field are identified early in the query evaluation, enabling Elasticsearch to bypass much of the index. However, this optimization is only applicable to fields with a relatively small cardinality, specifically on segments with no more than 1,024 unique values, and it is limited to keyword fields where the cardinality aggregation is the sole and top-level aggregation. This improvement has been notably beneficial in enhancing dashboard loading times within the Elastic Kubernetes integration, achieving a 90% reduction in latency for some queries.

Fingerprint, a UK-based regulatory technology company, has developed a platform to help regulated businesses monitor electronic communications for compliance, leveraging Elastic for enhanced data analysis. The platform's multi-tenant architecture allows clients to oversee communication risks across multiple channels, such as email and chat applications, from a single interface. It utilizes natural language processing, sentiment analysis, and machine learning to identify and rank potential compliance breaches, streamlining supervision and reducing manual workload by up to 90%. Elastic's capabilities enable Fingerprint to manage vast amounts of data, offering features like risk scoring, anomaly detection, and cross-cluster search to enhance oversight and compliance efforts. The company plans to expand its use of Elastic to incorporate additional data sources and improve its anomaly detection capabilities, cementing its role in automating compliance supervision efficiently.

In response to the increasing focus on software supply chain security due to breaches like those of SolarWinds and Codecov, Elastic has partnered with Chainguard to assess their software supply chain security using the Supply Chain Levels for Software Artifacts (SLSA) framework. This collaboration aims to address the complexities and potential exposures in software development by isolating threats, enforcing strong access controls, and verifying software integrity at every stage. The assessment, the largest of its kind by Chainguard, evaluated Elastic's security practices through interviews and surveys, resulting in strategic initiatives such as deploying software artifact signing with Sigstore and enhancing detection rules in Elastic Security. The SLSA framework has proved instrumental in identifying risks and guiding mitigative actions, and Elastic continues to engage with the community and technology partners to bolster its security measures while encouraging researchers to focus on supply chain threats through increased bug bounty awards.

Generative AI presents both opportunities and risks for cybersecurity operations, as companies face the challenge of managing its integration while avoiding the pitfalls of "Shadow AI," where employees use AI tools without official approval, potentially exposing the organization to security risks. The article advises organizations to embrace generative AI by implementing strong controls and using trusted solutions, such as Microsoft Azure OpenAI, to maintain data confidentiality, integrity, and availability. Elastic promotes transparency and user control with its AI Assistant, emphasizing the importance of maintaining control over data and encouraging organizations to adopt generative AI responsibly. By doing so, companies can enhance their security posture and empower their teams, ultimately preventing the unauthorized use of generative AI and supporting the effective operation of security operations centers (SOCs).

Elastic Observability 8.9 introduces the Elastic AI Assistant in technical preview to enhance key observability workflows using generative AI, which aids in troubleshooting and provides automated explanations for complex information in areas such as APM errors, log messages, and host processes. This release also includes significant cost savings through time series data streams (TSDS), enabling up to 70% reduction in metrics storage for various integrations, including Kubernetes and AWS. Available on Elastic Cloud and for self-managed setups, this version integrates AI assistance with OpenAI and Azure OpenAI support, while leveraging TSDS for efficient data management and streamlined analysis.

Elastic Observability's recent update to version 8.9 introduces significant storage optimizations by incorporating Time Series Data Streams (TSDS) for its integrations, leading to up to 70% reduction in disk space usage for metrics data. This update is aimed at popular integrations like Kubernetes, Nginx, AWS, Azure, and more, making metrics storage cost-efficient and scalable. The TSDS-enabled integrations utilize features such as downsampling, which reduces data granularity to manage storage within budget, and synthetic source, which reconstructs data from doc_values to cut down on storage space. Additionally, TSDS optimizes indexing and retrieval by organizing data based on timestamps and dimensions, resulting in faster performance and more efficient data management. These enhancements are automatically available without manual configuration and are designed to be integrated outside the usual Elastic release cycle, offering seamless upgrades for users. The introduction of TSDS supports better long-term data analysis, reduces mean time to resolution (MTTR), and bolsters overall application performance, allowing organizations to better manage their growing data storage needs.

Elastic Search 8.9 introduces several advancements, including hybrid search using Reciprocal Rank Fusion (RRF) to integrate vector, keyword, and semantic search techniques for enhanced search results. Performance improvements are highlighted with vector search and ingestion being over 30% faster due to the use of the Panama vector API for native hardware instructions. The release also includes a new SharePoint Online connector for improved data ingestion with document-level security, and public search endpoints through a search applications beta, allowing for publicly accessible search experiences. These features are available on Elastic Cloud and for download with the Elastic Stack, providing users with flexible options for implementing the latest enhancements.

Elastic Security 8.9 introduces a range of new features aimed at enhancing the experience of security analysts, including the integration of the Elastic AI Assistant, which leverages generative AI and large language models to aid in security operations. This release brings advanced detection capabilities, customizable rule tuning, and improved alert management, allowing analysts to automate responses, tag alerts, and monitor detection rules effectively. The update also includes enhanced capabilities for detecting lateral movement using machine learning and prebuilt detection rules, as well as a new upload response action that streamlines endpoint orchestration. Additionally, Elastic Security has simplified the deployment of its Cloud Security Posture Management (CSPM) feature on AWS using CloudFormation, ensuring consistent and error-free resource configuration. These improvements are designed to optimize security workflows and provide analysts with robust tools to manage and investigate threats more efficiently.

Elastic 8.9 introduces several new features and enhancements across its security, search, and observability platforms, all built on the Elasticsearch platform. Elastic Security now unifies SIEM, endpoint, and cloud security to improve threat detection and response, and includes the beta release of the Elastic AI Assistant for Security, which aids in analyst investigations. The update to Elastic Search includes a tech preview of Reciprocal Rank Fusion for hybrid search and new integrations like ServiceNow and Dropbox connectors. Elastic Observability offers a tech preview of the Elastic AI Assistant for Observability and aims for significant storage reductions through cost-saving integrations. Core improvements include faster search aggregations, semantic search with dense vector embeddings, and high cardinality aggregations, alongside the tech preview of Elastic’s Learned Sparse Encoder and Reciprocal Rank Fusion for enhanced search relevance. Elastic 8.9 is available on Elastic Cloud, though the release of features is subject to Elastic's discretion.

Elastic Stack 8.9 introduces significant performance enhancements, particularly in cross-cluster searches and metric aggregations, leading to faster insights and improved search relevance. Key improvements include a 90% reduction in aggregation latency, significant speed increases in cross-cluster searches by minimizing network trips, and enhanced management of alerting rules using Terraform. The update also addresses issues with Elasticsearch's default refresh behavior and introduces optimizations such as improved cardinality aggregation and geo line simplification, which aid in memory consumption and accuracy. Additionally, the release enhances Kibana features, including new visualization options and improved rule management through Terraform, facilitating better observability and security issue management. Elastic Stack 8.9 is available on Elastic Cloud, providing both hosted and self-managed deployment options, though the release and timing of features are at Elastic's discretion.

The blog post explores how to create a generative artificial intelligence (GAI) application using Amazon SageMaker JumpStart, Elasticsearch, and open-source large language models (LLMs) from Hugging Face. The combination of these technologies enables businesses to build AI solutions that generate original content across various domains while ensuring data security and cost-effectiveness. By leveraging Elasticsearch's capabilities in data retrieval and Amazon SageMaker's model hosting, the post outlines the implementation of a Retrieval Augmented Generation (RAG) system that efficiently handles domain-specific data, enhancing LLM responses with contextual relevance. The post also details the setup and deployment process, including the use of various tools and services such as Python, LangChain, Streamlit, and AWS Identity and Access Management, to create a seamless and effective AI-powered search experience.

As enterprises increasingly adopt AI solutions to stay competitive, it is crucial to address data security concerns, particularly in protecting sensitive information and securing AI models against adversarial attacks. The blog post emphasizes the importance of establishing a robust and secure data infrastructure as a foundation for AI implementation, balancing innovation with privacy protection, and implementing measures like data encryption, access control, anomaly detection, and regular security audits. The post also highlights the need to defend against adversarial and backdoor attacks through strategies such as adversarial training and model hardening. By focusing on these security considerations, businesses can responsibly harness the potential of AI while maintaining trust and compliance with data protection regulations.

Government and education leaders anticipate a 59% increase in data volume over the next three years, presenting challenges in effectively storing, accessing, and using data to achieve mission goals. A global study by Elastic and ThoughtLab found that the public sector ranks lowest in "data maturity," highlighting difficulties in managing large data volumes due to storage requirements like M-21-31 in the US, accessing historical data, and overcoming data silos for optimal utilization. The study suggests that integrating internal data with AI, particularly generative AI and large language models, can enhance data relevance and personalization, while a unified data strategy can improve efficiency and cybersecurity without adding new vendors. McKinsey Global Institute estimates that data and analytics could generate $1.2 trillion annually in value for the public sector, emphasizing the importance of reducing data silos and making data actionable to improve customer experiences, employee productivity, and revenue.

Elastic's Open Security initiative has led to the development of the Elastic AI Assistant, an AI-powered tool designed to enhance security operations by integrating large language models (LLMs) into the workflow of security analysts. Built on the Elastic Common Schema (ECS), this assistant aids users by providing guided investigations, crafting queries, and offering workflow suggestions, thus making security tasks more efficient for both novice and experienced analysts. By contributing to open-source projects like OpenTelemetry, Elastic fosters a collaborative environment where users and developers can enhance the common schema for logs and metrics. The Open Security approach ensures that Elastic's AI tools, like the Elastic AI Assistant, leverage publicly available data to optimize operations and accelerate response times, helping analysts manage security events with greater speed and accuracy.

Integrating Elastic Cloud and Confluent Cloud enables seamless data management and analytics by allowing data streams from Kafka topics to be indexed into Elasticsearch using the Elastic Sink Connector, managed through a UI without requiring code. This integration leverages Confluent Cloud's real-time data ingestion and processing capabilities with Elastic's search and analytics functionalities, facilitating end-to-end data processing and analysis at scale, enhancing operational efficiency, and unlocking actionable insights. The process begins by generating mock messages with the Datagen Source Connector, followed by configuring the Elasticsearch Service Sink Connector to move data into Elastic Cloud, where it can be visualized using Kibana. This powerful combination supports the development of real-time applications, offering businesses the ability to explore and extract insights from large datasets quickly, ultimately driving innovation and improving customer experiences.

Nation-states, defined as sovereign entities inhabited by people with a shared national identity, often engage in cyber warfare to assert dominance or achieve strategic objectives, utilizing sophisticated techniques and state-sponsored actors to target critical infrastructure and gather intelligence. The Locked Shields exercise, organized by the NATO Cooperative Cyber Defence Centre of Excellence, is one of the world's largest cybersecurity drills, designed to enhance the cyber defense capabilities of NATO member nations and partners by simulating realistic cyber threat scenarios. This exercise involves defending the fictional state of Berylia from attacks by Crimsonia, testing participants' abilities to protect critical systems while maintaining operational continuity. Preparation for such exercises involves stakeholder engagement, asset management, and the establishment of effective communication and cooperation strategies, all aimed at improving readiness and response to nation-state cyber threats. Part 2 of the blog series will further explore Locked Shields and the role of Elastic Security in enhancing threat detection and response.

In a detailed exploration of using Elastic to identify anomalies within Splunk's Zeek data, the article highlights the historical context and benefits of Splunk's schema on read principle while acknowledging its latency tradeoffs. Elastic's commitment to integrating the Elastic Common Schema (ECS) with the OpenTelemetry project aims to establish a unified schema for metrics, traces, and logs, promising cost and performance benefits. The piece outlines a step-by-step approach to leveraging Elastic's Zeek integration with Splunk to set up anomaly detection, including configuring Elastic Agent and using preconfigured machine learning jobs in Elastic's Anomaly Explorer. The article emphasizes the simplicity of setting up these tools and the potential for real-time alerts on detected anomalies, underscoring how Elastic's capabilities can enhance data analysis and security monitoring.

An Elasticsearch index is a logical namespace that organizes a collection of documents, each consisting of key-value pairs, within a cluster, allowing for flexible data input without the need for predefined schema, unlike traditional relational databases. Elasticsearch is distinguished by its ability to handle JSON documents, provide advanced search capabilities through its RESTful API, and accommodate dynamic data types, which can be easily adjusted to improve performance. It supports denormalized data storage for faster retrieval and utilizes shards for distributed storage and processing, enhancing scalability. Additionally, Elasticsearch offers inverted indices for text searches, BKD trees for numeric and geolocational data, and dense vectors for semantic search, enabling efficient data analysis and robust search functionalities, including integration with AI-based applications through the Elasticsearch Relevance Engine.

Activating Application Performance Monitoring (APM) in Kibana and Elasticsearch enhances alerting insights by enabling detailed views of rule execution through distributed tracing. This process involves setting up an APM server, configuring Kibana and Elasticsearch settings, and using a comprehensive waterfall chart to visualize each step of rule execution. The article explains how to monitor specific transactions and identify potential performance bottlenecks without relying on slow logs. Users can create custom dashboards to monitor alert success and failure rates over time, leveraging Kibana's data visualization capabilities. Despite the detailed instructions, the features discussed are not yet available on Elastic Cloud, and users interested in these capabilities are encouraged to contact Elastic Support.

CIOs face numerous hidden data challenges as they strive to accelerate business outcomes, primarily stemming from data silos, lack of visibility, and outdated security measures. These obstacles hinder operational resilience by increasing mean time to detection (MTTD) and mean time to resolution (MTTR) while also complicating efforts to improve customer experiences through effective data management and security. The existence of isolated data pockets, inconsistencies, and poor data management can obstruct real-time issue detection and resolution, frustrating customers and exposing systems to security threats. However, these challenges present opportunities for organizations to harness untapped data potential by adopting solutions that enable real-time data processing and integration, ultimately enhancing security, customer satisfaction, and operational efficiency. By addressing these underlying issues, CIOs can transform data challenges into opportunities, ensuring that their organization's data contributes positively to achieving business goals.

Generative AI (GAI), exemplified by models like OpenAI's ChatGPT, is poised to transform the retail industry by enhancing customer experiences and driving innovation through applications in ecommerce search, customer support, omnichannel marketing, and supply chain predictive maintenance. The Elasticsearch Relevance Engine™ (ESRE) offers a flexible suite of tools that integrate machine learning with text search, allowing retailers to build AI-powered applications that address GAI's limitations related to privacy, scalability, and cost. ESRE enhances personalized search experiences, improves customer service by automating responses, aids fraud detection through predictive modeling, modernizes brick-and-mortar experiences by combining demographic data with real-time insights, and optimizes operations via predictive maintenance. This integration of GAI with Elastic's capabilities enables retailers to deliver more relevant search results, safeguard sensitive data, and maintain a competitive edge in a rapidly evolving market.

Sagar Patel, an Elastic Certified Engineer with over six years of experience, has developed a strong expertise in enterprise search tools and technologies like Elasticsearch. Starting his IT career in 2016, he became intrigued by search technologies and transitioned from proprietary tools to exploring Elasticsearch through its documentation and community resources. His active involvement in forums such as Stack Overflow and the Elastic forum led to his participation in the Elastic Contributor Program, where he was recognized as a Gold Contributor in 2023. Patel's contributions include writing informative blogs and engaging with the community, which helped him gain visibility and professional growth. He values the supportive nature of the Elastic community and credits the program with offering opportunities for learning and networking, as well as tangible rewards like certification attempts and training. Patel expresses gratitude for the community's collaborative environment and is eager to continue contributing.

Elastic has been recognized as a Visionary in the 2023 Gartner Magic Quadrant for Application Performance Monitoring (APM) and Observability for the third consecutive year, showcasing its innovative approach and full stack observability capabilities. Elastic's platform, which is built on the AI-enhanced Elasticsearch, offers features like log analytics, APM, SLO-based alerting, and digital experience monitoring, with Universal Profiling soon to be available. The company scored among the top three vendors in the Gartner Critical Capabilities report for three key use cases: DevOps/AppDev, SRE/Platform Operations, and Digital Experience Monitoring. Elastic's offerings are noted for their open and flexible platform, which allows for high-dimensional data ingestion, and a deployment model that supports hybrid and multi-cloud environments. The company's commitment to innovation is further highlighted by its integration of AI and ML capabilities, contributions to OpenTelemetry, and the introduction of the Elasticsearch Relevance Engine for building AI search applications. Elastic's observability solutions have been widely adopted, with strong customer endorsements and over 4 billion software downloads, indicating its continued influence in the observability market.

Hannah Mudge pursued her dream of becoming a software engineer despite being one of the few women in her computer science program, driven by her early passion for technology cultivated during her homeschooling years. She joined Elastic through the Elastigrad program after completing her master’s degree, appreciating their culture and the opportunity for remote work. At Elastic, she primarily works on front-end development for Kibana's Dashboard team, with occasional back-end tasks involving Elasticsearch, and she appreciates the variety in her role. Promoted to Software Engineer II after a year, Hannah aims to become a Senior Software Engineer, taking on larger projects and higher-level code design. Throughout her journey in tech, Hannah faced challenges as a minority in her field but remained steadfast, encouraged by supportive mentors and colleagues, and she emphasizes the importance of perseverance for others pursuing similar paths.

The pharmaceutical industry aims for rapid drug discovery by harnessing both public and proprietary data, and the integration of generative AI/LLMs and the Elasticsearch Relevance Engine (ESRE) can significantly aid this process. ESRE enhances AI-based search applications by applying semantic and vector search, integrating large language models, and facilitating hybrid searches, thus improving the efficiency of R&D teams. However, challenges such as data fragmentation, obfuscation in patents, and AI hallucinations need careful handling. By leveraging tools like Kibana for data visualization and LangChain for sequential processing, organizations can improve patent analysis and decision-making. The innovative use of the PatChat app exemplifies how generative AI can be applied to patent exploration, offering interactive and context-aware insights while addressing issues of privacy and contextual understanding. The approach, while aimed at pharmaceuticals, is applicable to any R&D-focused organization, showcasing the potential for improved collaboration, knowledge extraction, and innovation.

In the competitive IT industry, embedding services and training into software sales is a strategic approach that enhances customer satisfaction and drives sustainable growth. By offering comprehensive enablement programs, software vendors empower customers to maximize the benefits of their solutions, leading to increased productivity and efficiency. These programs foster a sense of partnership, building trust and improving customer retention. Embedded services, acting as value-added components, simplify onboarding and enhance product adoption, resulting in higher retention rates and positive referrals. Customizing offerings to meet unique customer needs strengthens loyalty and opens upselling and cross-selling opportunities, expanding revenue streams. The role of a customer success manager is pivotal, as they enhance the customer experience by aligning software solutions with business needs and advocating for clients to ensure continuous product improvement. This holistic approach positions companies as trusted partners, differentiating them from competitors and propelling growth in the software industry.