February 2022 Summaries

Elastic Security Intelligence & Analytics Team reported on a new data-wiping malware campaign called HERMETICWIPER, targeting Ukrainian systems. This malware is designed to render systems unusable by altering the boot process and erasing data from the file system. Such data-wiping malware is a common tactic among cyber attackers, primarily aimed at clearing and wiping drive contents. As the situation evolves rapidly, Elastic Security has developed capabilities to automatically detect and block HERMETICWIPER, and other security vendors have also verified this malware. Current users of Elastic Security can access these protective features, while new users are encouraged to explore quick-start guides, free courses, or a 14-day trial of Elastic Cloud to evaluate the product's capabilities.

Elastic Security has identified a new data wiper malware campaign called HERMETICWIPER, which is targeting Ukrainian organizations and aims to render systems unbootable by tampering with the boot process and wiping data from the file system. This malware tactic is commonly used by adversaries to incapacitate drives. Elastic Security's solution can automatically detect and block HERMETICWIPER, and other security vendors have also confirmed its presence. As the situation evolves, Elastic Security plans to provide ongoing updates on the threat. Current users of Elastic Security have access to the necessary capabilities to combat this malware, while new users can explore the platform through Quick Start guides, training courses, or a free 14-day trial of Elastic Cloud.

Version 7.17.1 of the Elastic Stack has been released, offering improvements and crucial fixes, including a patch for a potential security vulnerability and solutions to several upgrade issues from version 7.17 to 8.x. The update addresses problems such as nodes failing to start during system index migration, errors in running system index migrations, and migration failures of the internal .tasks index originally created in version 6.x. Users planning to upgrade to 8.x are advised to install this version first to avoid known migration issues. Detailed information on the fixes and changes can be found in the release notes for Elastic Stack components like Elasticsearch, Kibana, Beats, Logstash, Elastic Enterprise Search, Elastic Observability, APM, and Elastic Security.

Elastic and AWS have partnered to create a unified platform that facilitates the seamless ingestion of logs and metrics from various sources, enhancing operational visibility and security across both cloud and on-premises environments. This collaboration offers 23 ready-to-use integrations for AWS services, enabling users to efficiently monitor, analyze, and secure their data through Elastic's Observability solutions. These integrations simplify data ingestion, allowing for real-time monitoring and analysis of AWS resources like Amazon CloudWatch, AWS Lambda, and Amazon S3, among others, using prebuilt dashboards and custom visualizations. By utilizing machine learning and the Elastic Common Schema, organizations can detect performance issues, correlate data for troubleshooting, and enhance security with extended detection and response capabilities. Elastic's solutions aim to eliminate data silos, reduce alert fatigue, and provide comprehensive insights, all while being easily accessible through AWS Marketplace, where users can start with a free trial and integrate Elastic's features into their existing AWS ecosystem.

Migrating Elastic workloads to the cloud requires careful consideration of five key factors to ensure a smooth transition and maximize benefits. Ensuring business continuity is crucial, as cloud migration is often delayed, and maintaining uninterrupted Elastic workloads is vital during this process. The total cost of ownership can be optimized by utilizing cloud-native features such as data lifecycle management, autoscaling, and cost-effective storage solutions, reducing operational overhead by leveraging managed services. Disaster recovery and high availability are enhanced through Elastic Cloud’s multi-zone and multi-region capabilities, allowing data replication across different cloud regions and providers to mitigate risks. Compliance with data sovereignty regulations is facilitated by Elastic Cloud's cross-cluster search, enabling unified data access without geographic data transfer. Security is a primary concern during cloud adoption, and Elastic Cloud addresses this with robust security measures, including encryption, auditing tools, and dedicated security personnel, ensuring data protection and rapid response to vulnerabilities.

Data visualization tools are pivotal in transforming organizations into truly data-driven entities by making data more accessible, actionable, and insightful across various teams and leadership levels. These tools, when effectively integrated into existing IT systems, can break data silos and encourage strategic decision-making, behavior change, and cultural shifts within organizations. High-performing companies in data visualization see significant advantages, including improved net profit margins, revenue growth, and customer experience, as demonstrated by a study from MIT's Center for Information Systems Research. Visualizations also empower employees, enhancing job satisfaction and career progression, as they can more clearly see and act on their impact on business outcomes. The emotional and competitive responses elicited by well-designed visual data can drive organizational change, as shown in a case where a healthcare firm used visualizations to promote investment in employee nutrition programs. Despite its benefits, many companies have yet to fully embrace data visualization as a common practice, highlighting an opportunity for growth and improvement in leveraging data within the enterprise.

Cloud data storage offers the allure of accessible, query-ready data without the infrastructure burdens, but its effective utilization hinges on strategic decision-making aligned with business needs, application demands, and resource considerations. CIOs are advised to categorize data into "hot," "warm," and "cold" storage tiers to balance cost with accessibility and performance requirements, ensuring critical data is available when needed while minimizing expenses. Security remains paramount, as organizations must assess the sensitivity of their data and implement robust authentication and access controls to prevent breaches. Additionally, sound data governance is crucial to avoid wasted expenditures on unused cloud resources, with Flexera's report highlighting a potential 30% waste in cloud service spending due to poor data management. The integration of cloud storage strategies with traditional principles of performance, availability, capacity, and economics (PACE) is essential to prevent reduced application performance and ensure a cost-effective and efficient cloud storage framework.

The article by the Elastic Security Intelligence & Analytics Team discusses new detection strategies for bypassing Windows User Account Control (UAC) to prevent malware from silently elevating privileges on a user's machine without their knowledge. Malware often targets administrative privileges to perform harmful actions, yet UAC's default medium integrity setting typically blocks access to resources requiring higher privileges. The research focuses on methods attackers use to elevate integrity levels silently, known as UAC bypasses, which exploit elevated Windows features. Common bypass techniques used by malware families such as DarkSide, LockBit, and TrickBot include ICMLuaUtil, ComputerDefaults Execution Hijack, and FodHelper Execution Hijack. Elastic Security's research, aided by community observations, has informed 26 prebuilt endpoint behavior protections against these bypasses, emphasizing the importance of understanding adversarial tactics to enhance security measures. The article encourages users to explore Elastic Security's capabilities through quick start guides and trials, offering robust defenses against complex malware strategies.

Elastic and Amazon have resolved a trademark infringement lawsuit concerning the term Elasticsearch, leading to the exclusive availability of Elastic Cloud as the sole Elasticsearch service on AWS and its marketplace. This agreement follows changes to the licensing of Elasticsearch and Kibana in 2021 and Amazon's subsequent service renaming, aiming to eliminate market confusion and provide clarity to customers. Elastic emphasizes the authenticity and quality of its Elasticsearch service, which is available across major cloud platforms like AWS, Azure, and Google Cloud. The resolution paves the way for further collaboration between Elastic and Amazon, focusing on improving data ingestion and simplifying onboarding for Elastic Cloud on AWS. Recent developments include over 20 new integrations to enhance data ingestion, certification of Elastic Cloud as part of the AWS ISV Workload Migration Program, and various initiatives to streamline the migration process for customers. Elastic expresses enthusiasm for ongoing partnership efforts to optimize the Elasticsearch service experience on AWS.

The Elastic Contributor Program is a recognition and rewards initiative that encourages participation in the Elastic Community, offering various benefits such as virtual badges, networking opportunities, and prizes like Elastic training subscriptions and certifications. Participants can earn points by contributing in diverse ways, such as speaking at meetups, writing articles, or creating tutorials, and can compete for Gold, Silver, or Bronze status. The program aims to enhance participants' skills in Elasticsearch and increase their visibility to potential employers. With over 250 contributors submitting nearly 3,000 contributions across different regions, the program also fosters community support and collaboration, now including new features for scoring points and a referral system to further engage contributors.

Elastic 8.0 marks a significant update to the Elasticsearch platform, offering enhanced vector search capabilities, native support for natural language processing (NLP) models, and improved data onboarding and security features. This release aims to provide faster, more scalable, and relevant search experiences by integrating modern NLP models directly into Elasticsearch, enabling tasks like sentiment analysis and text classification without additional coding. The platform also introduces approximate nearest neighbor (ANN) search for efficient vector-based queries and expands cloud-native observability with new Amazon Web Services (AWS) integrations. Additionally, Elastic 8.0 simplifies security by enabling it by default for self-managed clusters, ensuring data protection and ease of configuration. These advancements are available through Elastic Cloud, with new users offered a 14-day free trial to explore the full range of features.

With the release of version 8.0, Elastic has introduced the capability to upload PyTorch machine learning models into Elasticsearch, enabling modern natural language processing (NLP) within the Elastic Stack. This integration facilitates the use of PyTorch models, which are popular for their support of deep neural networks like BERT, to perform a variety of NLP tasks such as sentiment analysis, named entity recognition, text classification, and text embeddings. Elasticsearch aims to provide a seamless user experience for uploading and managing these models with tools like the Eland client and Kibana's ML Model Management interface, while ensuring scalability and performance across clusters using the native libtorch library for inference. By incorporating NLP models directly into Elasticsearch, users can benefit from improved infrastructure, scalability, data security, and privacy, while also maintaining centralized management of models. The platform currently supports inference at ingest time and plans to expand to query time in the future, offering developers tools to build AI-powered search applications. Elastic encourages users to try out these capabilities with a free trial and provides resources for further learning and community engagement.

Elastic Stack 8.0 introduces simplified security features to enhance user experience by enabling security defaults for self-managed clusters with minimal configuration effort. Starting from version 7.1, these essential security features have been available for free, and in version 8.0, they are enabled by default. The security enhancements include user authentication, role-based access control, multi-tenancy with Kibana Spaces, and encrypted communications, ensuring a secure environment without cumbersome configurations. For Elastic Cloud users, security is already managed, while self-managed users only need an enrollment token and superuser credentials to set up their secure deployment. The streamlined process automatically generates unique credentials, certificates, and encryption keys during installation, providing a secure environment out of the box. The update also prevents third-party plugins from altering security modules, ensuring consistent and reliable protection across Elastic products like Observability, Security, and Enterprise Search. Users can easily initiate their secure deployment with Elasticsearch and Kibana, with options to further customize configurations to meet specific organizational requirements.

Organizations are increasingly moving to cloud environments to leverage flexible, affordable, and scalable infrastructure, and Elastic offers solutions that enhance these benefits by providing faster deployments, improved security, lower costs, and greater reliability. Elastic's managed service allows for on-demand scalability and elasticity, enabling users to quickly adjust infrastructure resources without downtime, which is crucial in today's fast-paced digital landscape. Security is bolstered by leveraging the expertise of cloud providers, enabling organizations to meet compliance requirements more easily. The cloud also reduces total ownership costs by minimizing operational overhead, with Elastic facilitating automated data lifecycle management and efficient storage solutions to optimize expenses. Additionally, Elastic ensures resiliency and reliability through features like continuous backups and multi-zone architectures, which help mitigate the risks of unexpected downtime. Global coverage is further enhanced by Elastic's ability to operate across multiple cloud regions, providing seamless access to infrastructure and data, thus offering an optimal experience for a distributed user base.

Building software reliability requires an understanding that goes beyond writing good code and implementing tests; it involves actively ensuring that the software will perform as expected under various conditions. Distributed tracing is a key technique for achieving this, as it provides insights into the actual execution of code, thus transforming uncertainty into predictability. The article emphasizes the importance of instrumentation in development, which allows for the real-time observation of software behavior in production environments without disrupting operations. It also highlights the significance of focusing on the four golden signals—latency, traffic, errors, and saturation—from the customer's perspective to enhance system reliability. Challenges associated with distributed tracing in modern complex systems, such as the need for manual context propagation and understanding diverse technology stacks, are acknowledged. Finally, the article discusses the broader concept of observability, urging developers to adopt open standards like OpenTelemetry for a more seamless integration of metrics, logs, and traces in order to build a comprehensive understanding of their systems' performance and reliability.

Maximizing Elasticsearch performance when adding nodes to a cluster involves understanding the impact on key hardware resources such as compute, storage, memory, and network bandwidth. Adding nodes can enhance a cluster's capacity to handle larger workloads and improve request handling, but it can also introduce performance bottlenecks if not planned carefully. Common bottlenecks occur in compute and storage resources, especially when nodes share hardware on virtual machines or containers. To mitigate these issues, it is crucial to configure shard allocation properly and monitor resource utilization, particularly CPU and storage throughput. Effective capacity planning, including choosing an appropriate shard count, can help ensure that adding nodes improves performance rather than causing instability. Whether using dedicated hardware or virtualized environments, careful resource allocation and monitoring are essential for optimizing Elasticsearch's scalability and efficiency.

Elasticsearch 8.0 introduces approximate nearest neighbor (ANN) search, enhancing vector search capabilities alongside traditional methods like term-based scoring. The surge in vector search interest arises from machine learning models that convert content into high-dimensional vectors, capturing similarities beyond surface characteristics. Traditional k-nearest neighbor (kNN) searches for vectors most similar to a query but can be inefficient for high-dimensional data. ANN algorithms, such as Hierarchical Navigable Small World (HNSW) graphs, offer a more scalable approach by trading off some accuracy for speed and efficiency, making them suitable for large datasets. Elasticsearch's ANN search, built on the Apache Lucene library, allows for efficient retrieval by integrating deeply with existing functionalities and ensuring compatibility across versions. The new _knn_search endpoint, introduced as a technical preview, efficiently retrieves similar vectors and is designed to iterate quickly, addressing open questions and integrating with Elasticsearch's core search functionalities in future updates.

Elastic's search-based platform supports the U.S. Navy's Information Superiority Vision by enhancing real-time situational awareness, accelerating DevSecOps pipelines, and bolstering cyber protection and interoperable data analytics. As part of the new National Defense Strategy, the platform offers cloud or on-premises deployment to help Navy and Marine Corps personnel and systems dispersed globally to query, visualize, and share information rapidly. Elastic's technology aids in mission-critical operations such as supply chain management, network management, and facility security by reducing mean time to resolution through geospatial visibility. It also empowers development teams to make critical decisions and detect threats efficiently, while Limitless XDR provides deeper insights for cybersecurity service providers and cyber protection teams. Elastic standardizes data across siloed or legacy systems, enhancing joint operations through interoperable analytics and enabling swift data queries, which are crucial for long-term mission analysis.

Elastic Enterprise Search utilizes Elasticsearch and Kibana to create accessible and unified search experiences through its App Search and Workplace Search features. To ensure optimal performance, a continuous performance testing framework has been established using tools like Jenkins and Docker. This framework allows developers to create and run performance tests that track the solution’s intrinsic performance, alert developers to any regressions, and compare different versions of the code. The testing process involves deploying the latest code on dedicated servers, running scenarios multiple times, collecting metrics, and using a performance dashboard to monitor results. A particular focus is placed on minimizing noise and ensuring consistent test environments by using controlled hardware and sticky deployments. The framework employs statistical methods like the Student's T-test to detect performance changes, with alerts sent to developers via Slack if regressions are consistently observed. Future enhancements include automated bisection for pinpointing faulty changes, APM integration for detailed analysis, and the ability to benchmark pull requests against the main project branch. This comprehensive approach aims to proactively manage and improve the performance of Enterprise Search, leveraging insights to provide better service to customers.

As businesses accelerate their digital transformations and adopt cloud technologies, operational complexity has increased, prompting a strategic shift towards observability solutions. Observability is now considered a critical initiative, providing visibility into application performance and connecting business and operational KPIs. The rise of containerization and Kubernetes has led to new challenges, with technologies like eBPF emerging to address these issues with minimal overhead. The need for visibility across hybrid and multi-cloud environments is growing, with machine learning playing a key role in extracting actionable insights from vast amounts of data. Organizations are gravitating towards open standards to avoid vendor lock-in, and consumption-based pricing models are becoming more popular. A holistic approach involving the right skills, processes, and technology is necessary for effective observability, which also requires tighter collaboration between operations and development teams. The integration of observability with security practices is becoming more prevalent, highlighting the importance of ensuring security alongside innovation. As observability becomes mainstream, it promises to enhance digital experiences by providing the necessary insights to support digitization and cloud adoption.

Elastic Cloud has adopted Automated Certificate Management using Let's Encrypt to enhance security and streamline operations as it expands its services and regions. This transition to a fully automated regional ACM (ACME Certificate Manager) was driven by the cumbersome and time-consuming nature of the previous manual certificate management process. After evaluating several options, Elastic chose a solution based on Terraform, leveraging an internal terraform provider alongside a terraform ACME certificate provider, which is familiar to the Elastic SRE team. The implementation involved overcoming challenges such as Let's Encrypt certificate rate limits and DNS challenges, requiring collaboration across multiple teams to ensure secure and reliable certificate sharing across regions. The ACM solution automates daily certificate renewal and revocation, using Terraform to manage certificates and store them in HashiCorp Vault while monitoring is conducted through Elastic Uptime Monitoring and Kibana. This automation has improved service stability by reducing human intervention and allowing for more rapid updates and expansions.

Elastic Stack 8.0.0-rc2, the second release candidate following rc1, has been announced, with a reminder that it is not yet suitable for production use and may not be compatible with future or past preview releases or the eventual 8.0.0 general availability. Users are encouraged to participate in the Pioneer Program by testing various components of the Elastic Stack, such as Elasticsearch, Kibana, Beats, Logstash, and APM, and reporting any bugs they encounter in the relevant repositories with a "Pioneer Program" label. Support for upgrade preparations is available through Elastic's forums and community Slack channel.

eBPF (Extended Berkeley Packet Filter) is gaining traction for tracing and security applications, yet developers face challenges due to the lack of tooling in its ecosystem. To address this, Leonardo Di Donato created bpfcov, a tool for gathering source-based code coverage information for eBPF programs running in the Linux kernel. Traditionally, eBPF programs are written in C and compiled using LLVM to a specific instruction set architecture for execution by the eBPF Virtual Machine. However, existing source-based coverage tools do not work directly with eBPF due to the constraints imposed by the BPF verifier and the architecture of BPF ELF files. Bpfcov overcomes these issues by adapting LLVM's coverage instrumentation to generate valid BPF ELF files, allowing developers to track detailed execution paths in their eBPF programs. The tool enables the generation of code coverage reports, offering insights into function, line, region, and branch coverage, which is crucial for debugging and optimizing eBPF programs. Bpfcov is open-source, inviting contributions and improvements from the community to enhance its capabilities and integration with the broader eBPF ecosystem.

Elastic App Search offers a versatile and powerful mechanism for enabling search capabilities over various types of data, accommodating both web-based and structured data from different sources. The quickest method to integrate data is through the web crawler, introduced in Elastic 7.15, which indexes web content efficiently by crawling URLs and sitemaps. For non-web content or data that doesn’t fit predefined criteria, users can upload JSON files via file upload or text-paste options, though these are subject to a size limit of 100 KB. For large-scale and constantly changing datasets, Elastic provides APIs that facilitate continuous indexing and updating of content, demonstrated through an example of indexing Major League Baseball statistics from a SQL Server database. This process involves transforming relational data into a denormalized format suitable for Elastic, ensuring that large volumes of data are indexed efficiently. Once data is indexed, users can leverage tools like Kibana for data visualization and create customized search experiences. Elastic's flexible ingestion options ensure that regardless of data structure or location, users can build effective search solutions, with further exploration encouraged through a free trial of Elastic Cloud.

Elastic Stack 7.17 marks the final release of the 7.x series, featuring a new and enhanced Upgrade Assistant designed to facilitate a seamless transition to version 8.0 by addressing Elasticsearch and Kibana deprecations with both automated and manual remediation options. This version enhances performance and scalability, introduces deduplication to optimize data storage, and updates Docker images from CentOS 8 to Ubuntu 20.04 due to the former's end of support. As Elastic prepares for the more advanced features in 8.0, such as vector search and modern NLP, users are encouraged to upgrade to 7.17 to ensure a smooth transition. The release is available on Elastic Cloud, offering the latest in search, analysis, and visualization capabilities, along with self-managed options through Elastic Cloud Enterprise and Elastic Cloud for Kubernetes.