August 2021 Summaries

Ransomware attacks have become a significant threat to state and local governments in the U.S., costing over $18.9 billion in 2020 alone, and there is a collective responsibility involving government agencies, the private sector, and individuals to combat this issue. Elastic advocates for a proactive security strategy, emphasizing two key techniques: canary-based detections and searchable snapshots. Canary-based detection involves placing hidden files in key system locations to provide early warnings for ransomware tampering, thereby preventing attacks from spreading to data centers. This method, included in Elastic Security's 7.14 release, complements other protective measures like behavioral ransomware and anti-malware protections. Meanwhile, searchable snapshots allow agencies to retain and instantly access large volumes of historical data, enabling the detection of anomalies and malicious activity by comparing past and current data without the need for costly data rehydration. This feature supports a defense-in-depth strategy, crucial for combating increasingly sophisticated cyber threats, and provides an effective and affordable security solution for state and local governments seeking to enhance their cybersecurity posture.

The Elastic Infosec Detections and Analytics team leverages the Elastic Stack to secure its distributed enterprise network by utilizing innovative approaches tailored to its unique, remote-first environment. The team, known internally as Customer Zero, employs Cross Cluster Search (CCS) to manage and monitor data across multiple clusters, allowing analysts to access and analyze security events from a single Kibana interface. This architecture supports seamless incident response and threat detection, as evidenced by their ability to quickly investigate and respond to potential security breaches, such as compromised user accounts. By using CCS, Elastic avoids the pitfalls of centralized data storage, such as single points of failure, while maintaining data integrity and regulatory compliance. The setup also facilitates testing and upgrading without disrupting production clusters, ensuring that both Elastic’s internal processes and customer solutions remain robust and efficient.

As an Elastic Technical Account Manager (TAM), Asjad Athick plays a crucial role in aiding customers across various industries to harness the full potential of Elastic technologies for monitoring complex IT environments and establishing robust SIEM platforms. His work involves breaking down silos within large organizations to enhance customer experiences and supporting security teams in responding swiftly to threats, thereby ensuring brand security and trust. A notable project highlighted Athick's innovative orchestration and autoscaling of Elasticsearch to manage massive surges in log volume for a cloud-native fantasy sports platform, showcasing the scalability and adaptability of Elastic solutions. Through strategic and tactical engagements, he helps clients, including startups, transition from basic application logging to comprehensive observability solutions, driving growth, innovation, and cost-effective infrastructure optimization. The efforts of TAMs like Athick exemplify how Elastic's Embedded Consulting Services deliver technical expertise and strategic insights to amplify customer success and satisfaction.

Elastic has announced a collaboration with Cmd to enhance its cloud security offerings, specifically focusing on cloud workload runtime security. This partnership aims to integrate Cmd's expertise in infrastructure detection and response, which utilizes extended Berkeley Packet Filter (eBPF) technology for deep visibility into cloud workloads, into Elastic Security. The integration will provide customers with comprehensive protection throughout the cloud security lifecycle, from build-time to runtime, and enhance Elastic's capability to detect, prevent, and respond to attacks on cloud workloads. The collaboration aligns with Elastic's commitment to developer-friendly solutions and aims to offer frictionless security for Linux systems by leveraging eBPF technology. This move follows Elastic's acquisition of build.security and aims to expand Elastic's security capabilities, ensuring comprehensive cloud-native observability and security. The partnership also reflects Elastic's pricing strategy, which focuses on resource-based pricing to offer organizations the best protection without counting endpoints. As Elastic integrates Cmd's technology into its platform, customers can expect enhanced security capabilities and innovative solutions for cloud-native applications.

Elastic has launched the Elastigrad Program to recruit and train early-career engineers, aiming to diversify and strengthen its engineering team with fresh perspectives. Spearheaded by Chief Product Officer Ash Kulkarni, this initiative is part of Elastic's commitment to its people and innovation, offering full-time positions with benefits such as technical mentorship and competitive salaries. The program draws talent from diverse backgrounds worldwide, including self-taught individuals, bootcamp graduates, and traditional college alumni, fostering an environment rich in unique ideas and experiences. By partnering with experienced engineers, the Elastigrads are expected to contribute to key projects and address customer challenges creatively, supporting Elastic's mission to build a collaborative and inclusive company culture.

Elastic has announced its acquisition of build.security, a move aimed at enhancing cloud security by integrating build.security's expertise in policy definition and enforcement with Elastic Security's capabilities. The collaboration leverages the Open Policy Agent (OPA) to provide real-time security enforcement across cloud-native environments such as Kubernetes, enabling continuous monitoring and compliance with security policies. This partnership aligns with Elastic's mission to protect data and systems, expanding their security reach from endpoints to cloud workloads. The integration promises advancements in policy management, enabling users to validate their security posture against standards like CIS benchmarks and maintain secure configurations across cloud infrastructures. The move also establishes a stronger presence for Elastic in Tel Aviv, Israel, which is recognized for its innovative contributions to cybersecurity. This collaboration reflects Elastic's commitment to enhancing its security solutions and contributing to the open-source community, while setting the stage for further developments in cloud-native security practices.

Kibana offers a powerful platform for analyzing and visualizing data through the use of formulas, time travel, and interactive dashboards. Users can leverage custom formulas to create metrics by combining aggregated fields, enabling the exploration of data trends and patterns over time. Key functionalities include calculating error ratios, comparing performance metrics over different periods, measuring deviations from averages, and assessing changes in metrics such as CPU usage. Users can also utilize maps for geographic context, interactive dashboards for spatial filtering, and custom formulas for industry-specific metric calculations like Net Promoter Scores. The platform's flexibility allows for real-time adjustments and insights, making it a versatile tool for data analysis. Kibana's capabilities are further supported by community forums and documentation for users seeking to enhance their data visualization skills.

Google Cloud has introduced support for N2 general-purpose virtual machine (VM) types on Elastic Cloud, powered by Intel 2nd Generation Xeon Scalable processors, offering a notable improvement in price-performance over the previous N1 machines. Users provisioning an Elastic deployment on Google Cloud will automatically use N2 VM types, with options to select from default or custom configurations to optimize vCPU, RAM, and storage according to specific use cases. Elastic Cloud also provides deployment templates that align with the new hot data tier instance configurations, simplifying hardware selection. Existing deployments can be migrated to N2 VM types following a process that includes creating a snapshot, selecting settings, choosing a deployment template, and restoring the snapshot. For those interested, Elastic Cloud offers a free 14-day trial to explore these new features.

The blog post, created in collaboration with Lightrun team members Itai Tieger, Roy Chen, and Tom Granot, explores how developers can achieve full-cycle observability for applications using the Elastic Stack and Lightrun. It highlights the challenges developers face in managing logs, such as inconsistent formatting and distribution across multiple hosts, which complicate error tracking and data analysis. The article discusses using the Elastic Stack to centralize log storage and analysis, providing a step-by-step guide on integrating Java applications, like the Spring PetClinic, with Elastic for better log management. Additionally, it introduces Lightrun as a tool for real-time observability that allows developers to add logs on-demand without redeploying applications, which is beneficial for identifying and resolving issues in complex, fast-evolving software environments. The article concludes with an emphasis on how Lightrun and Elastic together can enhance developers’ abilities to monitor and debug applications efficiently.

Elastic Security facilitates robust cyber threat intelligence (CTI) by leveraging Elasticsearch's powerful search engine capabilities to collect and enrich data with threat intel feeds, enabling organizations to better understand and counteract cyber threats. CTI involves analyzing emerging threats and applying defensive measures to protect environments, using indicators of compromise (IOC) such as file hashes, IP addresses, and URL reputation data. Elastic Security's integration with SIEM detection capabilities helps prioritize alerts and detect active threats through indicator match rules, which are triggered when network or host events align with threat intel data. The Filebeat Threat Intel module and Elasticsearch enrich processors streamline the collection and enrichment of threat data, providing additional context to incoming data and enhancing the efficiency of threat detection. Elastic offers prebuilt and customizable detection rules, allowing security teams to identify and alert on suspicious activities, such as network connections to malicious IPs, thus improving the overall security posture. By utilizing CTI, organizations can make informed security decisions, protect their assets, and reduce the time needed for event resolution, with Elastic Security providing accessible resources such as Quick Start training and a free 14-day trial to facilitate adoption.

Elastic's release of Filebeat 7.14 introduces the filestream input, a more advanced successor to the log input, which enhances the reading of active log files by improving system reaction time, registry updates, and integration with external log rotation tools. This update significantly optimizes registry performance by adopting an append-only method for offset updates, reducing the number of fsync calls, and enabling more efficient removal of obsolete entries. Harvester management is improved, allowing parallel checking for open files needing closure, while registry operations are no longer contingent on output availability, thus expediting outdated state removal. The flexible reader pipeline now allows configurable ordering of multiline, JSON, and container log parsing, with plans for additional parsers like syslog. Enhanced cooperation with log rotation tools, particularly for copytruncate strategies, ensures correct processing of rotated files without duplicating events. Further improvements include the new include_files feature, more precise file tailing options, and a focus on continuing to refine log management capabilities based on community feedback.

On August 10, 2021, Elastic announced the release of the Elastic Stack 8.0.0-alpha1, marking the first public alpha version of their upcoming major release. This version introduces new features and significant changes, although it is not recommended for production use due to potential incompatibilities with future releases. Elastic is also reviving its Pioneer Program, which began with version 5.0, to encourage community feedback during the preview period. This program invites users to test the alpha version and report issues specific to their unique environments, which aids in refining the product before the general availability release. The company emphasizes the importance of user feedback, expressing gratitude for the community's role in enhancing the product's development.

Elastic has been recognized for its innovative efforts by major partners Google Cloud and Microsoft, highlighting its commitment to providing seamless customer experiences through strategic cloud partnerships. Elastic was awarded the 2021 Microsoft US Partner Award for Business Excellence in the Commercial Marketplace, reflecting its alignment with Microsoft's engineering and product teams to enhance user experiences on the Azure Marketplace. Additionally, Google Cloud named Elastic the 2020 Technology Partner of the Year for Data Management for the second consecutive year, acknowledging its excellence in innovation and customer service. These partnerships enable Elastic's customers to access its Observability, Enterprise Search, and Security solutions across various cloud platforms, including Google Cloud, Microsoft Azure, and Amazon Web Services, facilitating easier software procurement and operational efficiency. Elastic's CEO Shay Banon emphasizes the importance of collaborating with partners to help companies organize, analyze, and protect their data effectively.

Elastic Observability 7.14 introduces unified telemetry and centralized agent management with Elastic Agent and Fleet, enhancing data onboarding and security while accelerating root cause analysis in Elastic APM. The update allows for the collection of various data forms with a single agent per host, facilitating seamless endpoint security deployment across infrastructures. It also features automated root cause analysis using APM correlations to identify attributes linked to performance issues. The release is available on Elastic Cloud, and it supports centralized management and real-time updates of agents, improving scalability and flexibility in data collection. The Elastic Fleet application simplifies agent management, enabling easy deployment of upgrades and policies across numerous agents with a user-friendly web interface.

Elastic Cloud's 7.14 release introduces enhanced security features and expanded deployment options, notably through the integration of Microsoft Azure Private Link, which allows private connectivity between Azure virtual networks and Elastic Cloud deployments, keeping data off the internet. The update also anticipates the support of Google Cloud Private Service Connect for securing connections on Google Cloud. Additionally, there are new virtual machine options with Google Compute Engine's N2 VMs, offering up to 20% better CPU performance and significantly increased storage, making deployments more cost-effective. The release extends Elastic Cloud's regional availability to over 45 locations, including a new region in Seoul, South Korea, across three cloud providers. These updates are detailed in Elastic Cloud's release notes, with more information available through their Quick Start guides and a free 14-day trial offering.

Elastic Security's latest advancements in XDR (Extended Detection and Response) aim to redefine cybersecurity operations by merging SIEM and endpoint security into a unified platform. This integration allows users to ingest and analyze vast amounts of data from diverse sources while enhancing threat detection and response capabilities through machine learning and automated processes. Elastic Security's unique approach offers limitless data retention and analysis, enabling organizations to conduct thorough threat hunting and remediation at scale. The platform emphasizes democratizing security, making advanced threat detection and response accessible not just to large enterprises but also to smaller organizations without robust security programs. Elastic's solution is characterized by its open architecture, which supports extensive data integration and customization options through Elastic Agent and Logstash, while offering seamless collaboration and case management with key remediation vendors. This innovation promotes efficient automation of analyst workflows, allowing for rapid detection and response to evolving threats, ultimately providing a scalable and flexible security solution for protecting data across various environments.

Elasticsearch 7.14 introduces several enhancements aimed at improving geospatial capabilities, performance, and usability across its platform. Building on Apache Lucene 8.9.0, this release includes the addition of runtime fields for geo_shape queries, leading to more efficient geospatial data analysis. It offers a 15% faster geotile grid aggregation and introduces a new field type, match_only_text, which reduces storage requirements by 10%. The update also enhances aggregation performance, particularly for single bucket date_histograms, and optimizes composite aggregation memory usage. New APIs facilitate the migration of node attributes to node roles and support new range aggregations over histogram fields, aiding data analysis in observability and security contexts. Elasticsearch 7.14 also brings improvements to its machine learning features, such as support for top metric aggregation in transforms and a simplified anomaly detection reset job API. The release highlights Elastic's commitment to making Elasticsearch a powerful, efficient tool for data search and analysis, with updates also extending to Elastic Enterprise Search, Elastic Observability, and Elastic Security solutions.

Kibana 7.14 introduces enhanced ad hoc analytics capabilities, including custom formulas and time-shifted metrics in Kibana Lens, alongside a new time slider feature in Elastic Maps, all designed to facilitate quicker insights without compromising speed. The update allows users to compare data across different time periods, apply creative customizations to metrics, and edit geo indices, enhancing the data storytelling experience. Elastic Agent, now generally available, simplifies data collection and protection from multiple sources, offering over 100 pre-loaded integrations with Kibana assets for a seamless data-to-dashboard experience. Available on Elastic Cloud, these features enhance data exploration and visualization, allowing users to define custom formulas, edit maps, and rapidly pivot between visualizations, while Elastic Agent and Fleet streamline data ingestion. Existing users can access these features through the Elastic Cloud console, and new users can explore them with a free trial or via downloadable self-managed versions, as detailed in the Kibana 7.14 release notes and Elastic 7.14 announcement post.

Elasticsearch 7.14 introduces the match_only_text field type, designed to reduce disk space usage in logging datasets by approximately 10% compared to the traditional text field type. This new field type is particularly beneficial for logging use cases, as it indexes only a subset of the information, thereby decreasing CPU and disk space requirements without sacrificing the ability to query data interactively. Although relevancy scores and span queries are not supported, and phrase and interval queries run slower than on text fields, other queries perform equally or even slightly faster. The release follows a recent trend of index size reductions, including a similar 10% decrease in version 7.10 through improved stored fields compression. By upgrading to the latest version of the Elastic Stack, users can leverage these space-saving benefits, which are achieved by eliminating the indexing of length normalization factors, term frequencies, and positions that are not crucial for log analysis. Match_only_text also employs runtime field concepts to handle phrase queries efficiently when necessary, ensuring better performance than a linear scan.

Elastic Enterprise Search 7.14 introduces key enhancements, including the integration of App Search and Workplace Search into Kibana for a unified management interface. This update offers precision tuning for search results, flexible data ingestion, and synonym support to improve search relevancy and user experience. Users can benefit from Kibana's advanced visualization tools to gain insights into search behaviors and make informed decisions directly within the platform. The release also maintains the standalone Enterprise Search application, ensuring continuity for existing users, while encouraging exploration of the new Kibana experience. Elastic Enterprise Search 7.14 is available on Elastic Cloud, providing both hosted and self-managed options, with additional resources like training courses and a free 14-day trial to support onboarding.

Elastic 7.14.0 introduces the industry's first free and open Limitless XDR, combining SIEM and endpoint security capabilities into a single platform, enhancing security operations by automating analytics, preventing ransomware, and enabling rapid threat response. This release includes Elastic Observability and Enterprise Search solutions, offering simplified telemetry collection and centralized management through Elastic Agent and Kibana. Elastic Agent unifies data collection and protection across diverse sources, making it easier to manage and integrate data while securing endpoints. Elastic Cloud now supports Azure Private Link, providing secure network connectivity without internet exposure, and plans to expand this feature to Google Cloud. Elastic 7.14 is available on Elastic Cloud and as a self-managed option, offering improved visibility, security, and search capabilities across the Elastic Stack.

Elastic Security 7.14 marks the debut of the first free and open Limitless XDR solution, combining the features of SIEM and endpoint security to enhance threat prevention, detection, and response. This release introduces Elastic Agent, enabling quick endpoint quarantine and inspection, alongside improved host inspection and query capabilities via Osquery Manager integration. It offers advanced ransomware and malware protection across various operating systems, such as Windows, macOS, and Linux, and enriches alerts with threat intelligence for better triage and investigation. Additionally, the release includes new machine learning jobs to identify unusual authentication events and a feature to detect spoofed URLs, bolstering security teams' proactive threat monitoring. Integration with Swimlane’s SOAR platform further streamlines security workflows, while administration improvements like faster rule updates, role-based access control, and event filtering enhance the user experience. Users can explore these features on Elastic Cloud or through a free trial, with training resources available for new users.

Version 6.8.18 of the Elastic Stack has been released, offering fixes and minor enhancements across the stack's components, including Elasticsearch, Kibana, Beats, and Logstash. Users are encouraged to upgrade to this latest version to take advantage of these improvements. For a comprehensive list of changes and updates for each product within the stack, individuals are directed to review the 6.8.18 release notes.

Elastic has announced significant enhancements to its integration capabilities with the Elastic Stack, including the general availability of the Elastic Agent and Fleet. The Elastic Agent serves as a unified solution for observability and security, streamlining deployment and configuration processes by reducing the need for multiple installations. Fleet, a new Kibana application, allows centralized management of Elastic Agents, providing real-time monitoring, remote upgrades, and enhanced security features. Additionally, the new Integrations app in Kibana facilitates seamless integration of systems with the Elastic Stack, offering out-of-the-box log parsing, dashboards, and machine learning jobs. These improvements address challenges associated with data ingestion at scale, simplifying the deployment and management of agents across large infrastructures. The transition from the existing Beats system to Elastic Agent aims to enhance efficiency, although Beats will remain in use for specific scenarios. The enhancements are open-source, encouraging community collaboration and feedback.