April 2023 Summaries

Elastic Security has enhanced its offerings by providing over 1,100 prebuilt detection rules to enable quick setup for security monitoring, with a significant portion dedicated to SIEM detection across multiple log sources and endpoint security. The company emphasizes its commitment to transparency and collaboration with the security community by publicly sharing its detection logic on GitHub, allowing for communal learning and improvement. Elastic's Threat Research and Detection Engineering (TRADE) team focuses on emerging threats, developing detection and prevention rules, and maintaining high-quality content through continuous monitoring and tuning. This effort includes the creation of investigation guides that enrich security alerts with operational context, aiding analysts in triage and investigation tasks. Elastic also maps its detection rules to the MITRE ATT&CK framework, ensuring comprehensive threat coverage. The company supports community engagement through open-source contributions and provides Red Team Automation scripts for testing detection rules.

Elastic Enterprise Search enables users to create App Search engines for indexing documents with adaptable search capabilities, but its default language support can be limited. To include additional languages, users can establish an App Search engine with custom analyzers tailored to the preferred language. This process involves creating an Elasticsearch index with specific analyzers and mappings, updating the index to use these analyzers, and reindexing documents to ensure accurate language processing. For example, the blog post outlines how to add Romanian language support by incorporating language-specific stem and stopword filters. This customization ensures that search queries recognize linguistic nuances, such as plural forms, enhancing search accuracy and feature availability like Precision Tuning. The methodology discussed can be applied to support any language that Elasticsearch's analyzers accommodate, providing a robust framework for multilingual search optimization.

Philipp Kahr and Wei Wang explore the effectiveness of machine learning models in detecting anomalies in online chess games, specifically focusing on the Ponziani opening on the Lichess platform. They compare models using absolute values versus percentages, highlighting the limitations of total value-based models, which can be skewed by the overall trend in games played. By calculating the percentage of games featuring the Ponziani opening, they demonstrate improved anomaly detection, offering insights into significant events like the surge in games during the COVID-19 pandemic and the influence of popular chess content creators. The authors detail the process of configuring these models using Kibana Machine Learning, emphasizing the benefits of using percentages for more accurate anomaly detection. They conclude by encouraging readers to explore Elastic Cloud for implementing similar machine learning tasks and suggest exploring related posts in their chess series for further insights.

The State and Local Cybersecurity Grant Program (SLCGP), overseen by the Cybersecurity and Infrastructure Security Agency (CISA), has announced an increased funding of $379 million for the second year, up from $185 million in the previous year, with applications open from August 7 to October 6, 2023. This funding requires state and tribal governments to develop comprehensive cybersecurity plans, focusing on key practices like multi-factor authentication, enhanced logging, and data encryption. A significant portion of the funds must be allocated to local and rural areas, and plans must be approved by a cybersecurity committee and relevant state officials. Elastic offers a unified data platform to aid in the implementation of these plans by providing tools for data consolidation, threat detection, and continuous monitoring across IT infrastructures. This approach helps reduce risks of data breaches, ransomware attacks, and phishing scams, as evidenced by the successful application in the State of Arizona's Enterprise Security team.

Elastic has joined the Joint Cyber Defense Collaborative (JCDC) in the United States, an initiative led by the Cybersecurity and Infrastructure Security Agency (CISA) to fortify national cyber defense by uniting public and private sector efforts. The JCDC, established in 2021, aims to prevent security breaches and coordinate defensive actions through the sharing and analysis of cyber risk information. Elastic, a long-term partner of CISA, brings its expertise in data search and analysis to the collaboration, emphasizing the importance of data visibility and access across IT environments to combat sophisticated cyber threats such as ransomware and espionage. Under CISA Director Jen Easterly's leadership, the JCDC seeks to leverage collaborative data to protect against cyber attacks from criminal organizations and nation-states, aligning with national goals outlined in President Biden's Executive Order on Improving the Nation's Cybersecurity. Through its involvement, Elastic is committed to transforming data into actionable insights, contributing to economic prosperity, national defense, and public safety by addressing widespread security flaws and configuration issues in digital technologies.

Security teams face challenges due to the increasing complexity of the threat landscape and the overwhelming amount of data and alerts, which strains their resources. The integration of Elastic Security and Recorded Future threat intelligence aims to enhance Security Operations Center (SOC) workflows by streamlining threat detection and investigation processes. Elastic Security combines security information and event management (SIEM), endpoint security, and cloud security on an open platform, enabling teams to detect, protect, and respond to threats at scale. It uses flexible data tiers for real-time and retrospective analysis and supports proactive threat protection. Recorded Future provides comprehensive threat intelligence by combining automated data collection with human analysis, offering context and insights that help analysts prioritize responses and reduce alert fatigue. The integration allows for seamless access to threat intelligence, enabling analysts to correlate data, triage alerts efficiently, and gain meaningful insights into threats. This collaboration helps security teams proactively hunt for signs of compromise and reduces the dwell time of attackers, ultimately boosting analyst confidence and enhancing organizational security measures.

Containers and microservices have revolutionized software development by enabling faster application deployment and better resource utilization but have introduced significant security challenges. Despite efforts to enhance security by shifting left, which involves integrating security earlier in the development process, runtime security remains crucial because vulnerabilities can still emerge post-deployment. Elastic addresses this by offering a novel, agent-based Container Workload Protection (CWP) solution that provides real-time security insights and protections. This solution includes features such as lightweight data collection using eBPF, drift protection policies, and visibility tools like Session Viewer to help organizations detect and prevent unauthorized changes in containerized environments. Elastic's approach aims to simplify container security management, allowing teams to quickly respond to threats, ensure compliance, and maintain robust defenses against potential attacks.

Elastic has announced its contribution of the Elastic Common Schema (ECS) to OpenTelemetry (OTel) to promote the adoption of OTel-based observability and security. This initiative aims to standardize data from varied sources, facilitating improved analysis, visualization, and correlation across observability and security solutions, thereby expediting root cause analysis. By integrating ECS with OTel, Elastic intends to establish a mature common schema for metrics, logs, traces, and security events, enhancing operational efficiency and reducing data management costs. The collaboration will also support vendor-neutral semantic conventions and extend OTel's capabilities to include infrastructure support for Kubernetes application logs and other data types. This partnership reflects Elastic's commitment to open standards and its ongoing involvement with CNCF projects, aiming to establish OTel as a standard in the industry, which will benefit both the Elastic and broader OTel communities.

At KubeCon Europe, Elastic announced that its open-source project, Elastic Common Schema (ECS), will be contributed to OpenTelemetry (OTel) under the Cloud Native Computing Foundation to work towards a unified schema for observability and security data. This convergence aims to standardize data formats across security and observability platforms, enhancing data analysis, visualization, and correlation. By integrating ECS with OTel's Semantic Conventions, the initiative seeks to reduce vendor lock-in, streamline data transformations, and improve root cause analysis and operational visibility. Elastic ensures users that their investments in ECS will be preserved, offering guidance and tools for migration to the new schema. OpenTelemetry, a significant project in the CNCF ecosystem, benefits from ECS’s mature schema, particularly in structuring logs and security events, thus enhancing its utility for security use cases. Elastic continues to support OTel natively, allowing seamless integration into its APM capabilities, with no changes to ECS's Apache 2.0 licensing.

Flightwatching, a technology company based in France, helps aircraft carriers optimize operational maintenance costs and reduce environmental impact through a real-time digital platform. By using Elastic's technology for data storage, search, and analysis, Flightwatching enables carriers to achieve significant fuel savings and CO2 emissions reduction, along with predictive fleet maintenance improvements. Currently monitoring data from 600 aircraft, the platform helps customers create efficient dashboards and perform anomaly detection, leading to cost savings of over $100,000 per aircraft annually. Through partnerships with leading carriers, Flightwatching has become a key player in enhancing operational efficiency and sustainability, with plans to expand its capabilities by integrating more features like machine learning.

The article by George Teas discusses the traditional secretive and proprietary approach to intelligence analysis, which often involves reliance on external teams and closed data platforms. This can lead to a loss of control over data and analysis, making government agencies vulnerable to market and political fluctuations. In contrast, a transparent and collaborative approach rooted in open data platforms offers flexibility, cost savings, and increased resilience, enabling agencies to maintain control and adapt quickly to changing needs. This method enhances public trust and operational efficiency by democratizing data access and promoting cross-team collaboration. The article highlights the benefits of using open-source technology, like Elastic, which is supported by a community of experts and partners dedicated to continuous innovation and support, providing a robust foundation for intelligence analysis.

Elastic has been awarded the 2023 Data Breakthrough Award for DataOps Platform of the Year, highlighting the extensive capabilities of its Elasticsearch Platform. This platform is designed to manage the full data lifecycle across various environments, including on-premises, hybrid-cloud, and multi-cloud, enabling organizations to extract real-time insights from massive data volumes. As companies grapple with the increasing demands of data influx, they often turn to multiple point solutions, which can lead to inefficiencies and increased costs. Elastic proposes that these challenges can be effectively addressed through its comprehensive search capabilities, as evidenced by over 3.6 billion downloads of Elasticsearch. The platform has evolved from a search and analytics engine into a flexible data insights platform, offering solutions like Elastic Observability and Elastic Security, and allowing for the creation of custom applications. Elastic aims to transform data usage by optimizing infrastructure, enhancing content connectivity, and strengthening digital security.

Observability is increasingly critical for effective security implementation as it enables organizations to monitor systems, applications, and networks in real-time, detect security incidents, and respond swiftly. A unified observability solution consolidates data sources into a single platform, providing security teams with comprehensive visibility and control over their infrastructure, which helps in identifying and mitigating potential issues before they become critical. Observability and security are closely linked, with observability enhancing security by offering insights into system behavior and potential threats, while security protects the integrity of observability data. Utilizing observability data helps in detecting anomalies, improving security posture, and providing context for incident response, with machine learning further enhancing capabilities by enabling faster detection and response to threats. As software systems become more complex and cyber threats grow, the integration of observability and security will be essential in creating a robust and secure IT environment.

The blog post explores the impact of YouTube chess tutorials on the popularity of specific chess openings, particularly focusing on the influence of popular streamers like GothamChess and Hikaru Nakamura. By analyzing over 4.3 billion games on the Lichess platform using the Elastic Stack, the post demonstrates how to calculate p-values to detect significant changes in the usage of openings like the Ponziani and London System. The findings suggest a notable increase in the Ponziani opening's popularity following a tutorial by GothamChess, as indicated by a significantly low p-value, whereas Hikaru Nakamura's video on the London System did not show a similar effect. The analysis highlights the potential influence of streamers on chess trends, though it notes that the study's scope is limited to Lichess data and rated games, and does not account for other factors like tournaments or player ratings.

Elastic emphasizes the importance of company culture, encapsulated in a set of shared values known as the Source Code, which significantly influences employee interaction, job satisfaction, and productivity. Integral to Elastic's operations, these values are introduced during the hiring process and reinforced through onboarding. The Source Code highlights diversity, autonomy, authenticity, and flexibility, with recent additions like "Customer, 1st," underscoring the company's commitment to customer-centricity. Original values include "Home, Dinner," promoting work-life balance; "Space, Time," encouraging innovation; and "Progress, SIMPLE Perfection," which focuses on embracing growth over perfection. Other values like "01.02,/FORMAT" and "As YOU, Are" celebrate individual differences and authenticity, while "Humble, Ambitious" and "Speed, Scale, Relevance" reflect the company's ethos of humility, challenge, and delivering high-quality, scalable search solutions.

The blog post explores the cyclical nature of big data challenges, highlighting that issues such as metadata, distributed search, and dynamic categorization have persisted through generations due to inherent technological limitations. These recurring problems stem from our inability to effectively unify and analyze dispersed and large-scale data, often leading to temporary solutions that become obstacles over time. The text critiques traditional methods like metadata-based searches and federated systems, emphasizing their limitations in providing comprehensive data insights. It advocates for the adoption of distributed search, a relatively new paradigm made possible by advancements like Cross Cluster Search, which allows for fast, scalable, and comprehensive data access across various contexts. The author suggests that moving beyond outdated approaches will enable more effective data operations and analytics, addressing the core issue of accessing a unified data picture necessary for informed decision-making.

Elasticsearch 8 has introduced significant benchmark-driven optimizations to enhance scalability and performance, especially for high shard count use cases. Elastic has focused on managing increasing data volumes and higher shard counts while keeping resource usage in check. They have developed specific benchmarks to track improvements in indexing throughput and snapshot performance, utilizing nightly benchmarks to detect both regressions and advancements. By pivoting from bare metal to cloud-based environments for macro-benchmarking, Elastic has achieved notable enhancements, including almost doubling indexing throughput and significantly reducing latency in field capabilities API. Additionally, snapshot performance improvements have been realized, such as a 97% reduction in task time through decreased garbage collection activity, which allows for a larger default snapshot pool size in Elasticsearch 8.6.0. The rigorous benchmarking process ensures that any performance changes, whether positive or negative, are investigated to maintain the reliability and efficiency of Elasticsearch, providing an improved experience for Elastic Cloud customers.

Elastic Security for Cloud aims to address the unique security challenges posed by cloud environments by integrating security analytics with a Cloud Native Application Protection Platform (CNAPP). This approach consolidates various security capabilities, such as Cloud and Kubernetes Security Posture Management (CSPM and KSPM), Cloud Vulnerability Management, Cloud Workload Protection, and Container Workload Protection, into a single platform. Elastic’s platform provides a unified view of cloud risk, enhancing visibility and enabling organizations to manage security posture, detect vulnerabilities, and protect workloads across cloud-native applications. It offers features like continuous visibility, flexible policy enforcement, and integration with AWS services, while pricing is based on data consumption, making it cost-effective. The platform's capabilities are designed to facilitate collaboration between security and DevOps teams and provide comprehensive protection throughout the lifecycle of cloud-native applications. Existing Elastic Cloud customers have direct access to these features, and new users can start with a free trial to explore the platform's offerings.

Kibana dashboards, which facilitate quick and easy visualization of data indexed in Elasticsearch, can present performance challenges that are difficult to diagnose due to the UI abstraction. To address these issues, the blog presents four methods: Elastic Application Performance Monitoring (APM), Chrome Developer Tools, the Kibana Inspector, and the Elasticsearch Slow Log. Elastic APM helps identify slow performance by capturing errors and traces during dashboard navigation, particularly pinpointing issues with plugins. Chrome Developer Tools allow developers to step through the code to trace errors and unexpected behaviors. The Kibana Inspector provides access to queries executed by individual panels, offering insight into their performance, while the Elasticsearch Slow Log captures and logs slow queries that impact server performance, enabling identification of problematic dashboard controls. These tools collectively aid in diagnosing and optimizing dashboard performance by analyzing the execution times and resource usage of various components.

Financial institutions are increasingly adopting cloud technology due to its efficiency, security, and scalability, despite most banking workloads still being on-premise as of 2022. This shift is driven by the need to meet customer demands and leverage advanced data and analytics solutions, including AI. However, concerns about cloud security persist, even though cloud providers offer modern, secure environments with compliance and regulation certifications. The concept of shared responsibility in cloud security highlights the risks of client-side errors, with Gartner noting that 99% of cloud failures are due to customer mistakes such as misconfigurations. Cloud Native Application Protection Platforms (CNAPPs) are designed to help financial institutions improve their cloud security posture by automating the identification and remediation of misconfigurations, ensuring compliance, and providing visibility across multi-cloud environments. Elastic's security solution integrates CNAPP capabilities for AWS and soon for Google Cloud and Microsoft Azure, offering financial institutions a comprehensive tool to manage security across hybrid and multi-cloud setups, thereby reducing the risk of breaches and reputational damage.

Elasticsearch has introduced the frequent_item_sets aggregation, a technique for frequent item set mining, to identify patterns in large datasets, marking its general availability with version 8.7. This feature leverages the Eclat algorithm, chosen over the Apriori for its efficient use of resources through a depth-first approach, to scale better in runtime and memory. The process involves mapping and reducing phases where items and transactions are de-duplicated, encoded, and pruned based on minimum support parameters, optimizing the discovery of top-N frequent closed item sets. Despite being resource-intensive, the implementation incorporates various optimizations such as circuit breakers, async search, and filtering to mitigate its impact on runtime and resource usage. Frequent_item_sets can be integrated with other Elasticsearch functionalities, allowing users to discover data regularities, and is particularly utilized in AIOps Labs' Explain Log Rate Spikes for log analysis.

The Elastic Global Threat Report highlights credential access as a significant tactic in cybersecurity threats, representing about 10% of observed techniques, with a particular focus on on-premise Windows, Linux, and MacOS systems. The report identifies 17 MITRE ATT&CK techniques and 28 sub-techniques that enable the theft of credential materials like usernames and passwords, which facilitate unauthorized access to systems and data. The most prevalent method is OS Credential Dumping, where attackers extract credentials directly from the operating system using built-in utilities like reg.exe. The report emphasizes the importance of implementing a least privilege model and monitoring native system tools to prevent such attacks. Elastic has developed specific detection rules, with four key rules accounting for 73% of credential access detection events. The report also underscores the inevitability of credential-based threats due to the inherent capabilities within operating systems, urging enterprises to closely monitor user interactions with system utilities and registry access to mitigate risks.