April 2021 Summaries

The blog post by Thorben Jändling explores the process of centralizing event log collection using Windows Event Forwarding (WEF) and Windows Event Collector (WEC), highlighting the importance of setting up a WEC server to forward logs to Elastic Security. It explains the functionalities of WinRM and WS-Management protocol under Windows Management Instrumentation, detailing two modes of log forwarding: Source Initiated and Collector Initiated. The post outlines challenges and solutions in setting up WEF and WEC, emphasizing the use of a WEC Cookbook to navigate potential pitfalls. It discusses different strategies for managing event logs, such as creating new Channels on the WEC server to improve performance and organization. The text also mentions the role of Providers in defining Channels and suggests organizing logs by asset type to enhance access control and lifecycle management. The post provides guidance on configuring WEC subscriptions and highlights the automation capabilities offered by PowerShell scripts to streamline the setup process. It concludes by encouraging readers to utilize these tools and strategies to optimize their WEC server setup for efficient log management and security monitoring within an enterprise.

Version 7.12.1 of the Elastic Stack has been released, featuring a series of bug fixes and minor enhancements across its various components, including Elasticsearch, Kibana, Beats, Logstash, and Elastic Enterprise Search. Notably, the update addresses specific issues such as Kibana's inability to recognize a valid geo_shape index when creating a Tracking Containment alert and failures in updating or deleting sessions in non-default spaces. Users are encouraged to upgrade to this latest version to benefit from these improvements, and detailed information on all changes can be found in the release notes.

Elastic and Alibaba Cloud have successfully collaborated since 2017, having launched the Alibaba Cloud Elasticsearch service, which has grown to support over 10 petabytes of data. This partnership has been recognized with Alibaba Cloud being named Elastic's Ecosystem Partner of the Year, due to its role in promoting free and open technology and creating value for joint customers. The collaboration has expanded beyond traditional uses of data retrieval and log analysis into more business-oriented applications, attracting users from sectors like retail, logistics, finance, and manufacturing. The partnership thrives on a shared vision of integrating search technology with cloud-native solutions, providing a seamless, scalable experience for users without increasing team sizes. The relationship is celebrated as a model of partnership and co-prosperity, exemplifying the unity of open-source and cloud-native products, with both companies looking forward to continued success and further achievements in the future.

Elastic Cloud has announced its availability on Microsoft Azure in the South Central US (Texas) region, providing users with enhanced capabilities in enterprise search, observability, and security. This expansion allows for efficient searching of applications and workplace content, along with monitoring and analyzing application performance and security through centralized logs and metrics. Users have the flexibility to deploy Elastic Cloud across multiple platforms, including AWS, Google Cloud, and Azure, with a choice between managed services or self-management using automation tools. Azure customers can subscribe to the Elastic Cloud service through the Azure Marketplace, benefiting from integrated billing that combines Elastic Cloud charges with their existing Azure bill. Elastic Cloud offers a straightforward setup, allowing users to select their preferred region and cloud service provider, optimize deployments based on specific requirements, and provision workloads quickly.

Windows Event Logs are crucial for cybersecurity teams, and understanding how to effectively collect and process them can significantly enhance security efforts. This blog series explores various aspects of Windows Event Logs, starting with audit policies that determine which events are logged and moving towards building use cases like detection rules and reports. While audit policies help in logging security-relevant events, they do not cover all possible activities, necessitating the use of tools like Sysmon or Elastic's Endpoint Security for more comprehensive monitoring. Sysmon, a part of the Windows Sysinternals suite, can generate logs for low-level system calls, but it lacks warranty and support. In contrast, Elastic's Endpoint Security offers kernel-level event collection with vendor support, although it bypasses the traditional Windows Event Log system. Ensuring appropriate audit policies and enabling necessary event Channels are fundamental steps, but additional methods may be required for a more in-depth log collection. The series will further discuss central log collection with Windows Event Forwarding/Collector and emphasizes the importance of holistic data protection using Elastic Security.

The third round of the MITRE Engenuity ATT&CK® evaluations focused on the Carbanak and FIN7 threat groups, testing the effectiveness of security vendors against sophisticated adversary techniques, including living-off-the-land strategies. This year's evaluation introduced Linux systems for the first time, expanding the scope to critical business infrastructure. Elastic Security participated using its free and open capabilities, leveraging a unified architecture that integrates endpoint security directly into the Elastic Agent. The evaluation results were visualized using Kibana, showcasing the visibility and detection capabilities of various vendors. Elastic has emphasized its commitment to an open development approach, offering free resources and an evolving set of features, such as enhanced machine learning models and new data integrations. Despite not participating in the prevention portion of the evaluation, Elastic continues to advance its security offerings, with significant updates released since the evaluation period.

Elastic Observability offers a comprehensive solution for monitoring the health and performance of digital ecosystems by unifying logs, metrics, application trace data, and availability data under one platform. It facilitates the seamless integration of various data types through pre-built collectors for numerous data sources, and includes built-in alerting to monitor service level agreements (SLAs) and user experience scores. The blog provides a step-by-step guide to setting up Elastic Observability using the self-managed option, beginning with the download and preparation of Elasticsearch and Kibana, followed by enabling security measures such as role-based access control (RBAC) and SSL/TLS encryption. It covers the initial configuration, including starting Elasticsearch and Kibana, setting up security protocols, and loading sample data to explore Kibana's capabilities. Additionally, it highlights the potential to further enhance observability by centralizing logs and infrastructure metrics, with the option to leverage a managed service through the Elasticsearch Service on Elastic Cloud.

inq., a Pan-African cloud-based digital service provider operating in six countries, embarked on a digital transformation journey with Elastic to centralize and enhance its network monitoring across a geographically distributed infrastructure. Initially seeking a singular platform for diverse applications and environments, inq. leveraged Elastic’s scalable and customizable architecture to address the complexities of managing multiple monitoring tools and isolated databases. By deploying distributed Logstash and customizing Kibana dashboards, inq. achieved improved visibility and efficiency in monitoring its extensive network infrastructure, including routers, switches, and wireless stations. This transformation not only facilitated internal network management but also enhanced customer experience by integrating single-sign-on capabilities for network utilization insights. As a result, inq. expanded its use of Elastic to include Security Information and Event Management (SIEM) for predictive analytics and revenue capture, thereby positioning itself as the first Elastic Managed Security Service Provider (MSSP) partner in Sub-Saharan Africa, with plans to extend similar benefits to its customers and partners.

The blog post examines how attackers exploit Access Token Manipulation (ATT&CK T1134) in Windows environments to compromise Active Directory domains by leveraging the relationship between access tokens, logon sessions, and cached credentials. It explains various techniques attackers use, like stealing or creating new tokens to impersonate users and access network resources without needing to dump credentials, and highlights four common token manipulation attacks: using the NETONLY flag, Pass-The-Ticket, Pass-The-Hash, and Overpass-The-Hash. Each method involves manipulating access tokens or cached credentials to enable lateral movement within a network, often evading detection by security measures through sophisticated means such as using direct syscalls or manipulating security support providers. The blog underscores the importance of understanding these techniques for improving defense mechanisms and detecting unusual network logins or token-related activities, emphasizing the need for holistic security measures like those offered by Elastic Security.

Kibana 7.12 introduces a more streamlined navigation experience that enhances the efficiency of building dashboards by adopting a dashboard-first approach, allowing users to create and add visualizations directly from their dashboard workflow. The update includes a new saving experience that guides users to save their visualizations to either a new or existing dashboard, or the newly renamed Visualize Library for reuse, thus minimizing the time spent on managing visualizations separately. An added feature is the ability to see panels saved to the library at a glance, along with notifications for unsaved changes, helping users maintain focus and productivity. Efficiency is further improved with options like the ability to copy existing panels to new or different dashboards, and a new toolbar that includes shortcuts to frequently used actions. Users can experience these enhancements by trying out the latest version of the Elastic Stack or a free trial of Elasticsearch Service on Elastic Cloud, with opportunities to provide feedback through social media, the Kibana forum, and GitHub.

Elastic has been recognized as a Visionary in the 2021 Gartner Magic Quadrant for Application Performance Monitoring (APM), marking a significant acknowledgement of its modern and flexible approach to monitoring solutions. The recognition underscores Elastic's investment in expanding from a popular log analysis tool, known as the Elastic Stack, to a comprehensive observability platform that includes capabilities for infrastructure, application performance, and user experience monitoring. Elastic's platform is notable for its adaptability, allowing integration with new data sources, custom anomaly detection, and exportation to external tools, which has led many organizations to consolidate their monitoring needs with Elastic. The platform's deployment options are vast, with availability on major cloud providers and on-premises, ensuring visibility across hybrid and multi-cloud environments while managing costs through a resource-based pricing model. The free and open tier allows users to evaluate the platform's capabilities without limitations, providing proof of value before committing to a paid plan.

An adaptive business model centered on employee experience is crucial for building resilience and competitive advantage, as highlighted by a study from Forrester Consulting on behalf of Elastic. The global pandemic posed specific challenges for CIOs and IT leaders, who rapidly transitioned millions to remote work with limited resources. At Elastic, where a distributed workforce model has been in place since inception, the pandemic accelerated the adoption of new tools and business practices to enhance employee engagement and efficiency. The collaboration between IT and HR at Elastic emphasizes providing employees with necessary technology and support, as well as ensuring cybersecurity and mental health resources. Empathetic leadership and strategic investments in automation and simplification are vital for improving employee experience without incurring significant costs. As businesses anticipate a shift to hybrid work models post-pandemic, CIOs are poised to take a proactive role in reshaping employee experiences.

Elastic and Confluent have announced a partnership aimed at enhancing the integration between Apache Kafka and Elasticsearch, which are both integral to modern enterprise data architectures. This collaboration seeks to improve existing product integrations and co-develop new capabilities, making it easier for users to utilize the Elastic Stack and Kafka together. The partnership will focus on strengthening the native integration between Elastic Cloud and Confluent Cloud, providing seamless connections between Confluent's always-on data streams and Elastic's real-time search and analytics database. By investing in joint R&D resources, the companies aim to deliver a better integrated product experience, including improvements to the Confluent Cloud Elasticsearch Sink Connector and developing packaged solutions for specific use cases. This initiative builds on the longstanding relationship between Kafka and Elasticsearch, which has been widely used by customers like CSX and Kroger, and will enable new possibilities for organizations leveraging these technologies.

Elastic is significantly expanding its global presence by adding 100 inside sales positions across its office hubs in Austin, London, and Singapore. The company is not only focusing on hiring sales representatives but is also looking to fill various roles to support its growth, including Sales Development Representatives, Solution Architects, and Customer Success Managers. This expansion is part of a strategic initiative to better serve Elastic's customers by enhancing their ability to leverage Elastic's products and solutions on Elastic Cloud. The inside sales team, one of the fastest-growing at Elastic, provides employees with exciting career path opportunities, whether moving into field-based roles or cross-functional positions. Paul Appleby, Elastic's president of worldwide field operations, emphasizes the mentorship and collaborative environment offered to employees as they help customers transform data into business value. Elastic's unique company culture, which values humility and ambition, is highlighted as a key attraction for potential employees.

Building a facial recognition system using Elasticsearch and Python involves several key steps, including face detection, encoding facial features into a 128-dimension vector, and storing these vectors using Elasticsearch's dense_vector data type. The process starts with using Python libraries such as face_recognition and numpy to detect faces in an image and encode them into vectors. These vectors are then stored in an Elasticsearch index, which allows for efficient searching and comparison using functions like cosineSimilarity to find matches between different facial representations. The tutorial demonstrates how to set up the necessary environment with Python and Elasticsearch, create and index facial data, and perform searches to find matching faces. It also highlights the potential for integrating this system with advanced search queries in Elasticsearch, enabling complex use cases beyond basic facial recognition, such as combining geo-queries with cosine similarity for enhanced search capabilities.

Elastic Cloud has expanded its availability to Microsoft Azure's East US (Virginia) region, offering capabilities in enterprise search, observability, and security. Users can easily search apps, websites, and workplace platforms, monitor applications, and analyze logs and metrics for insights. Elastic Cloud provides flexibility in deployment, allowing users to run it on Amazon Web Services, Google Cloud, Microsoft Azure, or all three, with options for a managed service or self-management using built-in tools. Users can optimize deployments based on specific requirements, and provisioning is quick through the Elastic Cloud console. Microsoft Azure customers can also subscribe to Elastic Cloud via the Azure Marketplace, integrating usage charges into their Azure bill, with a free 14-day trial available for new users.

Elastic App Search's new web crawler simplifies the process of ingesting publicly available web content to make it instantly searchable on websites, but challenges can arise if pages are not indexed as expected. The setup involves deploying App Search, creating an engine, and configuring the web crawler with specific crawling rules to target desired content. A misconfiguration in the rules, such as the order in which allow and disallow rules are applied, can lead to issues where no documents are indexed. Troubleshooting involves using tools like Kibana to access detailed logs and identify errors, such as the rule_engine_denied message caused by improperly ordered rules. By adjusting the order of these rules and leveraging Elasticsearch's search capabilities within the Logs app, users can efficiently resolve issues and ensure the web crawler indexes the correct pages. The article concludes by encouraging users to try Elastic App Search with a free trial to explore web crawling capabilities firsthand.

Elastic's community spotlight for April 2021 highlights the stories and motivations of user group organizers from the Asia-Pacific and Japan region, emphasizing their roles in fostering knowledge sharing and engagement within the Elastic community. Sami Jan from Pakistan transitioned to using Elastic Stack for geolocation search services, focusing on educating users about its benefits despite its learning curve. Ashish Tiwari from Mumbai values the insights from meetups and is motivated by the opportunity to learn and assist others in the community. Jimit Rangras from Gujarat, a DevOps professional, appreciates Elastic's solutions for real-world problems and encourages new organizers to take initiative and engage with the community. Vinayak Malik from Delhi shares his journey of integrating Elastic Stack at Harman and emphasizes the importance of relatable learning experiences at meetups. Each organizer shares tips for hosting successful meetups and expresses their passion for technology, learning, and community engagement, highlighting their personal interests and professional backgrounds.

With the release of Elastic 7.12, Kibana's Discover feature now uses the fields API by default instead of reading from the _source, although the latter is still accessible via Advanced Settings. This change, building on updates in Elasticsearch 7.11, leverages the fields parameter to enhance data retrieval by utilizing both a document’s raw source and index mappings. The fields API offers benefits like handling multifields, field aliases, and consistent formatting of field values, thus simplifying complexities associated with _source and other data handling nuances. The introduction of runtime fields is one of the significant changes, allowing them to be treated like any other field type in Discover. Additionally, object and nested fields display differently, with improvements in handling multi-fields and leaf fields to provide clearer and more structured data presentation. These enhancements streamline the process of field retrieval, reduce confusion caused by discrepancies between ingested and mapped values, and allow for more flexible data handling without requiring deep expertise in Elasticsearch's underlying mechanisms.

Elastic Cloud enables users to efficiently manage their Enterprise Search deployments by offering features for time-based scaling to handle predictable traffic spikes, such as those during business hours. This can be achieved using the Elastic Cloud API, which allows for resizing deployments through API requests executed via cron jobs. Users can prepare deployment changes in the Elastic Cloud UI, which generates the necessary API calls for adjusting node numbers and memory allocation. By scheduling these API calls, users can ensure their deployments are ready to handle increased demand without waiting for traditional autoscaling triggers like high CPU usage. Additionally, Elastic Cloud's recently introduced autoscaling capabilities allow Elasticsearch and machine learning nodes to automatically scale with data growth, ensuring optimal performance and cost efficiency.