July 2016 Summaries

Setting up Elasticsearch with SSL/TLS and native authentication using Shield is a crucial step for securing data and managing users within a cluster. This process involves installing Elasticsearch and the necessary plugins, configuring the elasticsearch.yml file for both file-based and native authentication methods, and creating an admin user for initial access. Native authentication, managed via a REST API, is the default in Elasticsearch 5.x and simplifies user management compared to file-based methods. For SSL/TLS configuration, a Java Keystore is created using a wildcard certificate, which allows a single certificate to be used across multiple nodes in a cluster, with the caveat that risks associated with wildcard certificates must be considered. The setup ensures that Elasticsearch clusters are secure and ready for production, and feedback is welcomed to improve the security measures further.

The blog post discusses the evolution of Logstash output workers from version 2.1 through the upcoming 2.4 and 5.0 series, highlighting changes in performance and concurrency handling. Initially, Logstash had slow and safe defaults, which shifted to faster but less stable settings in version 2.2, causing some confusion and issues due to the introduction of the NG Pipeline. This version integrated Filter Workers and Output Workers into a single Pipeline Worker system, which altered the threading model and required users to adjust output worker settings for optimal performance. The post further explains the challenges faced when attempting to autoscale output workers, as many plugins were not thread-safe, leading to the decision to revert to a single output worker in subsequent versions. Looking ahead, Logstash aims to eliminate output workers entirely, adopting a 'shared' output model where concurrency is managed internally by each plugin, starting with the Elasticsearch output, to ensure both performance and correctness. Users are encouraged to consult the new Performance Tuning Guide and participate in community discussions for further improvements.

Vandis, a technology reseller in New York specializing in security solutions and infrastructure services, effectively utilizes Elastic's commercial products to enhance their operational efficiency and customer service. By integrating tools like Shield for security, Watcher for alerting, and Marvel for monitoring, Vandis is able to rapidly identify and address issues within customer networks, often before the clients themselves are aware of any problems. The integration with PagerDuty ensures that their team is notified of any events at any time, facilitating prompt response and resolution. An example highlighted is when Elasticsearch's speed in parsing logs allowed Vandis to inform a client of a network issue before the client's network operations center did. The use of these tools not only supports Vandis's technical operations but also demonstrates the value and efficiency of Elastic's solutions to their customers, resulting in strengthened client relationships and improved security management.

Elastic Cloud Enterprise (ECE) is an extension of Elastic Cloud, designed to address the needs of large enterprises that require centralized management of Elasticsearch and Kibana clusters while maintaining control over sensitive data. Unlike Elastic Cloud, which is hosted on AWS, ECE allows enterprises to deploy the Elastic Stack within their own data centers or private clouds, accommodating existing on-premises infrastructure investments. ECE employs a service-oriented architecture with Docker containerization to manage resources efficiently and securely, using components like Proxy, Allocator, Constructor, and Cloud UI for operations. It relies on ZooKeeper for consistent cluster-state coordination and ensures secure inter-service communication with Stunnel. This architecture provides scalability, high availability, and ease of management for enterprises, allowing them to consolidate cluster management, enforce uniform policies, and optimize hardware utilization without being tied to specific underlying infrastructure, making it adaptable to various deployment strategies.

JJ Food Service Ltd, a UK-based B2B food services company, utilizes the Elastic Stack for both log analytics and enhancing product search functionalities. By integrating Elasticsearch, Kibana, and Logstash, the company efficiently manages system logging and error detection, which allows for rapid response and improved system resilience. Innovatively, logs are piped directly into Redis Pub/Sub, bypassing file writing, and consumed by Logstash to feed Elasticsearch. The company employs Elasticsearch to power customer product searches, leveraging a microservices architecture that decouples search logic from server infrastructure. This setup enables flexible phased searches incorporating exact matches, store filtering, and fuzzy logic, significantly reducing zero-result instances and improving customer experience. The ongoing exploration of Elastic Stack's capabilities highlights the company's commitment to leveraging advanced technologies for operational excellence and customer satisfaction.

In 2016, Elasticsearch announced the removal of support for site plugins in its 5.0 release, citing security concerns and the platform's unsuitability as a web server. Site plugins, introduced in Elasticsearch 0.17, allowed users to create graphical interfaces for Elasticsearch using static files like HTML, Javascript, and CSS, but they accounted for two of the seven security vulnerabilities ever found in Elasticsearch. Instead of continuing this risky feature, Elasticsearch encouraged users to serve site plugins through external web servers and configure them to allow Cross-Origin Resource Sharing (CORS). Additionally, developers were advised to consider creating Kibana plugins, which offer server-side functionalities and avoid exposing Elasticsearch directly to the internet, thus aligning with the goal of minimizing security risks by restricting privilege escalation and file access.

Kibana versions 4.5.3 and 4.1.10 have been released to address issues with tile map visualizations after the discontinuation of MapQuest's tile service, which had been used by Kibana for map visualizations. The new versions now use Elastic's Tile Service by default, which does not require configuration but currently supports zooming only up to level 8 and provides less detailed tiles compared to the previous service. Users have the option to configure Kibana to use other leaflet-compatible tile services if they require more detailed maps or greater zoom capabilities. The switch to Elastic's Tile Service was necessitated by MapQuest's abrupt discontinuation of their direct tile access API, which led to broken map functionalities in Kibana. The Kibana team acknowledged this oversight and committed to improving internal processes to prevent similar issues in the future, while also providing immediate steps to ensure more reliable map services by moving away from default third-party providers.

Vandis, a New York-based technology reseller specializing in infrastructure services, successfully transitioned its Elasticsearch deployment from proof-of-concept to production within a tight eight-day timeframe, thanks to the close collaboration between Ryan Young, Vandis' Director of Engineering, and Jason Bryan, an Elastic Support Engineer. The support model at Elastic, characterized by personalized and proactive assistance, enabled Ryan to efficiently manage the deployment process, even while working remotely during the holidays. Jason's in-depth knowledge of the Vandis environment facilitated tailored solutions and fostered a strong, trust-based relationship that Ryan described as having an additional team member. The engagement exemplified the benefits of direct access to Elastic's engineering resources, allowing Vandis to address specific needs and expedite the implementation of desired features, significantly easing Ryan's workload and contributing to the project's success. Part 2 of the interview promises further insights into Vandis' utilization of additional Elasticsearch features such as Shield and Packetbeat.

Beats is a platform for creating lightweight, open-source data shippers that send various types of data to Elasticsearch for analysis. Notable Beats include Packetbeat for network traffic, Filebeat for server logs, and Metricbeat for system metrics. The article provides a step-by-step guide on building a custom Beat using the libbeat framework and the Beat Generator package, specifically focusing on creating "lsbeat," which indexes information about files and directories, similar to the Unix command "ls." The process involves setting up a Golang environment, generating a Beat skeleton with the Beat Generator, configuring parameters such as the event period and directory path, and implementing necessary code, including the Beater interface with Run() and Stop() functions. The guide also covers setting up dependencies, configuring Elasticsearch mappings, and running the finalized Beat, demonstrating the integration with Elasticsearch and Kibana to verify data indexing.

Beats are lightweight, open-source agents designed to collect and feed data to Elasticsearch, playing a crucial role in the Elastic Stack for telemetry data capture and transmission. They support a variety of use cases, such as intrusion detection, user behavior analysis, and network monitoring, by efficiently acquiring data from diverse sources. Available on multiple operating systems, Beats include specialized variants like Filebeat for log data, Metricbeat for system metrics, Packetbeat for network protocols, and Winlogbeat for Windows event logs. The article details a practical guide on setting up Beats, including Topbeat and Packetbeat, to work with an Elasticsearch cluster on AWS EC2, illustrating the straightforward nature of configuring centralized machine resource monitoring using these tools. Additionally, it introduces Libbeat, a framework for creating custom Beats, and highlights the growing Beats community contributing new monitoring solutions.

Return-Oriented Programming (ROP), once a prevalent technique for bypassing security measures like Data Execution Prevention (DEP), is becoming less relevant as attackers develop more sophisticated methods for exploiting vulnerabilities. Despite the existence of ROP defenses, such as Microsoft's Enhanced Mitigation Experience Toolkit (EMET), attackers have shifted towards exploiting Address Space Layout Randomization (ASLR) weaknesses by using techniques that allow for memory reading and writing, making ROP unnecessary. New approaches, like Microsoft's Control Flow Guard (CFG) and Endgame's Hardware Assisted Control Flow Integrity (HA-CFI), focus on preventing exploitation at earlier stages, rather than the post-exploitation stage where attackers often succeed. These innovations aim to thwart attacks by monitoring and controlling the execution flow more effectively, highlighting the need for evolving defense strategies in response to the changing landscape of cybersecurity threats.

Efficient management of time-based indices in Elasticsearch is crucial for handling log events, and the traditional index-per-day pattern, while straightforward, can be inefficient. The new Rollover Pattern offers a more effective solution by utilizing aliases and APIs to manage active and inactive indices. This approach allows for optimal resource usage by spreading active index shards across hot nodes and rolling over to new indices when they become too full or old. The rollover process is facilitated by the rollover API, which checks conditions such as maximum documents or age to determine when to create a new index. Shrinking inactive indices onto cold nodes further optimizes storage, making them read-only and reducing them to a single shard with deflate compression. By force-merging these inactive indices and adjusting the number of replicas, users can maintain redundancy and manage indices efficiently. The process also includes deleting old indices based on field statistics, ensuring data management remains streamlined. Future enhancements, such as automation through tools like Curator and X-Pack, aim to simplify these tasks by providing scheduling and notification features.

Kibana versions 4.5.2 and 4.1.9 have been released to address a low severity buffer overflow issue by updating the bundled version of Node.js. Users are advised to upgrade to these latest versions promptly to ensure their systems are secure, and the updated versions are available for download.

Elasticsearch 2.3.4, based on Lucene 5.5.0, has been released as a bug fix update, available on Elastic Cloud and recommended for all 2.x users. Key improvements include resolving a potential corruption issue with the transaction log following disk full exceptions, enhancing the plain highlighter for geo_point queries, and various updates across related products such as Marvel, which now allows viewing stale monitoring data and correctly counts relocating shards, Shield, which addresses a memory leak by making the index search wrapper a singleton, and Watcher, which ensures the accurate indexing of watch states. Users are encouraged to download the release, provide feedback, and report any issues via Twitter or the GitHub issues page.

Cybersecurity defenses based on signatures and Indicators of Compromise (IOCs) are increasingly inadequate against modern threats, as adversaries continuously adapt their techniques and infrastructures, often using encryption and legitimate tools to bypass traditional defenses. Signatures, while still valuable for identifying unsophisticated attacks and aiding in understanding the extent of known intrusions, fall short due to their specificity and the evolving nature of threats. To enhance detection and prevention, it's crucial to incorporate additional layers of defense by monitoring low-level attacker techniques within operating systems and conducting threat hunting for suspicious patterns across systems. This approach involves leveraging frameworks like Mitre's ATT&CK to identify common adversary techniques, employing automation, analytics, and machine learning to scale threat hunting, and combining these methods with traditional signature-based detections. By integrating these strategies, organizations can better detect unknown intrusions and increase their chances of thwarting adversaries before significant damage occurs.

BA Insight, a partner of Elastic, has successfully integrated Elasticsearch with SharePoint to enhance enterprise search capabilities, particularly benefiting financial services and oil and gas sectors. By utilizing Elasticsearch's real-time indexing, GeoSearch, and complex query support, the company has improved search efficiency and capability beyond SharePoint's native features. This integration is showcased through applications like a Fund Management Portal, which provides personalized dashboards and utilizes Kibana visualizations. The company has developed 55 pre-built connectors for Elasticsearch, allowing diverse content searches across different systems. Looking forward, BA Insight is excited about developments like Elastic's X-Pack and Graph API, aiming to leverage these for improved text analytics and expanding their reach across non-SharePoint environments. The company sees potential in contributing to and learning from the growing Elasticsearch community, while exploring new opportunities for product use beyond traditional platforms.

LateRooms.com utilizes Elasticsearch to enhance their search and autocomplete functionalities, providing a more efficient way for customers to find hotels. The company's transition to Elasticsearch from SQL Server aims to deliver more flexible and targeted search solutions. The autocomplete feature, powered by Elasticsearch's Completion Suggester, allows rapid indexing and matching by storing destination data and hotel information in a single index, resulting in response times averaging 15ms. The search functionality employs two main indexes for destinations and hotels, with data such as geo-locations and hotel attributes. To address challenges in mapping hotel locations, LateRooms.com uses multiple shapes for each destination and conducts A/B testing to optimize search results. Despite ongoing reliance on legacy systems for filtering and sorting, the company plans to integrate these functions into Elasticsearch to improve search speed and efficiency. The adoption of Elasticsearch facilitates advanced features like geo polygon searches, enabling LateRooms.com to focus on their hotel recommendation engine while leveraging Elasticsearch's robust search capabilities, ultimately enhancing the user experience.