July 2017 Summaries

Docker containers have transformed application deployment with benefits in scalability and reliability, but they also present challenges due to their ephemeral nature. To address these challenges, a new processor called add_docker_metadata has been implemented in Filebeat, enriching logs and metrics with Docker metadata to enhance visibility into infrastructure and applications. Filebeat can be configured to monitor Docker logs stored in specific directories, enabling the shipping of enriched logs to Elasticsearch for analysis. Once logs are in Elasticsearch, they can be explored using Kibana, where additional context from Docker, such as container name, ID, image, and labels, is available. This feature simplifies user experience by automating the complexity of managing dynamic containerized environments and invites user feedback for further refinement.

The concept of bucket span in machine learning for Elasticsearch is crucial for dividing continuous streams of data into manageable batches for processing, especially in anomaly detection tasks. With the release of Elastic Stack 5.5, a bucket span estimator was introduced as an experimental feature, aiding users in determining the minimum viable bucket span based on a subset of their data. The bucket span determines the frequency of data analysis and alerts, affecting the system's sensitivity to anomalies and balancing between detecting anomalies quickly and avoiding excessive noise. Choosing the optimal bucket span involves considering data characteristics, anomaly duration, and processing performance, with shorter spans detecting anomalies faster but potentially introducing noise, and longer spans smoothing data but possibly missing significant anomalies. Once set, the bucket span cannot be changed during an analysis as it would require recalibrating the entire model, emphasizing the importance of selecting an appropriate span initially.

Elasticsearch 6.0 introduces strict content-type checking for REST requests, a shift from previous versions where the content-type was optional and the server would attempt to guess the appropriate type if it was missing or unrecognized. This change is aimed at enhancing clarity and security by ensuring that each request explicitly states its content type, thus reducing unexpected errors and protecting against Cross Site Request Forgery (CSRF) attacks. While this adjustment requires users to include the correct content-type for request bodies, it aligns with Elasticsearch's commitment to providing more predictable and reliable operations. The update may necessitate adjustments for those using third-party clients or command-line tools like curl, but it is seen as a valuable step in bolstering the security and consistency of Elasticsearch's functionality.

Elasticsearch has announced the release of version 5.5.1, which is based on Lucene 6.5.1, as well as a bug fix update for Elasticsearch 2.4.6, marking the final release in the 2.x series. The latest stable release, Elasticsearch 5.5.1, is now available for deployment on Elastic Cloud, which provides Elasticsearch as a service. Users are encouraged to download the new version, explore its features, and provide feedback via Twitter or the Elasticsearch forum, with any issues being reported on the GitHub issues page.

"It depends" is a common phrase in technical support at Elastic, often used to address complex questions that require detailed explanations rather than simple answers. Although this phrase can initially seem negative or evasive, Elastic uses it as a starting point for in-depth discussions, providing thorough explanations and context to ensure that the person asking fully understands the issue at hand. This approach reflects Elastic's culture of transparency and empathy, where employees are encouraged to take the time needed to help others grasp complex technical concepts, even if it means staying late or canceling meetings. Elastic's unique support culture values detailed, well-informed guidance over quick, superficial answers, aiming to empower individuals to handle future challenges independently.

In Logstash 5.1, the Persistent Queue (PQ) feature was introduced as a beta and later became a production-ready feature in version 5.4, offering disk-based resiliency to mitigate data loss during application failures. Unlike the default in-memory queue, PQ stores input data on disk, ensuring that unacknowledged data is replayed upon restart, thus providing At-Least-Once delivery guarantees. While it helps prevent data loss during application crashes and manages data ingestion spikes without backpressure, PQ does not protect against storage hardware failures unless external replication technologies are used. The feature can impact performance, with tests showing about a 10% reduction in throughput compared to in-memory queues, depending on the specific configuration and hardware. PQ's design includes settings like queue.page_capacity, queue.max_events, and queue.max_bytes to manage disk usage, along with checkpoint settings that balance reliability and performance. The feature is particularly beneficial in maintaining data integrity during abnormal shutdowns, though it may result in duplicate data processing due to its At-Least-Once delivery approach. Users are encouraged to test the PQ on their configurations to assess its performance impact and to provide feedback for further improvements.

Process injection is a prevalent defense evasion method often used in malware and fileless adversary techniques, involving the execution of custom code within another process's address space to enhance stealth and sometimes persistence. The article surveys ten process injection techniques, including Classic DLL Injection, Portable Executable Injection, Process Hollowing, and Thread Execution Hijacking, among others, each with distinct methods of injecting and executing malicious code. These techniques can involve writing code directly into a process or forcing a process to load a malicious library, with methods like DLL injection using function calls such as CreateRemoteThread or LoadLibrary, and more covert approaches like Reflective DLL Injection that do not rely on additional Windows APIs. The article highlights the ongoing development of new techniques by attackers, emphasizing the need for continuous research and adaptation in security solutions, as exemplified by Endgame's work in integrating advanced detection and prevention measures against such techniques.

In the context of Elasticsearch v5.5, anomaly detection through Machine Learning in X-Pack is enhanced by integrating with the Alerting feature, also known as Watcher, to notify users of anomalies detected in real-time. This integration is facilitated by an easy-to-use option that allows users to create "watches" for ML jobs directly from the user interface, enabling alerts based on defined intervals and anomaly severity thresholds. Users can further customize these watches through the Watcher UI, which supports editing, testing, and simulating alerts. The interaction between ML and Alerting is flexible, with capabilities to create custom watches using Elasticsearch indices or the ML results API, giving users the option to target different levels of anomaly data (bucket, record, or influencer). The article emphasizes maintaining appropriate time intervals for watches to ensure anomalies are captured effectively, and it provides insights into the creation and management of watches, noting the importance of considering factors like interval settings and throttle periods to avoid missing or duplicating alerts.

The article delves into the development of the chatbot Artemis within the Endgame platform, emphasizing the challenges of transitioning from query syntax to natural language understanding (NLU) to create a conversational interface. It discusses the intricate components of chatbot architecture, including utterance, dialog scripts, context, action, intent, and entities, which are essential for interpreting user inputs and executing actions. The NLU pipeline, incorporating processes like tokenization, parts-of-speech tagging, and entity extraction, is crucial for understanding user intentions and processing queries effectively. The article underscores the importance of combining NLU with user experience design to provide a functional and efficient tool for security analysts and operators, illustrating how these components work together to enhance Artemis' capabilities in facilitating intelligent data searches and hypothesis testing.

Elasticsearch is presented as a practical tool introduced in a master's class on Information Retrieval and the Semantic Web, designed to help students understand and use it effectively through a hands-on lesson. The post aims to familiarize readers with Elasticsearch's basic features by guiding them through a tutorial involving the addition, retrieval, and deletion of documents, as well as searching with its REST API. The tutorial uses a Shakespeare dataset to demonstrate more advanced search and aggregation functions, emphasizing the Query DSL and the Bulk API for batch operations. Although it doesn't cover configuration or best practices for production deployments, it serves as an entry point for users to explore Elasticsearch’s capabilities. The document also highlights Elasticsearch's flexibility and extensive features, such as relevance scoring, security, monitoring, and integration with other tools like Kibana, Logstash, and Beats, encouraging users to explore its comprehensive documentation and existing capabilities before developing custom solutions.

Effective management of IT operations has traditionally relied on the expertise of staff to interpret operational data, but the integration of Elasticsearch and machine learning presents a new paradigm that enhances efficiency and accuracy. By leveraging Elastic's machine learning capabilities, IT teams can automate the identification of anomalies in server, application, and network infrastructure data, which streamlines search, reporting, and alerting processes. This approach overcomes the limitations of static threshold-based alerts by learning normal behavior patterns and dynamically adjusting alerts, thereby reducing false positives and the time needed for root cause analysis. Elastic's machine learning tools, integrated with the Elastic Stack, help operations teams monitor key performance indicators, detect unusual activities, and provide insights into the factors contributing to anomalies. This not only accelerates issue resolution but also optimizes system changes by validating their impact on application performance. Elastic's machine learning is particularly suited for time-series data, providing significant value in analyzing logs, application metrics, and network flows, and is accessible via the Kibana interface as part of Elastic's X-Pack, supporting IT operations in becoming more proactive and responsive.

This week's update on Elasticsearch and Apache Lucene covers a range of enhancements and developments designed to improve performance, reliability, and functionality. Elasticsearch 6.0 introduces a safeguard against nodes running out of disk space by halting write requests when space is tight, and it enhances sequence number functionality for better data recovery and replication. The aggregation rewriting feature aims to simplify and cache aggregations more efficiently, while a new Java High Level REST Client documentation is in development. Cluster monitoring has improved with better alert caching, reducing script compilation issues for users with multiple clusters. A more compact representation of document IDs reduces storage overhead, though the impact on index size might be minimal due to existing compression techniques. Several breaking changes in both the 5.x and master branches of Elasticsearch involve updates to REST endpoints, data mappings, and search functionalities. Meanwhile, Apache Lucene is preparing for a feature freeze on version 7.0 and has made strides in version compatibility checks, concurrent delete/update improvements, and discussions on query performance and safety. The blog promises ongoing updates and insights into the Elastic ecosystem.

The 5.5.0 release of Elasticsearch for Apache Hadoop (ES-Hadoop) brings several updates, including reintroduced support for Kerberos in the Elasticsearch HDFS Repository Plugin, which was previously dropped due to security model challenges. This release also introduces a new delimiter for index formatting, allowing both colon and pipe characters, with the colon set to be deprecated in future versions. It warns of the impending removal of support for TTL and Timestamp mapping configurations in Elasticsearch 6.0, while still maintaining them for backward compatibility with Elasticsearch 5.x and below. Furthermore, support for Hadoop 1.x is deprecated, with complete removal planned for version 6.0, aligning with the broader ecosystem's shift to Hadoop 2.2 and above. Additionally, the Elasticsearch on YARN project, which was a beta experiment for deploying Elasticsearch on YARN, is deprecated due to YARN's lack of support for long-running services, redirecting focus to other enhancements within ES-Hadoop.

Elasticsearch 5.5.0, based on Lucene 6.6.0, is the latest stable release available for deployment on Elastic Cloud, introducing a security fix for X-Pack Security users and a Windows MSI Installer, which simplifies setup with a graphical user interface. The update addresses a sensitive information leak issue in X-Pack Security, urging users to upgrade to prevent confidential data exposure through the _nodes API. It also includes support for ICU Collation Keyword fields for language-specific sorting, new ip_range field types for IPv6 range indexing, and enhancements to terms aggregations for reduced memory usage, preparing for future changes like the one-type-per-index shift in version 6.0. Users are encouraged to download and provide feedback on the release through social media and the Elastic forum.