June 2018 Summaries

Elastic APM Ruby Agent 1.0 has been officially released, marking its transition from beta to general availability, and allowing developers to integrate application performance monitoring into Ruby applications seamlessly. Elastic APM is an open-source solution within the Elastic Stack that provides insights into application performance, now supporting apps written in Ruby, Node.js, Python, JavaScript/RUM, Go, and Java. The Ruby Agent, in particular, can be easily incorporated into Ruby on Rails or other Rack-compatible servers by adding the 'elastic-apm' gem to the Gemfile and configuring it to report to an APM server. It automatically instruments incoming requests, database queries, outgoing web requests, and background jobs, while also enabling developers to add custom instrumentation for more detailed monitoring. The release invites the community to contribute feedback and feature requests through GitHub or forums, emphasizing the open-source nature of the Elastic stack.

A recent blog post explores methods to integrate system logs and metrics into Elasticsearch using Beats, specifically focusing on the System modules of Filebeat and Metricbeat. The blog provides a detailed walkthrough of deploying these modules to capture syslog and authentication logs from Unix-like systems, as well as system metrics like CPU and memory usage. It highlights the interactive capabilities of Kibana dashboards, which allow users to visualize and filter data efficiently. The article suggests setting up Elasticsearch and Kibana, either via Elastic Cloud or local installation, and emphasizes the ease of following Kibana's step-by-step instructions to enable data flow. Additionally, the post encourages proactive use of machine learning and alerts to enhance data monitoring and analysis, while assuring users of the straightforward process to extend logging to other services like NGINX without reinstalling Filebeat.

In Part 3 of a series on optimizing searches of Chinese, Japanese, and Korean text using Elasticsearch 6.2, Kiju Kim introduces a language detection technique to enhance efficiency and conserve storage space. By employing an ingest plugin for language detection, the method detects the language of the text and assigns it to a specific language field using the langdetect processor, which allows the original text to be copied into language-specific fields for analysis. This approach contrasts with the previously discussed multi-fields method, which stores text in multiple fields regardless of its language, potentially wasting resources. The language detection pipeline is demonstrated using content from the 2018 PyeongChang Olympic Games, showcasing how language-specific analyzers like kuromoji, smartcn, and openkoreantext-analyzer filter and process text. Although indexing with a language detector or multi-fields takes slightly more time than using a single field, the language detector method provides improved search results and storage efficiency, demonstrating a significant advantage over the multi-fields approach.

The article delves into the intricacies of monitoring Java applications using Metricbeat and Jolokia, focusing on how these tools can leverage Java Management Extensions (JMX) to extract valuable metrics from Java Virtual Machines (JVMs). JMX provides a framework for managing and monitoring Java applications via MBeans, which represent resources within a JVM. While JMX is traditionally accessible only to Java applications, Jolokia acts as an intermediary by exposing MBeans through a REST-like HTTP endpoint, enabling non-Java applications to access this data. By deploying Jolokia as a JVM agent or via other methods like WAR in Java EE environments, users can gather JMX metrics which Metricbeat can then collect and visualize through Elasticsearch and Kibana. The text also highlights advanced configurations such as grouping fields into different events, using wildcards for mapping multiple MBeans, and utilizing Jolokia Discovery for dynamic environments, which allows Jolokia agents to announce their endpoints and service information for more flexible monitoring setups.

In the June 25, 2018, edition of "Keeping up with Kibana," significant updates and developments in the Kibana project were discussed, including job openings and the introduction of the new Inspector feature, which replaces the visualization spy panels to provide more accurate data display and analysis capabilities. This update aligns with Kibana 7 and introduces different views based on the inspected element, but it is not stored in the URL and is unavailable in the Discovery tab. The design team has been busy enhancing the fixed positioning service, developing new components for Elastic UI (EUI), and updating the release script tool, with EUI now at version 1.0.0. The platform team continues to work on transitioning to a new server platform while addressing legacy plugin behaviors, with notable PRs including upgrades to RxJS 6 and improved plugin build failure messaging. Additionally, the sharing team worked on integrating a beta flag to the data module schema and transitioning dashboard components to TypeScript, with 17 PRs seeing activity over the past week, reflecting ongoing efforts to refine visualization features and address various technical issues.

In Kibana version 6.3, significant improvements were introduced to its query language, focusing on simplifying syntax and enhancing user experience by removing the complex functional syntax and refining the basic query syntax. This new language resembles the familiar Lucene query syntax, making it accessible for existing users, while introducing features such as autocomplete, which suggests fields, values, and operators to streamline query building. The changes are a direct response to user feedback and aim to unify the query and filter bars without causing confusion. Users can easily opt into the new language via the query bar's options, and further developments are anticipated as the language evolves, driven by ongoing user contributions and feedback.

Kibana 6.1 introduced an experimental visualization type called Controls, which allows users to add interactive inputs like drop-downs and range sliders to dashboards, enhancing data interactivity and customization. The blog post provides a step-by-step guide on using these features, starting with setting up the Elastic Stack using a dataset from France and creating a new index pattern in Kibana. Users can create drop-downs for filtering by region, populated through a terms aggregation, and range sliders for filtering by net salary, based on min and max aggregations. Additionally, Kibana 6.3 allows chaining of drop-down inputs, enabling selections in one input to filter options in another, demonstrated by linking region and town filters. The post also suggests combining these controls with other visualizations in dashboards for more dynamic data exploration and encourages users to provide feedback on the experimental feature to shape its future development.

Beats, a platform supporting container monitoring, has introduced Autodiscover in Filebeat and Metricbeat to enhance monitoring capabilities for Docker and Kubernetes environments. This feature allows users to define configurations that are dynamically launched by Beats to track the dynamic nature of containers, which often shift and lack fixed IP addresses. Autodiscover leverages events from providers like Kubernetes and Docker to trigger specific configurations based on condition matching, allowing for dynamic and responsive monitoring setups. With the introduction of hints in version 6.3, monitoring configurations can now be stored alongside application containers, empowering development teams to directly manage monitoring setups using Kubernetes Pod annotations or Docker labels. This hints-based approach simplifies the process by enabling flexible and context-specific monitoring configurations, making it particularly beneficial in multi-tenant environments. Users are encouraged to explore these features and provide feedback on their experiences with Autodiscover through the Beats forum.

In the June 18, 2018 update of "Keeping up with Kibana," several key initiatives and developments were highlighted, including the progress on the Canvas project, which is poised to be the first plugin to fully utilize Kibana 7's styling. The security team is advancing towards role-based access control, which will improve content organization for larger customers by enabling workspace segmentation. The design team is enhancing the visualizations editor, while also addressing issues related to tables and tabs. The Geo team has been working on the Elastic Map Service landing page and the GIS App. Additionally, there has been significant progress in index lifecycle management and rollup support in Kibana. The security team's first phase of the role-based access control project is under review, setting the stage for the introduction of Spaces, which will allow for more granular privilege management. The visualizations team has implemented explicit passing of filters and queries, improving application of global context and preparing for the support of roll-up indexes.

Vega is a powerful declarative grammar for creating visualizations, and its integration into Kibana 6.2 allows users to build rich visualizations utilizing Elasticsearch data. The tutorial provides a comprehensive introduction to using Vega, starting with simple examples and advancing to dynamic, data-driven graphs. Key concepts such as the use of marks, scaling, data transformations, and axes are explained, demonstrating how to position and style elements based on data inputs. It highlights the flexibility of Vega in handling various data types and transformations, including time-based line charts that leverage Elasticsearch queries. The guide also covers advanced topics like band scaling and dynamic data manipulation, showcasing the tool's ability to handle complex visualizations through customization and scaling techniques. Additionally, it encourages further exploration with links to documentation and examples, setting the stage for more in-depth discussions on handling Elasticsearch results and nested data in future posts.

The text explores the challenges and solutions for indexing and searching multi-language documents, particularly in Chinese, Japanese, and Korean, using Elasticsearch 6.2. It discusses the limitations of using a single field with a standard analyzer, which is not effective for languages with postpositions like Korean, and highlights the utility of multi-fields with language-specific analyzers like kuromoji for Japanese, smartcn for Chinese, and openkoreantext-analyzer for Korean. By implementing multi-fields, each sub-field can be analyzed by a dedicated language-specific analyzer, improving search accuracy across different languages. The document also mentions the potential use of a language detector to further enhance search capabilities in a multi-lingual context, which is to be covered in the subsequent part of the series.

Elasticsearch 5.6.10, based on Lucene 6.6.1, has been released as the latest stable version in the 5.x series and is available for deployment on Elastic Cloud. This release addresses a sensitive data disclosure flaw in the repository-azure plugin, where Azure credentials could be logged inadvertently if the plugin is set to TRACE level. Additionally, a bug affecting bulk requests could cause divergence between primaries and replicas under certain conditions when the master is under load, particularly if a document's mapping update times out or is not acknowledged. Users are encouraged to download and test Elasticsearch 5.6.10, and provide feedback via Twitter, forums, or GitHub.

Logstash has introduced a new Java execution engine, now in the production candidate stage, that promises faster performance, reduced memory usage, and significantly lower configuration startup and reload times in version 6.3.0. This engine, which can be activated with the --experimental-java-execution flag, optimizes the integration between Ruby and Java, crucial in performance-sensitive scenarios, and includes improved config parsing and bytecode generation. It offers significant performance enhancements, most notably in scenarios requiring high CPU and IO resources, by replacing the Ruby source code generation phase with direct compilation to JVM bytecode using Janino. The engine also optimizes the interaction points between Ruby and Java to minimize performance costs, although the Ruby execution engine is expected to be phased out eventually. These enhancements not only improve the current system's efficiency but also pave the way for future developments, like a new Java plugin API for Logstash, while providing incremental performance improvements across both Ruby and Java execution engines. Users are encouraged to upgrade from earlier versions and provide feedback on this new feature.

Elasticsearch 6.3.0, built on Lucene 7.3.0, introduces several significant enhancements and new features, including an experimental SQL feature aimed at making Elasticsearch more accessible to users familiar with SQL who wish to leverage its speed and full-text search capabilities. Additionally, the release includes rollup jobs for efficient data aggregation and a new support matrix accommodating Java 10, despite its short-term nature. The update addresses two security vulnerabilities related to information exposure and offers recommendations for managing these risks. A known issue with JDK 10 affecting Elasticsearch 6.3.x, particularly involving AVX-512 instructions, has been identified with a suggested workaround, and the community is encouraged to provide feedback on these developments.

Kibana 6.3.0 introduces several enhancements and new features aimed at improving user experience and functionality, particularly with the opening of X-Pack. Notable updates include license management changes, allowing users access to all Basic features of Kibana X-Pack by default, and the introduction of auto-complete in the Query Bar to simplify syntax and improve ease of use. The release also adds a "Recently viewed" section and time picker to help users return to previous work, refines Dashboard panel sizing, and introduces chained input controls for more dynamic data filtering. Additionally, the version enhances index management through a new UI, integrates interactive map support in Vega, and streamlines monitoring activation under the Basic license. Developers benefit from console enhancements and an updated method to inject server information into the browser, improving API usability and overall system performance.

In the June 2018 update for Kibana, significant developments include the introduction of TypeScript support, enhancing the way JavaScript developers plan and design their code by providing stricter data contracts and easing the creation of Kibana plugins. The update also highlights new job openings for roles such as Senior Product Designer and Senior JavaScript Engineer, alongside advancements in platform features like the saved object API documentation, improvements in EUI components, and updates to the HTTP service implementation. Other notable updates involve enhancements to the visualizations and management facets of Kibana, such as improved console autocompletion, fixes in GIS and Canvas tools, and ongoing efforts to support SQL and reporting integrations, all contributing to a more robust and user-friendly experience for developers and users alike.

The blog post discusses how to refine machine learning analyses by selectively curating input data in Elastic machine learning jobs, focusing on the importance of customizing the datafeed to target specific anomalies. By filtering out irrelevant data, such as traffic from bots or web crawlers, users can concentrate on anomalies that genuinely reflect user behavior, improving the relevance and productivity of their analysis. The process involves creating a filtered query using a Terms Lookup query in the datafeed, which allows users to exclude non-essential data, like bot-generated traffic, from analysis. This approach ensures the detection of meaningful anomalies, such as unusual HTTP response codes in NGINX web access logs, by focusing on real user interactions. Consequently, the machine learning job produces more accurate and actionable insights by aligning more closely with the intended use case, thus reducing unnecessary alerts and enhancing the overall utility and efficiency of machine learning applications in data analysis.