September 2018 Summaries

Hacktoberfest is an annual event dedicated to promoting open source contributions throughout October, providing participants an opportunity to engage with open-source projects by addressing issues tagged with "Hacktoberfest" on GitHub. This initiative is particularly embraced by Elastic, which sees it as a chance to strengthen ties with the open-source community and benefit from the fresh input and feedback it generates. The event aims to demystify open-source contributions, likening the learning process to learning to ride a bike, and encourages newcomers to focus on projects that match their skills and interests, such as Logstash or Beats for those interested in Java, Ruby, or Go, and Rally or Curator for Python enthusiasts. Participants are advised to read contribution guidelines, choose manageable tasks, and maintain open communication with project maintainers. The ultimate goal is to facilitate newcomers' participation and emphasize that contributing to open source is a gradual process, akin to a marathon rather than a sprint, with the potential for significant personal growth and learning.

The tutorial provides a comprehensive guide on building a robust and user-friendly search experience using React and the Elastic App Search JavaScript client. It emphasizes the importance of delivering fast, relevant, and intuitive search results to enhance user satisfaction. The process involves creating an Engine in Elastic App Search, ingesting data for indexing, and using React to build a responsive search interface. Key steps include setting up a search box, handling user queries, and employing a debounce function to optimize search request rates. The tutorial encourages further enhancements such as adding styling, implementing dynamic features like Facets and Relevance Tuning, and utilizing the Analytics API for insights. Elastic App Search is highlighted as a powerful tool for integrating effective search functionalities into web applications, with easy management through its user-friendly dashboard.

Keeping up with Kibana highlights the ongoing developments and updates within the Kibana project, focusing on areas such as security enhancements, Canvas integration, rollup support, and design improvements. The team is actively working on refining Saved Object Namespaces and implementing granular application privileges for enhanced security. Canvas integration efforts include new features like dynamic progress elements and the push to move the Canvas interpreter into the core of Kibana for more robust visualization capabilities. The rollup support phase is transitioning from core development to polishing and bug-fixing, with improvements like a user-friendly cron editor for the Rollup Job Wizard. Design updates include the conversion from Less to Sass for stylesheets, particularly in Machine Learning, and the introduction of a new palette service. The Kibana team is also managing operations such as the Saved Object Migrations review and K7 header work for version 6.5, while platform efforts focus on unblocking K7 and starting the Alerting project. Additional tasks involve addressing flaky tests, enhancing cross-browser testing capabilities, and progressing on KQL telemetry and visualization developments.

eBay manages an immense volume of data, processing 1.2 petabytes of logs daily and 5 million metric data points per second, by employing a sophisticated system of containerization and monitoring tools. Initially, eBay's team faced challenges in adapting to the dynamic nature of applications and environments, prompting the adoption of Docker and Kubernetes to streamline deployments. To handle the rapidly evolving data landscape, they turned to Beats, specifically Filebeat and Metricbeat, for log and metric collection, and developed Collectbeat for autodiscovery of new pods in Kubernetes clusters. They further enhanced data tagging with the 'add_kubernetes_metadata' processor to append essential metadata, enabling better analysis and visualization. As data volumes continue to grow, eBay employs strategies like tier-based quota and retention limits and prioritizing event data to efficiently manage resources. These efforts are part of their broader strategy, demonstrated in their internal system, Sherlock.io, to maintain operational visibility and adapt to technological advancements.

In an effort to upgrade Azure DevOps Search with zero downtime, Microsoft embarked on a project to transition from Elasticsearch 2.x to 5.x, alongside upgrading from NEST 1.x to 5.x. The primary motivation was to ensure compatibility with the latest Azure DevOps Server and to capitalize on performance enhancements. The team executed the upgrade in three phases: moving to NEST 2.x, running Elasticsearch 5.x parallel to 2.x, and ultimately transitioning fully to 5.x while eliminating 2.x. Key strategies included timeboxing large reindexing tasks, improving monitoring and retry mechanisms, and optimizing the indexing topology by splitting large clusters into smaller ones and increasing the number of shards per index. Throughout the process, the team emphasized documentation, capturing issues and suggestions to facilitate continuous improvement. They also leveraged community support and resources, such as Elastic{ON} conferences and meetups, to enhance their understanding and application of Elasticsearch. The upgrade saw a significant increase in data handling capacity and indexing speed, demonstrating the benefits of the transition.

Elastic App Search has introduced a comprehensive Analytics API suite designed to enhance user search experiences by offering valuable insights into search patterns, clicks, and queries. This suite enables users to understand not only the explicit search terms but also the underlying search behaviors, thus allowing businesses to improve overall search relevance and user engagement. By leveraging this analytics tool, companies can identify potential barriers in search experiences, such as the underperformance of certain documents, and take corrective measures like altering document titles or using API features to promote specific results. Moreover, the suite provides a detailed look at the general performance of searches, with endpoints that offer specific metrics on queries and clicks, enabling businesses to refine their search strategies effectively. Through additional features like the Clickthrough API and Tags, deeper insights into user interactions can be gathered, facilitating a more dynamic and responsive search environment. Elastic App Search offers a free 14-day trial for users to explore these capabilities without a credit card, emphasizing its commitment to delivering relevant and adaptable search solutions.

Signal Media's blog post by Joachim Draeger outlines the company's journey to optimize its Elasticsearch cluster, a critical component of their platform for indexing and searching millions of documents daily. Initially struggling with high costs and performance issues due to a rapidly expanding content base, they engaged with Elastic's community and support to refine their approach. Key strategies included upgrading Elasticsearch versions, adopting automation tools like Terraform and Packer, implementing a blue-green deployment strategy, and optimizing data sharding practices. By reducing the number of shards and utilizing AWS's i3 instances with enhanced I/O capabilities, Signal Media significantly improved performance and halved their infrastructure costs, all while maintaining a robust 24/7 operational environment. This evolution demonstrates the importance of continuous learning and adaptation in leveraging Elasticsearch's capabilities effectively.

The latest updates in the Kibana project highlight several developments, including job openings for various JavaScript engineering positions and the integration of Canvas into the core repository, which is aimed at enhancing user experience with operational analytics through new workpads. The Rollup Job Wizard has been fully operationalized to simplify data rollup configuration, and efforts are being made to improve the integration of rollup data with visualizations and dashboards. The Spaces feature is under review, focusing on secure access and space-aware advanced settings, while a new webinar provides a 30-minute introduction to creating visualizations and dashboards in Kibana 6.4. Additional design work is underway, including the Sass conversion for visualization apps and enhancements to EUI components, such as Table actions and Search Bar. Accessibility improvements, telemetry opt-in facilitation, and various updates in Kibana Apps, like Visualizations and Sharing, reflect ongoing efforts to refine user interfaces and functionalities across the platform.

On September 18, 2018, Paul Sanwald announced the release of Elastic Stack version 6.4.1, which primarily focuses on addressing significant bugs present in the previous version. Key bug fixes include enabling query caching by default for the standard distribution, resolving an issue that prevented 6.4.0 nodes from restarting after rollovers, and correcting the installation path for the modules.d directory in the Metricbeat DEB and RPM packages. This update aims to enhance the functionality and reliability of the Elastic Stack platform.

The blog post by Emily Stolfo discusses the transition from the ActiveRecord pattern to the Repository pattern in the context of the elasticsearch-persistence gem for Ruby domain objects, following its 6.0 release which deprecated the ActiveRecord pattern. This change requires users to migrate their applications, but it is designed to better align with Elasticsearch's non-relational nature, avoiding the technical difficulties associated with ActiveRecord. The Repository pattern allows for a separation of domain objects from persistence code, accommodating Elasticsearch's advanced features without the constraints of schema and structure typical of ActiveRecord. The post provides a detailed migration guide using a sample music application that illustrates the necessary changes in models, controllers, views, and tests, emphasizing the importance of repository classes for handling persistence and search operations. It also suggests best practices for defining attributes on Plain Old Ruby Objects and highlights the implications of using Elasticsearch's join datatype. The guide aims to support developers either building new applications or transitioning existing ones to leverage Elasticsearch more effectively.

Elastic App Search introduces multi-language support, allowing users to optimize their search engines for 13 languages, such as English, Spanish, German, and Chinese, enhancing search precision by using language-specific analyzers. The system simplifies the search process by abstracting complex configurations through an API-based solution, ensuring that text analysis and tokenization are adapted to each language's nuances. This approach prevents confusion in languages like Chinese, where characters represent concepts rather than individual words, by recognizing terms as bigrams rather than separate tokens. The integration of language-optimized analyzers occurs both during indexing and querying, providing a seamless experience without additional configuration. Elastic App Search offers an intuitive dashboard and APIs, enabling users to create language-specific search engines effortlessly, thereby improving search accuracy and user experience across different languages. With the introduction of these advanced language capabilities, Elastic App Search aims to deliver high-quality search experiences, supported by a 14-day trial for new users to explore its features.

The blog post discusses the Task Management API in Elasticsearch, focusing on its integration with the NEST .NET client to manage and trace tasks in a cluster. This API, which evolved from experimental to beta in version 6.0, allows users to retrieve details about running tasks, including their duration and execution nodes. A key feature highlighted is the use of the OpaqueId within the NEST .NET client (version 6.2.0 and above), which is set in the request header to uniquely identify and later trace or cancel tasks. This OpaqueId facilitates task management in multi-tenant clusters by providing user-defined identifiers that can include application, department, or user information, making it easier to track task origins. The client exposes this feature through the RequestConfiguration object, enabling users to manage tasks efficiently, including canceling them when necessary to maintain optimal cluster performance.

The Elastic APM Node.js agent has been updated to version 1.12.0, introducing the ability to identify routes and collect errors for applications using the Restify server framework, without requiring special configuration. This update allows for seamless transaction naming and error reporting in Restify-based applications by simply upgrading to the latest version of the agent. Additionally, Elastic APM is actively working on enabling distributed tracing, which will allow the correlation of requests across multiple servers to create a unified trace, potentially highlighting areas for performance improvements in user-facing request lifecycles. Users are encouraged to try out the updated agent, provide feedback, and contribute to its development through the GitHub repository.

Elastic machine learning provides two primary methods for detecting anomalies: temporal and population-based analysis. Temporal anomaly detection focuses on comparing the behavior of an entity with its past behavior over time and is the default mode, but it struggles with high cardinality or sparse data elements. Population anomaly detection, activated through specific configurations, compares individual entities against a collective model of all peers, making it more suitable for high cardinality or sparse data. The text illustrates these concepts using a hypothetical scenario involving document downloads from a company's server. Temporal analysis might fail to flag a new user's unusual download activity due to lack of historical data, whereas population analysis would highlight the same activity as anomalous by comparing it to typical collective behavior. This approach is more memory-efficient and adept at handling sparse data, and users are advised to construct homogenous populations for accurate analysis. The text encourages trying Elastic Stack for machine learning and suggests setting up trials to explore these features.

Elastic, an open-source software company, relies heavily on community contributions to enhance its products, which are hosted in various GitHub repositories. The process of submitting a pull request (PR) involves creating a fork of a repository, making code changes, and then following a structured procedure to propose these changes for integration into the master branch. Elastic provides a pull request template to guide contributors through the submission and review process, emphasizing the importance of avoiding duplicate submissions, including tests for code changes, and ensuring PRs are made against the master branch. Once submitted, a PR is labeled and assigned to the appropriate team for review, during which discussions, additional commits, and testing occur to ensure quality and adherence to coding standards. Documentation changes are handled more simply, without the need for forking or tests. The time taken to commit a PR varies based on complexity, but contributors are encouraged to engage with the process and seek assistance via Elastic's Discuss forums if needed.

With the release of Elasticsearch version 6.4.0, the software can now operate within a FIPS 140-2 environment, which requires a Platinum subscription. FIPS 140-2 is a U.S. Government standard set by NIST that outlines security requirements for cryptographic modules, aiming to protect sensitive but unclassified data. While Elasticsearch itself is not a cryptographic module, it can comply with FIPS 140-2 by running in a Java Virtual Machine (JVM) that utilizes approved cryptographic algorithms through Java's Cryptography Architecture and Extension providers. This compliance ensures that data encryption, password hashing, and secure communications within Elasticsearch adhere to stringent security standards. To enable FIPS 140-2 mode, users must configure the "fips_mode" setting in Elasticsearch, and they have the option to switch from bcrypt, which is not FIPS-approved, to the compliant PBKDF2 algorithm for password hashing. Elastic aims to integrate FIPS 140-2 support across more of its products, reflecting its commitment to security.

Elastic Maps Service (EMS) enhances geospatial visualizations in Kibana by providing basemap tiles, region map boundaries, and key attribute data, including new vector map layers for country subdivisions like Australian states and Swiss cantons. These layers use the ISO 3166-2 codes for unique identification, allowing users to visualize web traffic by country subdivisions. To achieve this, users can leverage the Logstash GeoIP filter or the Ingest GeoIP Processor to add location data from the MaxMind GeoIP databases, and subsequently create a Painless scripted field to join the data with region maps. Kibana version 6.4 introduces a preview feature for scripted fields, facilitating the creation of region maps for specific countries using these subdivisions. EMS continues to expand its offerings with more vector map layers, which are accessible without requiring updates to Elasticsearch or Kibana, with region maps available from Kibana version 5.5 onwards.

Pablo Pérez's article discusses a method for analyzing air quality data using Elasticsearch on Elastic Cloud, focusing on the city of Madrid. The process involves transforming raw air quality data from CSV files provided by Madrid’s City Hall into JSON documents to be indexed in Elasticsearch, enabling efficient data querying and analysis. By leveraging the capabilities of the Elastic Stack, including Kibana for visualization, users can transform otherwise opaque chemical measurements into actionable insights, such as pollution hotspots or trends related to human activities. The article highlights the potential for automated data transformation and visualization, offering a streamlined approach to understanding urban air quality, with the promise of further simplifying data ingestion processes in future posts.

Elastic machine learning anomaly detection is a fully unsupervised method that builds a dynamic model of the data in real-time to automatically identify statistically anomalous events. However, to refine this detection process, domain knowledge is crucial, prompting the introduction of custom rules in version 6.4 of the Elastic Stack. These custom rules allow users to incorporate domain-specific insights to adjust the behavior of anomaly detectors, enhancing the relevance of detected anomalies. For instance, a custom rule can be set to prevent anomaly alerts for low CPU utilization that falls below a certain threshold, ensuring only significant deviations are flagged. Similarly, in security analytics, rules can exclude known safe domains from triggering alerts, thereby focusing on potentially suspicious activities. The rule actions, including 'skip_result' and 'skip_model_update,' provide flexibility in handling anomalies by either preventing their creation or influencing model updates. Custom rules can be applied to both numerical and categorical data, and while they immediately impact new results in real-time jobs, applying them to historical data requires cloning and re-running jobs. This functionality enhances anomaly detection by aligning it with the user's domain expertise, allowing for more meaningful scoring and ranking of anomalies.

Elasticsearch 6.4 introduces Kerberos authentication support for Platinum subscriptions, marking an initial step towards a fully Kerberized Elastic Stack by providing secure authentication in distributed systems. The blog post explains the configuration process for enabling Kerberos authentication for HTTP traffic in Elasticsearch, using a practical scenario involving a user named Alice, who operates a single-node Elasticsearch cluster. The setup requires a Kerberos realm, a Key Distribution Center (KDC), and a client machine, with necessary configurations in the Kerberos configuration file and Elasticsearch settings. By following the outlined steps, which include configuring JVM options, setting up the Kerberos realm in Elasticsearch, restarting the node, and mapping Kerberos users to roles, users can successfully implement Kerberos authentication. The post concludes by noting that this integration is a foundational step, with plans to expand Kerberos support across other components of the Elastic Stack in future updates.

An improved version of the Elastic APM intake protocol has been developed to enhance memory efficiency and predictability for APM agents and servers. The original protocol relied on a simple, HTTP- and JSON-based system, where agents buffered transactions in memory before sending them to the APM server, potentially leading to memory inefficiencies. The new protocol draws inspiration from the Elasticsearch Bulk API, using newline-delimited JSON (NDJSON) to allow agents to serialize and stream events directly to the server without extensive buffering, thus simplifying memory management. This change enables agents to handle events with predictable memory usage, as each transaction and span is treated independently, reducing the need for large, memory-intensive HTTP request bodies. Initial tests suggest that this method not only optimizes memory usage on both the agent and server sides but also improves throughput, although certain issues remain to be addressed before full implementation in Elastic APM.

Benjamin Wohlwend's guide explores the process of creating custom framework integrations for the Elastic APM Python Agent, focusing on the Pyramid web framework as an example. Given the vast array of Python web frameworks, Elastic APM cannot offer built-in support for all, but Wohlwend demonstrates that building a custom integration is feasible and efficient, requiring less than 100 lines of code. The integration process involves calling `elasticapm.instrument()` early during the app startup, initializing an `elasticapm.Client` object, managing transactions, and gathering contextual information from requests and responses. Additionally, the guide covers handling exceptions to ensure comprehensive application monitoring. By utilizing Pyramid's "tweens" and configuration capabilities, developers can seamlessly integrate Elastic APM, enhancing their ability to trace, monitor, and troubleshoot applications. The complete code example is available on GitHub, and the guide encourages users to engage with the Elastic community for further support and to suggest future framework integrations.