November 2017 Summaries

The blog post discusses how to ship Kubernetes logs to Elasticsearch using Filebeat, highlighting the importance of enriching logs with metadata for effective filtering and correlation. With the release of Filebeat 6.0, users can leverage Kubernetes metadata to enhance logs, enabling better troubleshooting by correlating logs and metrics. The add_kubernetes_metadata processor enriches logs with details such as pod name, container name, and Kubernetes labels by maintaining a local cache of running containers. Filebeat is deployed as a DaemonSet on Kubernetes, ensuring an agent runs on each node, and it accesses Docker logs to enrich them before sending them to Elasticsearch. The post provides a step-by-step guide to deploying this setup, emphasizing the role of metadata in improving log monitoring and analysis.

In the blog post by Christian Dahlqvist, the issue of bulk rejections in Elasticsearch clusters is addressed, focusing on the causes and potential solutions. Bulk rejections occur when the indexing workload surpasses the cluster's capacity, leading to parts of the bulk requests being rejected to protect the cluster from overload. The post explains the process of handling bulk requests and the role of the coordinating and data nodes. It highlights that increasing the bulk queue size is not a viable solution as it may result in cluster instability due to increased memory usage. Testing with different cluster configurations shows that adding more data nodes reduces rejection rates by distributing the load more evenly. The post advises against merely enlarging the bulk queue size and instead suggests understanding the source of rejections, ensuring even load distribution, and potentially scaling the cluster. It also emphasizes the importance of handling rejected documents properly by inspecting responses and retrying as necessary, as done by tools like Logstash and Beats.

Elastic's machine learning anomaly scoring system is designed to identify unusual patterns in data using a hierarchical approach, which includes record, influencer, and bucket scoring. Record scoring evaluates the unusualness of specific data points, influencer scoring assesses the contribution of specific entities to anomalies, and bucket scoring measures the overall anomalousness within a specific time window. The anomaly scores are normalized on a scale from 0 to 100 to provide a clearer understanding of the data's unusualness, with severity labels attached for context. This scoring system is crucial for proactive alerting, allowing users to detect significant deviations based on their specific needs. Influencer scores help identify entities responsible for anomalies, while bucket scores provide an aggregated view of anomalies over time. The choice of score for alerting depends on the desired granularity and frequency of alerts, with bucket-based scores recommended to prevent alert overload due to their rate-limited nature.

Elasticsearch 6.0 marks the beginning of a transition away from using mapping types, allowing only a single mapping type per index and setting the stage for their complete removal in version 7.0. Historically, mapping types were used to categorize different kinds of documents within an index, similar to tables in a SQL database, but led to complications due to shared field definitions across types, which could hinder data compression and efficiency. As a solution, Elasticsearch encourages users to adopt alternatives such as using a custom type field or creating separate indices for different document types, which can improve data density and search accuracy. The gradual phase-out of mapping types is planned to minimize disruption, with ongoing support for backward compatibility features until version 9.0, and new methods such as the join field replacing parent-child relationships. Users are advised to migrate multi-type indices to single-type configurations using the Reindex API to align with these changes.

Elasticsearch 6.0.0 GA, based on Lucene 7.0.1, marks a significant upgrade from version 5.0.0, featuring 2,236 pull requests by 333 contributors. The new release offers enhancements such as no-downtime rolling upgrades from Elasticsearch 5.x, improved search scalability with Cross Cluster Search, and the introduction of sequence IDs for more efficient shard recovery. Major improvements include reduced disk space usage for sparsely populated fields and faster query times through index sorting. Additionally, Elasticsearch 6.0 requires indices to have only a single mapping type, moving towards the removal of mapping types altogether. Security updates in X-Pack include mandatory TLS/SSL between nodes and the elimination of the default password "changeme," replaced by a tool to generate strong passwords. Users are encouraged to explore Elasticsearch 6.0.0 and provide feedback through various channels.

Logstash 6.0.0 introduces several significant enhancements to streamline data processing and management for system administrators and engineers. The release allows the concurrent operation of multiple pipelines within a single instance, enabling users to isolate processing logic for different data sources and reducing the need for complex conditional logic in configurations. This version also introduces centralized pipeline management through the Elastic Stack, allowing for easier orchestration and management via Kibana's UI, with configurations stored in Elasticsearch. Additionally, a new Pipeline Viewer tool in the Logstash Monitoring UI offers visualization of pipeline configurations and identifies performance bottlenecks, although it is currently in beta and may undergo changes. To facilitate the transition from Elasticsearch Ingest Nodes to Logstash for more complex use cases, a pipeline conversion tool is now available, and the upgrade to JRuby 9000 enhances compatibility with modern Ruby syntax and libraries.

In response to increased security concerns following a SHA-1 collision announcement, Elastic Stack has transitioned to using SHA-512 cryptographic hashes for verifying the integrity of its binary artifacts, despite continuing to provide SHA-1 checksums for backward compatibility with older versions. With the release of Elastic Stack 5.6.2, both SHA-1 and SHA-512 checksum files were available, but starting from version 6.0.0, only SHA-512 checksums are generated, featuring a more user-friendly format that combines the hash value and the associated filename on the same line. This change aims to enhance security and streamline the process of verifying downloaded files. Elasticsearch has also updated its plugin installation component to support SHA-512 checksums, requiring official plugins to adhere to this format, while community plugins can still use SHA-1 until it is phased out in future updates. Users are encouraged to update their verification scripts to accommodate these new SHA-512 files to ensure continued file integrity checks for Elastic Stack releases.

Elastic{ON} has launched the 2018 Opportunity Grant to support individuals from underrepresented groups in technology by offering discounted tickets, hotel accommodations, and travel grants for attending the conference. This initiative builds upon a previous informal scholarship program that enabled attendees from diverse backgrounds to participate in Elastic{ON} 2017, where they gained valuable insights into the Elastic Stack. Stories from past recipients, such as those involved in medical research, web development, and data engineering, highlight the impact of the conference on their professional growth and project advancements. The grant aims to amplify diverse voices at the event, with applications open to those over 18 who belong to underrepresented groups, and selections made based on need and potential impact. The application deadline is December 1, 2017, with notifications by January 5, 2018.

In an episode of the TV show "Mr. Robot," the character Elliot uses Kibana, a visualization tool from the Elastic Stack, to illustrate the Dark Army's data theft efforts, delighting the Elastic team and fans globally. The show, known for its realistic portrayal of cybersecurity, reached out to Elastic to feature Kibana, opting for version 4 to align with the show's timeline. Ryan Kazanciyan, a technical consultant for the series, created an authentic Kibana dashboard using data from Windows and Linux systems, showcasing malicious activities by building an ELK VM, comprised of Elasticsearch, Logstash, and Kibana, now known as the Elastic Stack. The inclusion of Kibana was met with enthusiasm from Elastic's team, who appreciated the show's technical accuracy and its portrayal of cybersecurity tools, prompting reactions highlighting the intersection of popular culture and technology.

Swiftype, a company known for its SaaS-based Site Search and Enterprise Search products, has joined forces with Elastic, a search technology company, to enhance their offerings and expand their reach in the search market. Swiftype's Site Search, which powers search experiences for major companies like Asana and TechCrunch, utilizes Elasticsearch to index and store searchable content, and will now be available at an introductory subscription plan starting at $79/month. With the integration, Swiftype's products will gain additional capabilities from the Elastic Stack and X-Pack, and the Enterprise Search product, which connects with cloud applications such as Salesforce and Slack, will be available as a beta trial. Founders Matt Riley and Quin Hoxie are enthusiastic about the partnership, emphasizing their commitment to providing high-quality search experiences and leveraging Elasticsearch's capabilities to innovate further in the field. The collaboration aims to offer tailored solutions and meet the increasing demand for intuitive search tools across various industries.

Elastic has been notified that the Google Chrome webstore has labeled a forked version of the Sense browser plugin as malware, although this version is not affiliated with Elastic. Originally developed by Elastic, Sense was a tool designed to simplify interactions with JSON for Elasticsearch and later became part of Kibana as the Console feature. While the flagged plugin, found on GitHub, did not show any obvious threats during a VirusTotal scan, Google likely had valid concerns, possibly related to outdated or insecure libraries. Elastic advises users of the Sense Chrome plugin to switch to Kibana's Console feature, which is actively maintained, and emphasizes the importance of verifying the source of software before installation. The incident underscores the broader lesson about the risks of using software with uncertain pedigree, highlighting the need for vigilance in software management as outdated software can pose security risks.

The text outlines the process of deploying Elasticsearch and the Elastic Stack on Microsoft Azure, highlighting the availability of an official deployment template from the Azure Marketplace that simplifies installation and configuration. It discusses the suitability of different Azure instance types for various components of the Elastic Stack, such as memory-optimized instances for Elasticsearch data nodes and smaller instances for master nodes and Kibana. The article also emphasizes the importance of redundancy and availability in deployment, recommending the use of Azure's geographical regions, availability sets, and zones to ensure robustness. Additionally, the text touches on scaling strategies using Scale Sets and data backup via the Snapshot Restore API. It mentions recent enhancements in data collection using Beats and Logstash, which now include Azure metadata enrichment, and provides insights into leveraging Azure-specific plugins for data ingestion.

AWS re:Invent is a significant event for Elasticsearch users, offering numerous opportunities for engagement and learning. Elastic will be present at booth #2031 in the Sands Expo Hall at The Venetian, providing personalized demos of the Elastic Stack, including Elastic Cloud Enterprise and security analytics with machine learning. Attendees can participate in Q&A sessions with Elastic experts to delve into search, logging, analytics, and metrics questions, and explore the differences between Amazon’s Elasticsearch Service and Elastic Cloud. The event also offers a chance to collect unique Elastic-themed swag, such as Kibana socks, and earn points through the AWS re:Invent app. For those unable to attend, Elastic invites users to their User Conference, Elastic{ON}, in San Francisco, with early bird pricing available.

Media organizations face increasing pressure to manage and review content effectively as free speech intersects with the need to monitor toxic content. Utilizing tools like the Elastic Stack and X-Pack's Graph API, organizations can proactively and reactively identify undesirable content. Proactively, they can use algorithms similar to recommendation systems to trace connections among users who favor questionable content, thereby narrowing down potential sources of hate speech. Reactively, they can rely on user reports for content removal while being cautious of orchestrated takedown requests by coordinated groups. The use of significance algorithms in graph databases aids in identifying meaningful connections, while greylisting provides a middle ground for content that cannot be immediately blacklisted. Ultimately, the challenge lies in balancing content management to ensure fair enforcement of service terms while maintaining user trust and advertising revenues.