March 2015 Summaries

In March 2015, Shay Banon announced the deprecation of rivers in Elasticsearch due to their contribution to cluster instability, which stemmed from their reliance on external systems and libraries. While initially successful in facilitating data integration into Elasticsearch, rivers introduced overheads and bugs that compromised system resiliency. The decision to phase out rivers, starting from version 1.5, was made to encourage more stable methods of data ingestion, such as using official client libraries or tools like Logstash, which can also munge data before it enters Elasticsearch. Despite the success and community involvement in developing river plugins, the focus has shifted towards using external processes or integrating with Logstash inputs, offering a smoother transition for developers and maintaining ease of data ingestion. Banon acknowledged the community's efforts and expressed a commitment to assist developers in adapting their river plugins through alternative means.

Inbot, a Berlin-based company, has developed a mobile-first CRM solution for field salespeople that leverages Elasticsearch for data management and real-time analytics. The application captures customer communications automatically from various apps, providing real-time insights and facilitating collaboration directly from mobile devices, which are ideal for data collection due to their proximity to users. By utilizing Elasticsearch's powerful aggregation features, Inbot delivers actionable insights and reports on sales activities, transforming traditionally tedious data entry tasks into automated processes. The company decided to build its backend on Elasticsearch, moving away from a MySQL prototype, due to its ability to handle vast amounts of data, real-time querying capabilities, and scalability. Despite initial industry skepticism about using Elasticsearch as a database, the Inbot team found that its advantages outweighed the challenges, especially after Elasticsearch addressed some resilience issues. The CRM features include real-time ranking of contacts based on user activity and integration of tagging systems to manage sales pipelines. Since its launch in the iOS app store, Inbot has experienced rapid growth and is focused on supporting its initial customers while adding new features, with Elasticsearch proving to be a robust and evolving solution.

This Week in Elastic introduces a new blog format featuring weekly updates on the ELK Stack, including Elasticsearch, Logstash, and Kibana, with a focus on granular project updates and broader ecosystem overviews. Readers are encouraged to provide feedback and share community stories. Recent developments include the release of Elasticsearch 1.5, Curator 3.0, and Shield, alongside updates from Logstash engineering meetings and a multi-tenant application design guide. The post highlights significant Elastic-related events, like the Elastic{ON} conference, and features user stories such as Spot Trading's ELK stack adoption for hardware monitoring and Norway's postal service migration to Elasticsearch. Additionally, various global meetups and presentations are announced, offering opportunities for users to engage with the Elastic community, while training sessions across several countries are available for those interested in learning more about Elasticsearch. The post also encourages community involvement in meetups and highlights job opportunities within Elastic.

This weekly update on Elasticsearch and Apache Lucene, released by Elastic, covers a recent developer all-hands meeting and the latest developments in the Elasticsearch ecosystem. The meeting, the first since June of the previous year, brought the distributed engineering team together to discuss the future direction of their projects and foster better communication and camaraderie. Key highlights include the release of Elasticsearch 1.5.0, which offers significant resiliency enhancements and bug fixes, alongside new experimental features like inner hits and shadow replicas. Additionally, various internal improvements, deprecated features, and plugin updates for platforms like AWS and Google Compute Engine have been implemented to enhance functionality and compatibility with version 1.5. The post encourages community feedback and engagement while promising regular updates and insights into the broader ELK ecosystem.

The Elasticsearch and Apache Lucene update highlights recent developments and community activities, including the release of Elasticsearch 1.5.0, which features significant resiliency enhancements, bug fixes, and new experimental features such as inner hits and shadow replicas. The Elasticsearch developer team recently convened for an all-hands meeting to discuss future directions, team growth, and improvements in communication and culture, emphasizing the collaborative spirit within the group. Additionally, there have been numerous updates to plugins and internal components, enhancing compatibility with the latest Elasticsearch version and improving performance in various aspects. The post encourages engagement from the community for feedback and suggestions on resources to feature, while also spotlighting upcoming advancements in the ELK ecosystem.

Shield 1.1 and 1.2 have been released, offering new features and performance improvements based on user feedback, with the primary distinction between the two being their compatibility with different Elasticsearch versions. Shield 1.1 supports Elasticsearch 1.4.2 and newer versions of 1.4.x, while Shield 1.2 requires Elasticsearch 1.5.0 or higher, and the plugin download service now automatically selects the appropriate Shield version based on the Elasticsearch version. Notable additions include LDAP user search, which enhances user authentication flexibility and performance, anonymous access that allows fine-grained control over actions available to non-authenticated users, and dynamic IP filtering that can be managed without restarting the cluster. The updates also include support for mapping LDAP users to roles, filtering sensitive settings from node settings API outputs, and improvements to facilitate the upgrade process.

Elastic{ON}, the inaugural conference for Elastic products, has released its first wave of content, featuring presentation slides and a selection of videos from a range of prominent organizations such as Facebook, Microsoft, NASA's Jet Propulsion Laboratory, and Yale University. The event emphasized knowledge sharing through its sessions, informal gatherings, and an "Ask Me Anything" booth staffed by developers and employees. Unfortunately, due to technical issues, audio was lost for the first day's main stage content, including the keynote, prompting considerations for creative solutions like dubbing. Despite this setback, the available materials highlight various impressive use cases of Elastic products, with more content to be released as processing continues.

Elasticsearch 1.5.0 has been released, featuring significant resiliency improvements and bug fixes, along with two experimental features: Inner Hits and Shadow Replicas. Inner Hits allow users to return child documents that match a query alongside their parent documents, enhancing the search functionality in nested or parent-child queries. Shadow Replicas enable scaling of search throughput by using distributed file systems for redundancy instead of making multiple copies of each shard. The update also addresses various issues from previous versions, including corrupt indices detection, improved shard management, and enhanced checksum validation, ensuring a more robust and reliable system. Users are encouraged to upgrade for these enhancements and to provide feedback on their experiences.

The blog post introduces a new series titled "This Week in Logstash," focusing on developments within the Logstash ecosystem. It highlights the activities from an Engineering All-Hands meeting held at Elastic's new office in Mountain View, following their first user conference. Key discussions included enhancing the plugins ecosystem by separating them from the core to encourage community involvement, improving pipeline semantics with a standardized API for easier plugin development, and strengthening the testing infrastructure with more integration testing and performance tracking. The team also aims to streamline the development workflow by aligning GitHub labeling with Elasticsearch and Kibana, ensuring peer-reviewed pull requests, and enhancing documentation to include detailed architecture and use case-driven guides. The meeting emphasized community engagement and collaboration among engineers to drive future Logstash innovations.

Konrad Beiske discusses the challenges and strategies involved in implementing an indexing strategy for multi-tenant applications using Elasticsearch, emphasizing the importance of data separation, efficient resource use, and scalability. Initially, the fictive blog hosting provider used a separate index per tenant but faced memory issues, leading to a shared index approach with filtered queries based on tenant IDs. To address cache warming inefficiencies and ensure performance, they adopted custom routing using tenant IDs and later adjusted shard numbers to accommodate growth. As tenants' data and traffic expanded, particularly the largest tenant, the team implemented special cases and increased replicas to manage search loads effectively. Concerns about scalability and uneven shard distribution prompted the development of a configuration scheme allowing dynamic allocation of indices based on tenant needs, using index aliases for smooth data transitions. The article concludes by highlighting the importance of flexibility, especially as tenant sizes and demands evolve unpredictably, and suggests encapsulating complexity within the application's data access layer to maintain performance and scalability.

Elasticsearch, a search platform known for its versatility in various use cases, has significantly expanded its capabilities by developing language clients, embracing Kibana for data visualization, and introducing Logstash for processing time series data. To further enhance its offerings, Elasticsearch aimed to provide "CloudES," an Elasticsearch as a Service solution for both on-premise and cloud environments. This vision aligned perfectly with Found, a company recognized for its exceptional "Elasticsearch as a Service" offering, which features a scalable proxy architecture, smart cluster management, and robust security measures. Found's expertise in using Linux containers and providing features like periodic snapshots and support for multiple Elasticsearch versions made it a fitting partner for Elastic's ambitions. Consequently, Elastic acquired Found, integrating its team to advance their shared vision, which also coincides with Elastic's rebranding efforts.

Founded in 2012, Elasticsearch has grown significantly beyond its initial focus on search, evolving into a suite of products that provide diverse data insights and analytics capabilities. By 2015, the company had expanded its offerings to include five additional products, serving a wide array of use cases across industries, including healthcare, seismology, financial services, and space exploration. This expansion prompted a rebranding to "Elastic" to better reflect the versatility and utility of its solutions. The company also announced the acquisition of the Elasticsearch SaaS provider Found, which enhances their ability to manage multiple clusters through both SaaS and on-premise solutions. Elastic aims to support its growing user base by offering a centralized management infrastructure, catering to an increasing number of organizations deploying multiple ELK clusters for various purposes. This strategic shift and the rebranding underscore Elastic's commitment to innovation and adaptability in handling large volumes of data while inviting users to join their ongoing journey.