November 2018 Summaries

Full-text search is a crucial component in many user-facing applications, and Lucene is a widely used text search engine that serves as the foundation for tools like Solr and Elasticsearch. Originally developed by Doug Cutting in 1999 and made open source by the Apache Foundation in 2001, Lucene supports fast and stable search capabilities but lacks comprehensive database features like geo-replication and high availability. Solr, developed on top of Lucene, and Elasticsearch, which evolved from the Compass project, both provide enhancements, with Elasticsearch offering better support for distribution, multi-tenancy, and integration for log analytics through the Elastic stack. Combining Elasticsearch with ScyllaDB offers significant advantages over the Cassandra-Solr pairing, as it enhances scalability and performance while avoiding Java Virtual Machine limitations associated with Cassandra. While Elasticsearch and ScyllaDB operate separately, this separation allows for flexible scaling of search and operational database workloads, with ScyllaDB’s C++ base providing additional performance benefits.

ScyllaDB has announced the release of ScyllaDB Enterprise 2018.1.7, a minor yet production-ready update to its enterprise NoSQL database solution, focusing on new features and bug fixes for the 2018.1 branch. A highlight of this release is the introduction of in-memory tables, which combine RAM and persistent storage for SSTables to achieve lower and more consistent read latency, making them suitable for mostly-read workloads with small data sizes. These in-memory tables incorporate a proprietary compaction strategy optimized for RAM storage and maintain high availability and persistence by mirroring data to disk. Users can enable this feature in the scylla.yaml configuration and modify table properties using CREATE or ALTER CQL commands. The release also introduces new metrics for monitoring in-memory storage usage and addresses specific bugs, such as stale gossiper application states affecting schema change propagation. ScyllaDB Enterprise customers are encouraged to upgrade to this version with the support of the ScyllaDB team.

The 2018 ScyllaDB Summit celebrated the innovative applications of ScyllaDB by honoring various users with awards for their exceptional use cases. Nauto was recognized for its technical use case in enhancing road safety with real-time data processing, while Grab was acknowledged for its industry use case, efficiently managing peak loads for its Southeast Asian platform. Comcast was awarded for replacing existing technologies with ScyllaDB for its Xfinity X1 platform, and GE Digital was highlighted for its time-series use in managing an extensive IoT platform. The Child Rescue Coalition received the Humanitarian Award for using ScyllaDB to protect children, while Sizmek was recognized for its widespread data center deployment. Meshify was commended for its swift production deployment, and Samsung for its broad use of ScyllaDB across multiple sectors. Alexys Jacob from Numberly and Red Hat were honored for their contributions to ScyllaDB and the Seastar Engine, respectively, while Tobias Macey and Jonathan Ness were recognized for their roles as the best evangelist and issue reporter. The awards showcased the diverse and impactful ways ScyllaDB is utilized across industries, encouraging further innovation and participation in future awards.

The blog post discusses integrating Spark Structured Streaming with ScyllaDB to handle infinite streaming data for real-time analytics, specifically using stock quotes as an example. It outlines the setup of a microservice that leverages Spark, ScyllaDB, and Kafka to process and store stock data, enabling the computation of daily statistics such as price changes. The article describes how to perform streaming computations using Spark's DataFrame API and introduces a custom solution for writing streaming data to ScyllaDB, as the default Datastax connector does not support this functionality out of the box. Additionally, it explains the use of ScyllaDB's materialized views to optimize data queries for different key structures, facilitating efficient computation of aggregate statistics across different dimensions. The blog emphasizes the importance of managing streaming queries' lifecycle and highlights resources for further learning about streaming systems.

ScyllaDB Summit 2018 was a notable event showcasing the advancements and applications of ScyllaDB, featuring keynotes and presentations from both ScyllaDB executives and high-profile customers like Comcast, GE Digital, and Grab. The summit highlighted ScyllaDB's evolution, including its Open Source 3.0 release and new cloud offerings, with discussions on its improved capabilities such as mc format SSTables, in-memory storage, and per-user SLA features. Customer use cases demonstrated significant performance improvements and infrastructure simplifications, such as Comcast's integration with its REST API services and GE Digital's use in its Predix platform. The event addressed challenges in combining OLTP and OLAP workloads, offering solutions through single, scalable clusters. It also featured technical deep dives into features like Materialized Views and Secondary Indexes, underscoring ScyllaDB's commitment to providing robust database solutions. Attendees benefited from a diverse array of sessions and insights, underscoring the versatility and scalability of ScyllaDB in various industries.

ScyllaDB Open Source 3.0 introduces stateful range scans, building on the stateful paging mechanisms introduced in version 2.2 for singular partition queries, significantly improving efficiency in handling range scans, which are queries that don't use the partition key in the WHERE clause. This new feature enhances throughput by 30% and reduces disk reads by 39% and disk operations by 73% compared to its predecessor, version 2.3. The stateful range scans utilize a special cache to save the reading pipeline, allowing queries to resume where they left off, which reduces the computational load on the cluster and shifts the performance bottleneck from the disk to the CPU. This change allows ScyllaDB to handle a greater number of reads per second, accommodating more concurrent loaders and improving overall database performance. These optimizations make range scans more efficient and reliable, providing significant improvements over the previous stateless implementation, and are expected to offer even more benefits with non-fully compacted tables.