August 2018 Summaries

ScyllaDB, a database solution incorporating Apache Cassandra's scale-out design, emphasizes fault tolerance and high availability by leveraging distributed workload and storage along with eventual consistency. It adheres to the CAP Theorem, prioritizing availability and partition tolerance over consistency, especially during network partitions. The database's architecture allows for the selection of a Replication Factor, determining the number of nodes holding data copies, and a tunable Consistency Level, which specifies how many nodes must acknowledge read or write operations. This flexibility allows users to balance the trade-off between consistency and latency according to their needs. ScyllaDB's fault tolerance is enhanced by replicating data across multiple nodes, allowing operations to continue even if a node becomes unreachable, with the Consistency Level and Replication Factor impacting both performance and fault tolerance. The blog post further encourages users to explore ScyllaDB's documentation for more detailed insights into its architecture and fault tolerance mechanisms.

ScyllaDB is launching its first annual User Awards to celebrate the innovative accomplishments of its users, with nominations open until September 28, 2018. Categories for the awards include the Most Interesting Technical Use Case, Best Use of ScyllaDB for Time Series, and the ScyllaDB Humanitarian Award, among others. Winners will be announced at the ScyllaDB Summit 2018 and will receive complimentary attendance to the event, along with a limited-edition ScyllaDB hoodie. The awards encourage users to share their unique implementations and contributions to ScyllaDB's open-source community, highlighting cases that are pioneering or beneficial to society. Users can nominate themselves in multiple categories, and the initiative aims to showcase the diverse ways ScyllaDB is being utilized across different industries and applications.

In the continuation of the Spark and ScyllaDB integration series, this blog post explores the execution of data transformations in Spark, focusing on the RDD abstraction and the transition to higher-level SQL and DataFrame interfaces. It delves into Spark's execution model, detailing how transformations create specialized RDD subtypes and how jobs, stages, and tasks are structured within Spark's lazy execution framework. The post highlights the importance of understanding Spark's execution visualization tools, such as the application UI, to monitor the progress and details of jobs. It also discusses the intricacies of integrating Spark with ScyllaDB using the DataStax Connector, which aligns ScyllaDB's token ranges with RDD partitions for efficient data processing. The article further examines the limitations of the RDD API, such as the opacity of closures and serialization challenges, and introduces the benefits of using Spark SQL and the Dataset API, which offer optimized query execution through Spark's Catalyst engine. This includes automatic column pruning and filter pushdowns, enhancing performance by reducing unnecessary data transfers. The post concludes with a preview of topics to be covered in the next installment, including saving data back to ScyllaDB and exploring the Spark Streaming API.

ScyllaDB's Open Source 3.0 release introduces enhanced filtering support, addressing the reliance on the ALLOW FILTERING keyword by the Spark-Cassandra connector when generating CQL queries. Although filtering can be useful, it may negatively impact performance, prompting a need for alternative solutions such as schema changes, secondary indexing, or materialized views. These alternatives can help avoid the inefficiencies of filtering, which executes only after fetching all potential rows, leading to unnecessary data processing. The discussion highlights the importance of thoughtful data model design to mitigate performance issues related to filtering, and it provides insights into the conditions when ALLOW FILTERING might be beneficial, particularly in low selectivity queries. Additionally, the blog post underscores the impact of filtering on performance through a local test, suggesting that sequential scans can sometimes outperform random index lookups, especially in specific scenarios.

ScyllaDB's streaming process, a vital internal operation for data movement between nodes, has undergone significant improvements in its Open Source 3.0 release, enhancing efficiency and performance. The new streaming mechanism eliminates the use of memtables by directly writing data to SSTables on disk, leading to reduced memory and CPU usage, as well as fewer compactions. This upgrade utilizes the Seastar RPC Streaming interface, which improves bandwidth by 240% and reduces the rebuild operation time by 70%, as demonstrated in tests with clusters on AWS. These enhancements are set to accelerate cluster operations such as adding, decommissioning, and repairing nodes, and are part of ScyllaDB's effort to optimize its distributed database system.

The article by Vladislav Zolotarov explores the performance implications of running ScyllaDB, a shard-per-core database architecture, within Docker containers, highlighting both the benefits and drawbacks of containerization. Docker's widespread adoption simplifies database management by standardizing deployment across platforms, but introduces a performance cost due to virtualization layers and relaxed resource isolation. Initial tests on AWS showed up to a 69% reduction in write throughput with default Docker settings, but performance improved significantly through optimizations like CPU pinning and network interrupt isolation, reducing the penalty to just 3%. While Docker's network virtualization remains a bottleneck, bypassing it with the --network host parameter further narrows the performance gap. The article concludes that while Docker offers ease of use, achieving close-to-native performance requires specific tuning, suggesting that users prioritize direct installation on latency-sensitive applications.

The Mutant Monitoring System blog series by ScyllaDB offers an engaging and educational approach to training users on fundamental ScyllaDB concepts through a fictional narrative involving mutants. The series covers various topics over 16 days, including setting up ScyllaDB using Docker, building and analyzing a tracking system for mutant data, managing node failures, visualizing data with Apache Zeppelin, and deploying multi-datacenter configurations. It also explores monitoring with Prometheus and Grafana, connecting to ScyllaDB using Node.js and Java drivers, backup and restore procedures, using materialized views, integrating Apache Spark for data analytics, storing binary files, and developing a web console for the monitoring system. This comprehensive training aims to equip users with practical knowledge of ScyllaDB's capabilities and its integration with third-party applications, enhancing their ability to develop scalable and efficient applications.

At the Big Data Montreal meetup, Mina Naguib from AdGear and Glauber Costa from ScyllaDB discussed the benefits of using ScyllaDB for real-time big data processing, specifically highlighting its impact on AdGear's online ad serving platform. By switching from Cassandra to ScyllaDB, AdGear was able to achieve 1 million queries per second with significantly reduced latencies and hardware requirements, downsizing from a 31-node Cassandra cluster to a more efficient 16-node ScyllaDB cluster. This transition resulted in reducing read latency from 21 milliseconds to less than 5 milliseconds during peak traffic. The presentation emphasized ScyllaDB's ability to handle high throughput with low latencies and auto-tuning capabilities, addressing common issues experienced with Cassandra such as unpredictable latency and tuning complexity. The meetup underscored the advantages of ScyllaDB's close-to-the-hardware architecture, which allows for efficient scaling and improved performance, making it a superior choice for handling real-time bidding processes in online ad exchanges.