April 2020 Summaries

ReversingLabs, a company specializing in threat intelligence, has developed the Titanium platform to handle the rapidly evolving landscape of cybersecurity threats by automating processes and utilizing scalable solutions. At the ScyllaDB Summit 2019, ReversingLabs software architect Goran Cvijanovic discussed the integration of ScyllaDB into their platform to manage file reputation and optimize storage through compression. Their TitaniumCloud platform maintains a vast database of over 20 billion files, categorizing them as benign, malign, suspicious, or unclassified, using static and dynamic analysis methods. To enhance its capabilities, ReversingLabs aggregates data from over 40 antivirus partners and stores metadata and reputation analyses in a highly available database system, handling more than a billion malware-infected files across five petabytes of storage. The company has implemented a custom distributed key-value store to manage storage efficiently, and performance tests demonstrated ScyllaDB's ability to handle high traffic with low latency, leading to its adoption in ReversingLabs' production systems.

ScyllaDB Cloud, a fully managed NoSQL Database as a Service (DBaaS) that supports DynamoDB and Apache Cassandra APIs, offers an Extract Metrics feature that allows users to extract cluster metrics in the widely-used Prometheus format. This feature facilitates the tracking of infrastructure metrics, aiding database and system administrators in optimizing system utilization and correlating trends across multiple systems to identify business impacts. By using an internally-developed proxy, ScyllaDB securely ships these metrics to customer systems, enabling integration with platforms like Datadog, Sensu, and New Relic. Users can leverage ScyllaDB’s Monitoring Stack or create custom dashboards to better visualize performance data, define business-specific alerts, and manage thousands of metrics effectively. Key metrics include node status, CPU load, data directory size, and request latency, among others, which can be monitored using Grafana dashboards. This feature is readily available for both existing and new ScyllaDB Cloud users, simplifying the process of performance monitoring and optimization.

The blog post discusses a lesson from ScyllaDB University on using the CDRS Rust driver with ScyllaDB, highlighting its application in creating a Rust program to connect to a ScyllaDB cluster for storing and querying temperature time-series data. The example application demonstrates setting up a keyspace and table using ScyllaDB's SimpleStrategy for replication, and executes queries to store and retrieve temperature measurements based on device ID and time range. Dependencies like the CDRS driver, UUID, and time packages are defined in the Rust project, with connection logic and data handling implemented in separate Rust modules. The post also mentions ScyllaDB's load balancing strategies and the use of procedural macros for data conversion in Rust. The lesson is part of a broader course offering available for free at ScyllaDB University, aiming to enhance understanding of NoSQL databases and Rust programming.

Brazil's leading food delivery service, iFood, which dominates 86% of the market and operates in several Latin American countries, leverages ScyllaDB to efficiently manage over 100 million monthly events from restaurant orders. Initially utilizing Amazon DynamoDB, iFood transitioned to ScyllaDB to address challenges with DynamoDB's autoscaling limitations during peak demand times and its higher costs. The migration to ScyllaDB, facilitated by its compatibility with existing query models, resulted in significant cost savings—reducing database expenses from $54,000 to $6,000 annually for one service—and improved performance, notably decreasing event handling times from 80ms to 3ms. iFood's infrastructure on AWS uses various services like Amazon SNS and SQS, and the shift to ScyllaDB also allowed for more effective event processing and reduced code complexity by 40%. Despite minor challenges, such as managing TTLs and understanding ScyllaDB's collection updates, the transition proved advantageous, offering scalable, reliable solutions amidst surging demand, particularly during the COVID-19 crisis, when iFood also expanded support for its delivery partners and restaurant network.

ScyllaDB, while adept at managing large partitions, can encounter performance issues with very large rows and cells due to the need for allocating substantial contiguous memory, potentially increasing latency. To address this, ScyllaDB introduced tools to detect large rows and cells, with updates to the SSTable format in ScyllaDB 3.0 and ScyllaDB Enterprise 2019.1, which are enabled by default in ScyllaDB Open Source 3.1 and above. These tools use system tables to store data about large rows and cells, capturing details like keyspace and table name, SSTable name, row size, clustering key, and, for cells, the column name. Users can query these tables to troubleshoot performance issues, and they are alerted through warnings in the ScyllaDB log when specified thresholds are surpassed. The system.large_rows and system.large_cells tables store data with a 30-day Time To Live (TTL) to prevent stale data, highlighting the importance of early data modeling decisions in avoiding performance bottlenecks. To help users, ScyllaDB offers free data modeling courses at ScyllaDB University, aimed at both beginners and advanced users.

ScyllaDB has announced the release of ScyllaDB Manager versions 2.0.2 and 1.4.4, both of which serve as bug fix updates for their respective branches of the ScyllaDB Manager, a system designed to automate maintenance tasks on ScyllaDB clusters. Version 2.0.2 addresses issues such as a memory leak in the AWS SDK that affected memory consumption during S3 uploads, introduces a new script for setting up cgroups for the Manager Agent, and enhances the resilience and stability of S3 uploads and large file listings. Version 1.4.4 focuses on improving ssh communication reliability and adjusts repair processes to skip keyspaces without replicas in specific data centers. ScyllaDB Enterprise customers are encouraged to upgrade to these latest versions for better performance and stability, with guidance available for transitioning from earlier versions.

ScyllaDB Enterprise 2019.1.7 is a production-ready patch release focused on enhancing the stability and resolving various bugs of the high-performance NoSQL database, specifically tailored for real-time big data workloads. This release encourages ScyllaDB Enterprise customers to upgrade in collaboration with the ScyllaDB support team to ensure a seamless transition. Key issues addressed include resolving a long-running cluster problem where incorrect gossip generation occurred upon node restarts and preventing active nodes from being wrongfully marked as down. Additionally, a rare issue causing node crashes during upgrades, related to table scanning with specific partition configurations, has been fixed.