April 2019 Summaries

ScyllaDB Monitoring Stack 2.3 has been released, providing an open-source monitoring solution for ScyllaDB Enterprise and Open Source, utilizing Prometheus and Grafana. This release supports various versions of ScyllaDB Open Source and Enterprise, as well as ScyllaDB Manager, and introduces precompiled dashboards for ScyllaDB Enterprise 2019.1 and ScyllaDB Manager 1.4, reducing the dependency on Python for users not modifying dashboards. Enhancements include new panels for cross-shard traffic identification, multi-cluster installation simplification by displaying cluster names, and improved configuration options with multi-datacenter support. Additionally, Prometheus has been upgraded to version 2.7.2, and new panels for CPU usage by compaction and storage usage over time have been added, alongside configurable alerts and a warning for running Docker as root. Bug fixes and other minor improvements have also been incorporated in this version.

ScyllaDB Monitoring Stack 2.3 introduces several enhancements aimed at improving robustness, security, and usability, particularly with the upcoming ScyllaDB Enterprise 2019.1 release. Notable changes include pre-generated Grafana dashboards to eliminate the dependency on Python, refined Docker container usage to mitigate permission issues without resorting to insecure workarounds, and the ability to override Prometheus alert and Alertmanager configuration files via the command line. The release also enhances the genconfig.py utility for better configuration of datacenters and clusters, while adding new panels to the CQL optimization dashboard to identify cross-shard traffic and a disk usage panel for tracking disk space over time. These updates are designed to streamline the monitoring process and bolster security, with ScyllaDB encouraging user feedback on the open-source platform available on GitHub.

Veramine, a company that provides cybersecurity solutions for enterprises and national security, faced significant challenges in managing the vast amounts of sensitive data required for advanced threat detection and intrusion response. Initially using PostgreSQL and then Cassandra for data storage, Veramine encountered operational difficulties, particularly with Cassandra's tendency to crash and its high administrative demands. In search of a more reliable and low-latency database solution, Veramine transitioned to ScyllaDB, a NoSQL database praised for its stability and minimal management overhead. The switch to ScyllaDB allowed Veramine to improve their system's performance, focus on business logic, and reduce the complexity of managing their data infrastructure, ultimately leading to enhanced developer productivity and more efficient cybersecurity operations.

ScyllaDB Open Source Release 2.3.5, announced by Tzach Livyatan on April 18, 2019, is a bugfix update for the ScyllaDB Open Source 2.3 stable branch, ensuring backward compatibility and supporting rolling upgrades. Although users are encouraged to upgrade to the latest stable branch, release 3.0, this update addresses specific issues such as a CentOS 7 setup error due to a change in EPEL, potential disk space doubling during nodetool cleanup, indefinite delays in schema change statements with constant schema pulls, exceptions in JSON serialization of decimal and variant data types, and potential aborts in row_cache during concurrent MemTable flushes. The release is available in Docker and binary packages, and users are invited to report any issues encountered.

Natura, a leading Brazilian beauty products company with a global footprint, transitioned its data management system from DataStax Enterprise (DSE) to ScyllaDB to enhance performance and reduce operational costs. With its massive operations spanning 70 countries and serving over 100 million consumers, Natura required a robust data infrastructure to handle the vast amounts of consumer data. The company, led by Big Data Engineer Felipe Moz, opted for ScyllaDB to overcome the performance issues faced with DSE, particularly related to JVM problems. The new architecture, which leverages technologies like NGINX, Node.js, Apache Kafka, and Apache Spark, enabled significant cost savings and performance improvements, with batch processing times reduced by up to 90% and streaming analytics sped up dramatically. The switch from DSE on Microsoft Azure to ScyllaDB on AWS resulted in over a 50% reduction in annual hardware expenses, while also achieving faster database performance, evidenced by a dramatic decrease in write latencies. ScyllaDB's enterprise support and lower costs provided Natura with a more efficient and scalable solution to support its extensive consumer network.

ScyllaDB Open Source 3.0.5 has been released as a bugfix update for the 3.0 stable branch, addressing several key issues, including a regression from version 3.0.4 that caused increased latency in write operations for users with Hinted Handoff enabled. This release maintains backward compatibility and supports rolling upgrades. Among the other resolved issues are problems related to materialized views generating redundant timeouts, failures in migrating Apache Cassandra SSTables in certain formats, inefficiencies in cluster restarts, potential memory access errors, and various hardware-specific and script-related errors. The update aims to enhance the stability and performance of ScyllaDB, offering multiple ways to access the new version, including Docker, binary packages, and EC2 AMI.

ScyllaDB Cloud is a managed NoSQL Database as a Service (DBaaS) that alleviates users from the complexities of cluster management, such as scaling, node replacement, and monitoring security settings. The onboarding process involves creating an account, setting up a cluster with customizable options like cloud provider, instance type, and replication factor, and optionally enabling VPC peering. Users can monitor their clusters through a user interface that displays key metrics, and they can resize or delete clusters via the "Actions" tab. ScyllaDB Cloud offers robust support through a Zendesk account, enabling users to submit tickets for technical issues with prioritized responses based on the severity of the problem. The service also includes regular updates and maintenance, ensuring the ScyllaDB Enterprise edition is up-to-date with critical patches. Future content will provide guidance on migrating data to ScyllaDB Cloud, with users encouraged to provide feedback through various channels.

ScyllaDB Cloud is a newly launched managed NoSQL database service that is designed to deliver high performance and scalability at a lower cost compared to other Database as a Service (DBaaS) offerings. Initially available on Amazon Web Services (AWS) with plans to expand to Google Cloud and Microsoft Azure, ScyllaDB Cloud utilizes the latest server technologies to minimize the number of servers needed, thereby reducing costs. It offers features like automatic backups, upgrades, performance optimization, and security hardening, and is compatible with the Cassandra ecosystem, facilitating easy data migration. ScyllaDB Cloud promises consistent low latencies and high throughput, making it suitable for both startups and large enterprises seeking a cost-effective solution to manage large-scale applications. The service is designed for easy integration and quick deployment, allowing developers to focus on creating engaging user experiences rather than managing database infrastructure.

Helpshift, a digital customer service platform, transitioned to a sharded database model to accommodate its rapid growth and evolving user profile requirements, ultimately selecting ScyllaDB for its high throughput and scalability. This decision was driven by the need to decouple user identity from devices and to facilitate proactive support, which necessitated a more robust database capable of managing an expected 150+ million users and 30K requests per second. After evaluating various databases like Cassandra and Google Cloud Spanner, ScyllaDB was chosen for its ability to handle complex queries, ease of scaling, and superior performance metrics. Helpshift successfully benchmarked ScyllaDB with a cluster of 10 nodes, achieving 32,000 user reads per second and a 114 ms average latency, which later scaled to 73K user reads per second with a 13 ms latency as the platform continued to grow. The seamless integration and scalability of ScyllaDB have proven instrumental in supporting Helpshift's expanding operations, allowing the company to manage increased demands without bottlenecking its backend systems.

Peter Corless's blog post discusses various strategies and tools for migrating data to and from Cassandra or ScyllaDB clusters, emphasizing the importance of choosing the right approach based on specific use cases. The article covers both online and offline data migration techniques, highlighting the complexities and considerations involved, such as schema modifications and data validation. Offline migrations allow for downtime, simplifying the process, while online migrations require dual writes to both old and new systems, maintaining operational continuity. The blog introduces multiple methods for existing data migration, including CQL COPY for loading data from CSV files, SSTableloader for bulk loading from disk-compatible databases, and Mirror Loader for creating temporary clusters to facilitate fast data transfer. The ScyllaDB Spark Migrator is presented as a robust option for streaming data between CQL-compatible databases, offering resilience to failures and the ability to preserve timestamps and schema changes. Throughout the discussion, Dan Yasny, a Field Engineer at ScyllaDB, provides insights on the practical aspects of data migration, emphasizing the need to carefully manage resources and system load during the process.