September 2017 Summaries

At the ScyllaDB Summit 2017, Nayden Kolev from mParticle shares insights into their migration from Cassandra to ScyllaDB, emphasizing the challenges and processes involved in adopting a new NoSQL database solution. As a Systems Architect and Production Operations lead at mParticle, Kolev discusses his role in overcoming scalability challenges and his passion for automating infrastructure processes. His talk targets DevOps professionals and engineers, detailing mParticle's pre-ScyllaDB environment, the deployment and migration processes, and maximizing the implementation post-migration. He highlights the crucial role of ScyllaDB's support engineers in ensuring the success of their deployment, suggesting that without such support, the migration could have been significantly more challenging. The ScyllaDB Summit, taking place in San Francisco, features over 40 talks, covering technical insights, the ScyllaDB roadmap, and hands-on workshops to optimize ScyllaDB cluster usage.

Lubos Kosco and Andrej Chu from Rocket Fuel are set to present at the ScyllaDB Summit 2017 on their use of ScyllaDB for Page Context Categorization, a technique to improve ad targeting by categorizing web page contexts. Their presentation will focus on how Rocket Fuel employs ScyllaDB for fast caching of page context information and IP address blacklists, crucial for maintaining data freshness and meeting strict SLAs with low latency. By leveraging ScyllaDB's big-table capabilities, Rocket Fuel aims to efficiently handle over 150,000 requests per second across multiple data centers, ensuring high availability and rapid response times. The talk is geared towards developers, system administrators, and DevOps who are interested in processing large data volumes with ScyllaDB, and will take place at the summit in San Francisco on October 24-25, 2017.

Vladislav Zolotarov's blog post explores the combination of ScyllaDB and Intel Optane SSD DC P4800X drives to achieve the performance of an in-memory database while addressing cold-cache and data persistence challenges. The experiment involved testing various scenarios using three ScyllaDB servers and three client servers, with impressive results in terms of latency and throughput. Using tools like Diskplorer and cassandra-stress, the study found optimal concurrency points and demonstrated ScyllaDB's ability to fully utilize infrastructure for high-throughput, low-latency workloads. The findings revealed that Intel Optane provides great latency results and is more cost-effective than traditional DRAM or Enterprise SSD configurations, achieving similar performance to all in-memory databases without compromising on essential metrics such as throughput, latency, and performance.

Burak Yavuz, a software engineer at Databricks, will be presenting at the ScyllaDB Summit 2017 on the topic of stateful streaming applications using Apache Spark's Structured Streaming. Yavuz, who is part of the team responsible for building internal streaming ETL pipelines at Databricks, will discuss how Structured Streaming simplifies the processing, cleaning, enrichment, and reporting of data by allowing the same business logic to be applied in both batch and streaming contexts, thus enabling lower latency data processing. His talk will focus on stateful operations and demonstrate how NoSQL stores can be integrated as fault-tolerant state stores and streaming sinks, which are beneficial for applications requiring real-time updates and low-latency data delivery, such as updating user recommendations based on recent purchases. The summit will also feature other technical discussions and workshops on big data and NoSQL technologies, scheduled to take place in San Francisco on October 24-25, 2017.

At the ScyllaDB Summit 2017, Duarte Nunes, a software engineer at ScyllaDB with expertise in concurrent programming and distributed systems, discussed the implementation of Materialized Views in ScyllaDB. Materialized Views in ScyllaDB are designed to optimize data access by storing a subset of columns from a base table, enabling diverse query capabilities while ensuring consistency between view and base tables. Nunes highlighted the challenges of supporting these views in a multi-master, eventually consistent environment, where decentralization is key to maintaining cluster availability and minimizing latency. The talk delved into the design and algorithmic strategies employed to manage update propagation and ensure consistency, particularly addressing the challenges posed by concurrent updates. Attendees interested in distributed algorithms and ScyllaDB's architecture were the primary audience for this technical discussion, and further insights were available through ScyllaDB's public Slack and mailing list.

ScyllaDB 1.7.5 has been released as a bugfix update for the stable 1.7 branch, maintaining backward compatibility and supporting rolling upgrades. Users upgrading from version 1.6.x are advised to review the release notes due to changes in some metric names. The release addresses several bugs, including issues with experimental Counters that may affect data when upgrading from 1.7.4, installation on CentOS, potential deadlocks and core dumps during initialization, shutdown problems with TLS, incorrect cell names in promoted index blocks, and schema update errors. Users are encouraged to report any problems encountered during the upgrade process.

ScyllaDB's new documentation series aims to enhance user understanding and optimization of ScyllaDB deployments by exploring various concepts and architectural design elements borrowed from Apache Cassandra, such as distributed workload and storage with eventual consistency. The focus is on ScyllaDB's Ring Architecture, where data is distributed across nodes in a cluster using tokens that are mapped via a hash function to token ranges within a keyspace. This approach is common in distributed databases like Amazon's DynamoDB and Apache Cassandra. The documentation covers essential components like clusters, nodes, keyspaces, tables, partitioners, and hash functions, providing a glossary of terms and practical guidance using the nodetool command. The site also offers resources for administrators and developers, instructions for modeling with CQL, and information on contributing to the ScyllaDB community, along with options to download or try ScyllaDB in various environments.

John Turner, VP of Engineering at MediaMath, will speak at the ScyllaDB Summit 2017 about how the company utilizes Apache Cassandra and ScyllaDB to manage user data and enhance Real-Time Bidding (RTB) processes. Turner’s talk will focus on the MediaMath Data Management Platform (DMP), which empowers marketers to effectively manage and activate data in real time across multiple channels. He will discuss the challenges faced with Apache Cassandra, such as performance at scale and how transitioning to a more on-demand, streaming process can address these issues. Additionally, Turner will outline the current and future architectural strategies involving ScyllaDB, emphasizing the importance of integrating machine learning models built by software rather than manually. The talk is aimed at professionals in the ad-tech and big data sectors, providing insights into overcoming data sparsity, processing time, and scalability challenges.

ScyllaDB 2.0 introduces a feature called heat-weighted load balancing to address performance issues that occur when a node in a ScyllaDB cluster loses its cache, such as during rolling upgrades. Traditionally, when a node restarts with an empty cache, the equal distribution of workload among nodes can lead to overloading cold nodes, resulting in higher latency and reduced throughput. Heat-weighted load balancing mitigates this by allocating fewer requests to the cold nodes, allowing their caches to gradually warm up without overwhelming them with disk I/O operations. This approach adjusts the distribution of requests based on the cache hit rates of each node, aiming to equalize cache miss rates across the cluster. The solution involves complex algorithms to manage these adjustments, especially when consistency levels require multiple replicas to handle requests. Experimental results demonstrate that this method successfully maintains low latency during node cache warm-ups, ensuring consistent performance.

Joshi Fullop of Los Alamos National Laboratory is set to present at the ScyllaDB Summit 2017, discussing the evolution of high-performance computing (HPC) and its applications in monitoring supercomputers. With a background in the Symera NT Supercluster project at NCSA, Fullop emphasizes the thrill and challenges of working in HPC, particularly in scaling and monitoring systems across thousands of nodes. He highlights the development of specialized tools like LDMS, Baler, and HELO to manage data collection and log classification, underscoring the need for scalable solutions in the field. His presentation will explore distinctions between HPC and cloud computing, evolving infrastructure, and a specific use case involving ScyllaDB. The summit promises a diverse range of talks for those interested in supercomputers and monitoring, offering insights into industry advancements and opportunities for networking with experts.

The Equifax data breach serves as a cautionary tale about the critical importance of timely software updates, as the incident was caused by the company's failure to patch a known vulnerability in the Apache Struts software, which had been disclosed two months prior. The breach highlights a common issue where many companies do not promptly update vulnerable software components, despite clear instructions from security experts, leading to significant security risks. While some argue that running cutting-edge software releases can introduce bugs and downtime, the author asserts that the risk of a security breach from outdated software is far greater. The solution lies in adopting agile processes for regular software updates, leveraging Continuous Integration/Deployment environments to test and deploy updates efficiently, and ensuring that all infrastructure runs on secure and stable software versions. The piece concludes by emphasizing the importance of maintaining up-to-date systems to protect sensitive data and company reputation.

ScyllaDB 2.0 introduces advancements in workload conditioning, particularly through the implementation of a userspace disk I/O Scheduler that efficiently manages disk resources to balance different database tasks. This system ensures Quality-of-Service (QoS) by isolating tasks and limiting resource utilization, notably improving latencies from 3.6ms to 0.8ms for the 99th percentile. The update supports the faster NVMe disks and the Amazon i3 family, addressing the shift from being disk-bound to potentially CPU-bound systems. ScyllaDB 2.0 incorporates an experimental CPU scheduler to manage CPU usage for background tasks like memtable flushes and compactions, crucial for maintaining stable foreground loads without sacrificing performance. By automatically determining optimal CPU quotas for memtable flushes and capping CPU usage during contention, ScyllaDB reduces high percentile latencies and enhances consistency, as evidenced by improved benchmark results. Future versions aim to further automate this quota adjustment in response to varying workloads, promising even more robust performance optimization.

At the ScyllaDB Summit 2017, Lukasz Pachciarek and Szymon Szymanski from Allegro Group discussed the advantages of ScyllaDB over Apache Cassandra, particularly in addressing some of Cassandra's shortcomings. Allegro, a major e-commerce platform in Central Europe, has been managing nearly 20 Apache Cassandra clusters and was motivated to test ScyllaDB's performance in their environment after attending a previous summit. Their talk focused on achieving predictable and low response times, exploring both performance and administrative tasks, and they devised a hardware reduction plan suggesting that fewer ScyllaDB nodes could replace multiple Cassandra nodes. The discussion was aimed at system and database administrators, developers, and managers, with further information about their findings available only through direct contact on LinkedIn, as their internal tests were not published. The summit featured a range of talks and workshops, providing insights into the ScyllaDB roadmap and practical guidance for optimizing ScyllaDB clusters.

IBM software engineers Chin Huang and Ted Chang are set to present at the ScyllaDB Summit 2017, where they will share performance test results and insights from their evaluation of ScyllaDB as a storage backend for JanusGraph, compared to Apache Cassandra and HBase. Their talk will target performance engineers, data architects, and IT/Operations professionals interested in NoSQL technologies, focusing on scenarios involving high-volume reads and writes. The engineers aim to identify the best-performing backend for the highly scalable graph database solution JanusGraph, traditionally supported by Cassandra and HBase. Attendees can connect with Chin and Ted through GitHub for further interaction. The summit, featuring various talks on ScyllaDB use cases and a hands-on workshop, is scheduled for October 24-25 in San Francisco, CA.

Pythian's Principal Consultant, John Shulz, will be presenting at the ScyllaDB Summit 2017 in San Francisco, CA, where he will discuss how to extract useful operational information from databases, particularly comparing MySQL with Apache Cassandra and ScyllaDB. His talk is aimed at users of traditional relational databases who are considering transitioning to Apache Cassandra or ScyllaDB and need insights on cluster management. John will demonstrate how to utilize MySQL's information and performance schemas to gather operational insights and contrast this with methods used in Apache Cassandra and ScyllaDB. The ScyllaDB Summit, scheduled for October 24-25, will feature a variety of technical talks, workshops, and discussions on the ScyllaDB road map, providing a platform for NoSQL developers and users from startups and enterprises to share ideas and innovative use cases.

ScyllaDB has created a comprehensive monitoring cheat sheet to assist users in efficiently overseeing their ScyllaDB deployments. This resource consolidates various monitoring methods and tools, covering everything from operating system statistics to database performance metrics. It includes basic administrative monitoring commands and instructions on gathering metrics using Prometheus, JMX via JConsole, and the REST-API. By providing this all-in-one document, ScyllaDB aims to streamline the monitoring process, making it more accessible and straightforward for users to maintain their distributed platform.

Phillip Tribble interviews Hojjat Jafarpour of Confluent.io about his upcoming presentation at the ScyllaDB Summit 2017, focusing on streaming ETL from Apache Kafka using KSQL. Jafarpour, who works on KSQL, an open-source streaming SQL engine for Apache Kafka, discusses its role in simplifying real-time data processing through SQL-like semantics. The conversation highlights the performance advantages of using NoSQL with Kafka despite its complexities and targets an audience of software engineers, data scientists, and solution architects. The summit, set to take place in San Francisco on October 24-25, 2017, will feature various talks and workshops, providing opportunities for participants to engage with NoSQL developers and learn about the ScyllaDB ecosystem.

CERN's Miguel Pedreira discusses the ALICE experiment's database enhancements at the ScyllaDB Summit 2017, focusing on improving performance and scalability through migration to Apache Cassandra/ScyllaDB. The talk targets individuals interested in transitioning to an Apache Cassandra-based approach, highlighting lessons learned, performance results, and differences between SQL and NoSQL schema paradigms. It also covers the global metadata catalog facilitating distributed computing across 80 centers worldwide, and the use of big data tools like Spark for processing database information. The summit, held in San Francisco, offers opportunities to learn from NoSQL experts and explore technical innovations and ScyllaDB's roadmap.

Boaz Avital, the tech lead of Twitter's key-value storage team, is set to present at the ScyllaDB Summit 2017, where he will discuss the complexities of managing stateful systems, particularly distributed databases like Twitter's in-house Manhattan database. His talk will explore the challenges of operating distributed storage systems, such as maintaining availability and performance while managing data movement across thousands of nodes. Avital will share insights into Twitter's approach to simplifying the management of Manhattan, emphasizing the importance of generalized infrastructure to ensure operational efficiency and team sanity. The summit aims to bring together NoSQL developers and users to exchange ideas and learn from real-life use cases, featuring various technical talks, including those from the ScyllaDB team.

IBM software engineers David Pitera and Keith Lohnes are set to present at the ScyllaDB Summit 2017, discussing their work on JanusGraph on Compose, a managed GraphDB cloud solution utilizing ScyllaDB as its primary datastore. Their talk will introduce attendees to the realm of graph databases, detail the JanusGraph project and data model, and explain how ScyllaDB fulfills their requirements. The Summit, which will be held in San Francisco on October 24-25, 2017, offers a platform for NoSQL developers and users to engage in various technical discussions, including monitoring, latency improvements, and ScyllaDB's roadmap, complemented by a hands-on workshop. Attendees can find more information about Pitera and Lohnes's presentation on the JanusGraph website and can contact them via email for further inquiries.