October 2018 Summaries

In a preview of the ScyllaDB Summit 2018, senior software engineers Ľuboš Koščo and Michal Šenkýř from Sizmek, a leading Demand-Side Platform in ad tech, discuss their experiences and challenges in processing 50 billion user profiles in real time using ScyllaDB. Ľuboš, with a background in data center management, and Michal, who transitioned from a media company, both emphasize the importance of ScyllaDB's low read latency and scalability for handling massive datasets crucial for efficient ad targeting and real-time bidding. They highlight Sizmek's commitment to delivering high-performance, AI-driven programmatic marketing solutions that stand out in the industry. The engineers also share insights into their data management strategies, including the use of ScyllaDB to replace core infrastructure components and manage large-scale data growth, which supports complex decision-making in ad customization. Additionally, they express anticipation for updates from ScyllaDB, particularly version 3.0, and the opportunity to learn from other users at the summit.

IBM's acquisition of Red Hat in 2018 marked a significant move in the technology sector, as Red Hat was a pioneer in the open-source business model, significantly impacting the industry's landscape by promoting openness and innovation. Red Hat's influence is evident in its substantial contributions to Linux and leadership in projects like Kubernetes and OpenStack. The merger intended to leverage both companies' strengths to capitalize on the growing hybrid cloud market, expected to reach nearly $100 billion by 2023, by offering flexible, non-lock-in solutions that integrate seamlessly with existing cloud services. IBM and Red Hat's combined assets, such as OpenShift and IBM Cloud Private, aim to provide a competitive edge through a comprehensive service offering that matches the convenience and cost-effectiveness of public cloud vendors, while maintaining customer choice and flexibility. This strategic alignment underscores the potential for hybrid cloud solutions to offer better financial margins compared to solely using public cloud services, as demonstrated by companies like Dropbox. The success of this endeavor depends on providing a unified and straightforward user experience with out-of-the-box solutions, positioning IBMHAT to capture market share in this evolving landscape.

In a preview for the ScyllaDB Summit 2018, speakers Brian Hawkins and Bin Wang discuss the integration of ScyllaDB and KairosDB in smart vehicle diagnostics. Brian highlights KairosDB's scalability and customizability, making it a standout choice among time-series databases, while Bin explains its application at Faraday Future for managing automotive real-time data. Faraday Future utilizes a data pipeline involving RabbitMQ, Spring Cloud, and Kafka to encode and store hundreds of signals per second from vehicles in KairosDB and persist them in ScyllaDB. This system is complemented by Grafana for visualization, Spark for data analysis, and Spring Cloud for API management. Brian emphasizes the importance of scaling to handle data flow and understanding tag cardinality to optimize query performance. The summit aims to showcase such innovative uses, encouraging participants to explore these technologies further.

ScyllaDB announced the release of three maintenance updates: ScyllaDB Enterprise 2018.1.6, ScyllaDB Open Source 2.3.1, and ScyllaDB Open Source 2.2.1, encouraging users to upgrade with assistance from their support team. These updates address a critical issue with the "nodetool cleanup" feature, which could mistakenly erase local token ranges, alongside several CQL syntax errors and performance issues. The releases are backward compatible, supporting rolling upgrades, and they resolve various bugs including race conditions, schema compatibility issues, and problems with CQL aggregates and queries. Users who have questions or concerns about these updates, particularly those related to the "nodetool cleanup" issue, are advised to contact ScyllaDB support for assistance.

MongoDB's transition from the AGPL to the SSPL license reflects the challenges open-source software vendors face in protecting their business models amid the rise of cloud services and software as a service (SaaS). The SSPL license aims to prevent companies from offering MongoDB as a cloud service without contributing back to the community, broadening copyleft requirements to include all associated service software. While the intention is to curb unfair commercial use, critics argue that SSPL's demands could be overly burdensome and impractical, potentially stifling collaboration and innovation. The article suggests that instead of creating a new license, MongoDB might have been better served by adopting the Commons Clause or sticking with AGPL, as SSPL's broad reach may inadvertently generate more complications than solutions. Despite these licensing shifts, the open-source movement remains robust, with evolving business models to sustain its growth, but MongoDB's approach raises questions about the balance between open-source ideals and commercial interests.

ScyllaDB Manager 1.2 introduces enhancements to streamline the maintenance of ScyllaDB clusters, notably through improved repair features. The update simplifies configuration by removing the "repair unit" concept, allowing tasks to dynamically encompass new tables matching task definitions. It offers multi-datacenter repair capabilities, enabling users to isolate repairs to specific nodes, tables, or token ranges, and schedule them with precision using timestamps and time deltas. The update also simplifies SSH connectivity setup, enhancing security without requiring changes to database configurations, with the introduction of the scyllamgr_ssh_setup script for automated SSH key management. Additional improvements include the deployment of the ScyllaDB Manager server API using HTTPS by default, topology-aware cluster registration, and the dynamic determination of shard counts, allowing for more efficient management of diverse node sizes within a cluster.

Child Rescue Coalition (CRC), a non-profit organization, uses cutting-edge technology to protect children from online predators by developing and maintaining tools like the Child Protection System (CPS), which is utilized by investigators across the U.S. and in over 90 countries to track and prosecute pedophiles. CRC's software solutions handle more than 17 billion records, generating millions of leads daily to assist law enforcement in identifying and apprehending offenders. The organization employs ScyllaDB for efficient data storage and management, allowing for scalable and cost-effective operations with minimal server usage. CRC also leverages technologies such as Kubernetes, Kafka, and Apache NiFi to enhance platform flexibility and data processing capabilities. Ensuring data privacy and proper use of information is a critical aspect of their operations, continually validated in court to demonstrate compliance. The coalition invites public support through donations, membership, and corporate sponsorships to fund awareness programs, training, and software development.

In an interview leading up to the ScyllaDB Summit 2018, Aravind Srinivasan, a Staff Software Engineer at Grab, discusses his journey into database technologies and the strategic decision to employ ScyllaDB at Grab, Southeast Asia’s leading logistics company. He recounts his transition from working with filesystems at Isilon Systems to building a queuing system at Uber using Cassandra, which eventually led to operational challenges. At Grab, he sought a high-performing, low-overhead metadata store and found ScyllaDB suitable due to its ability to handle rapid increases in traffic volume. His upcoming talk at the Summit will cover Grab's use of ScyllaDB, its integration with AWS and Kafka, and the unique scaling patterns of their use case, while also advising attendees to familiarize themselves with technologies like Kafka and stream processing to gain the most from his presentation.

In an interview leading up to the ScyllaDB Summit 2018, Kefu Chai, a Software Engineer at Red Hat, discusses his presentation on rebuilding the Ceph distributed storage solution using the Seastar framework. Chai, who has an extensive background in developing kernel modules and middleware, explains how the need for higher throughput and lower latency in storage solutions led to adopting Seastar, known for its share-nothing model and zero locking tolerance. He highlights Ceph's appeal as an open-source, decentralized storage platform offering unified block, object, and filesystem access. Chai also notes the challenges of integrating Seastar into Ceph, a dynamic project not originally designed for Seastar's architecture, and mentions the ongoing process of rebuilding Ceph's infrastructure with Seastar. He plans to explore Seastar's impact on performance, emphasizing the importance of understanding threading models for attendees of his talk.

ScyllaDB Manager 1.2 has been released for ScyllaDB Enterprise customers, offering enhanced control over repair tasks by enabling repairs on specific data centers, nodes, and token ranges, improving deployment and setup processes, and bolstering security features. This version introduces the capability to perform more granular repairs with new command-line flags, such as --dc, --host, and --token-ranges, and includes a new script to simplify managing SSH configurations across clusters. It now supports Debian 8 and Ubuntu 16 packages and defaults to using HTTPS for connecting with ScyllaDB nodes. Additionally, ScyllaDB Grafana Monitoring 2.0 has been updated to include a dashboard for ScyllaDB Manager 1.2, with revised metrics for better monitoring. ScyllaDB Manager, which offers centralized cluster administration and task automation, is positioned to become the hub of ScyllaDB Enterprise cluster management with plans for future enhancements such as rolling upgrades and a GUI frontend.

In an interview leading up to the ScyllaDB Summit 2018, Hojjat Jafarpour, a software engineer at Confluent and creator of KSQL, discusses his upcoming presentation on scalable stream processing with KSQL, Kafka, and ScyllaDB. Jafarpour highlights Kafka's rapid adoption due to its ability to decouple data producers and consumers, simplifying enterprise data management architectures and serving as a core component in microservices architectures. He explains that KSQL, unlike Cassandra Query Language (CQL), operates as a Streaming SQL engine with continuous queries, enabling real-time data processing. The flexibility and ease of use of KSQL make stream processing accessible to a wider audience beyond just software engineers. Jafarpour advises that attendees of the summit only need a basic understanding of Kafka, as he will provide an introduction to KSQL during his talk.

ScyllaDB Enterprise 2018.1.5 is a minor release of the enterprise NoSQL database that primarily addresses bug fixes for the 2018.1 branch, along with significant enhancements to single partition scans. Customers are advised to upgrade to this version while coordinating with the ScyllaDB support team, though they should be aware that the downgrade procedure differs from previous versions. The release resolves multiple issues, including problems with CQL operations like DISTINCT queries, user-defined type keyspace drops, and certain partition scans, as well as addressing performance and stability concerns such as latency spikes and infinite loops in specific scenarios. Additionally, a new monitoring counter for speculative retries has been introduced.

In this part of the blog series on integrating Apache Spark with ScyllaDB, Itamar Ravid explores how Spark DataFrames can be written back to ScyllaDB, building upon previous discussions on reading data. The article provides a practical guide on writing Spark DataFrames to ScyllaDB using the DataStax Cassandra connector, highlighting the importance of schema compatibility and the potential issues that can arise from data type mismatches. It details the process of creating DataFrames, mapping their schemas to ScyllaDB tables, and executing write operations in parallel across partitions. Additionally, the post emphasizes the laziness of data fetching during these operations, which allows for efficient memory usage by only loading necessary data batches. The blog concludes with a teaser for the next installment, which will cover streaming workloads and their integration with ScyllaDB.

DC/OS, an open-source distributed operating system based on the Apache Mesos kernel, facilitates the management of multiple machines either in the cloud or on-premises through a single interface, enabling the deployment of containers, distributed services, and legacy applications. This article discusses the deployment of ScyllaDB, a high-performance NoSQL database, on DC/OS, highlighting the process of optimizing container performance through host and network tuning, and CPU core allocation. The setup involved creating a small cluster with a DC/OS Master node and three Agent nodes, where ScyllaDB containers were configured to maximize performance by tuning the hosts and pinning containers to specific CPU cores. Performance checks through stress tests demonstrated the cluster's capability to sustain over 1.3 million operations per second. Additionally, the article outlines day-to-day operations such as running queries and commands, monitoring using Prometheus and Grafana, and handling potential host failures, ensuring that ScyllaDB runs efficiently in a DC/OS environment.