May 2017 Summaries

The blog post explores a financial use case involving the integration of Apache Kafka and ScyllaDB for streaming and near-real-time messaging to track stock prices. It describes a system where stock price updates are pushed to a Kafka queue and then consumed by subscribers interested in specific companies, with the consumed messages being stored in ScyllaDB for future analysis. The post explains the data modeling approach using Kaggle's New York Stock Exchange dataset to simulate real-time messaging and outlines the data pipeline stages, including parsing data, defining Kafka topics, simulating message streaming and consumption, and using ScyllaDB for data storage. It also provides details on the software setup prerequisites, such as Java, Maven, Kafka, Zookeeper, and the Cassandra Java core driver, and includes links to code samples and configuration files to help readers build and run the described system. The post emphasizes the importance of speed, reliability, and scalability in handling financial data, highlighting the performance benefits of using Kafka and ScyllaDB.

Building a highly available time-series solution involves using KairosDB and ScyllaDB to efficiently manage and scale data storage and retrieval. KairosDB acts as a front-end framework for ingesting and retrieving sensor information or metrics, while ScyllaDB serves as a high-performance backend database capable of indefinitely scaling to store vast amounts of time-series data. The integration of Collectd helps push ScyllaDB metrics into KairosDB, which then stores the data on a separate ScyllaDB cluster. This setup eliminates issues related to siloed monitoring and scaling by allowing the ScyllaDB storage layer to expand infinitely. The process involves configuring KairosDB to work with ScyllaDB, ensuring SELinux is disabled on the monitored ScyllaDB cluster, and using tools like Cassandra-stress to generate traffic, which can then be monitored using KairosDB's web UI or REST-API for executing queries and visualizing metrics.

ScyllaDB 1.7, a minor release of the open-source NoSQL database compatible with Apache Cassandra, introduces several enhancements and new features aimed at improving performance and compatibility. Key updates include the support for experimental Counters as a native data type, an upgrade of Java tools to align with Cassandra 3.0, and a CQL update to version 3.3.1, which introduces new data types like tinyint, smallint, date, and time. The release also debuts a new intra-node sharding algorithm designed to enhance performance across large clusters by better distributing data among cores. Debian support has moved from experimental to production-ready status. Numerous bug fixes address issues such as early memtable flushing on many-core machines, ensuring better disk usage, and several CQL-related improvements, including support for floating-point data types in aggregate functions. The release notes also highlight changes to metrics, urging users to update their monitoring dashboards, and provide detailed instructions for upgrading and configuring systems to utilize the new features effectively.

Amazon's introduction of the AWS i3 instance family offers significant advantages for I/O intensive applications, with features such as up to 15TB of fast storage and powerful Intel processors allowing up to 64 vCPUs per instance. ScyllaDB, known for its shard-per-core, lockless architecture, is well-suited to leverage the capabilities of i3 instances, achieving impressive performance metrics such as over one million operations per second in a single server. While ScyllaDB has yet to release official AMIs for i3 instances, customers have reported successful production deployments, although manual tuning is currently required due to kernel compatibility issues with the standard CentOS kernel. The i3 family is not only more powerful but also more cost-effective than its predecessor, the i2 family, promising significant reductions in operational costs. An upcoming ScyllaDB version is expected to offer full support for i3 instances, and a webinar is scheduled to guide users on optimizing the use of these new instances.

The blog post by Seva Morotskiy demonstrates how to utilize the Spark Scala API in conjunction with ScyllaDB to analyze flight delays and cancellations using a public dataset from the Research and Innovative Technology Administration (RITA). The dataset, containing approximately 120 million records of U.S. commercial flight data from 1987 to 2008, is processed to extract average arrival and departure delays, and cancellations for various air carriers. The analysis involves loading the dataset into ScyllaDB using the "Loader" module and then querying it with the "Extractor" module to identify top destinations and carriers with the highest delays and cancellations. The post details the setup and configuration of the required environments, including ScyllaDB, Java, Scala, Sbt, and Spark, and provides code snippets for executing the analysis. The analysis reveals insights such as the top three destinations with the highest average arrival delays and the carriers with the most cancellations, showcasing ScyllaDB's capability to handle large datasets efficiently for real-time data analysis in the airline industry.