January 2015 Summaries

The Cassandra Cluster Manager (CCM) is a Python-based tool that enables users to create Apache Cassandra clusters and perform various cluster operations on localhost. It was recently updated with Windows support, allowing native running of Cassandra instances on the platform. To set up CCM on Windows, users must install Java Development Kit (JDK), Apache Ant, and Python 2.7, as well as configure system variables accordingly. Once installed, CCM can be used to create clusters, start and stop nodes, and access cluster information via cqlsh or nodetool. Although not recommended for production use, CCM provides a convenient development environment for working with Apache Cassandra on Windows systems.

The presenter, Helena Edelson, discussed Spark, Spark Streaming, Cassandra with Kafka and Akka in a webinar attended by over 1700 participants from around the world. She highlighted how these technologies are suitable for lambda architecture due to their shared features and strategies. The integration of these tools can create a streaming data platform for real-time delivery of meaningful information at high velocity, within a highly distributed, asynchronous, parallel, fault-tolerant system. The Q&A session covered various topics such as the speed of Hive or Spark when using BYOH (with Spark being faster), memory requirements for Spark cluster to query HDFS data, status of Spark R (Alpha), user friendliness of the solution (API requires technical knowledge/training but a UI is planned), implementation of moving/sliding windows with Spark Streaming (possible as it's built-in), use of Spark Streaming from HDFS (yes, through several operations for streaming from and to any HDFS-compatible sources and sinks), elasticity of scale in Cassandra (easier with VNodes enabled but not required), real-time data calculation capabilities of the solution (good due to Spark's in-memory computing), availability of Spark Cassandra Connector (in DSE 4.5 and greater, publicly available from 4.6 onwards), handling of splits by Cassandra with VNodes on (a single spark partition queries data residing on a single Cassandra node), common deployment patterns for Kafka -> Spark -> Cassandra (co-located nodes for Spark and Cassandra, separate nodes for Kafka, local Kafka clusters in each DataCenter recommended), Kafka's management of consumer overwhelm (consumers are responsible for fetching data from Brokers and managing their own state), differences between Spark, Logstash, and Kibana (Spark is a cluster computing framework for large-scale data processing, while Logstash and Kibana are tools for aggregating and processing text logs and providing UI respectively), interaction with JDBC datasources by Spark (via JdbcRDD), handling of transactions in Cassandra (through PAXOS to implement a quorum based "lightweight" transactional system), use of any column in a WHERE clause in SparkSQL (yes, all columns are supported now), and overcoming flushwriter blocking issues using this methodology (by reducing cores, not using batch size in rows = 1).

Erich is the CTO of SimpleRelevance, a company that specializes in dynamic content personalization using data science tools. Before joining SimpleRelevance, he worked on scalable distributed architectures and studied mathematics and computer graphics in college. He enjoys studying category theory and functional languages like F# and Clojure. This article introduces Apache Spark, a recent Hadoop successor that supports both batch and stream processing, multiple programming languages (Scala, Java, Python), in-memory computations, an interactive shell, and an easier-to-use API compared to Hadoop. The focus of the article is on setting up Spark with Cassandra and providing a small example of what can be done with Spark. Erich shares his experience learning Spark and how he used it in a talk at the Chicago Cassandra meetup. He emphasizes the benefits of in-memory computations, interactive shell, and Spark's API design, which is based on Resilient Distributed Datasets (RDDs). The article also provides steps to set up a server with both a Spark node and a Cassandra node, as well as instructions for connecting Spark to a Cassandra cluster using the Cassandra Connector.