May 2017 Summaries

The DataStax Enterprise (DSE) Resource Manager is a custom version of the Apache Spark Standalone cluster manager that provides functionality similar to a Spark Master when running Apache Spark™ applications with DSE. It enhances the open source Spark Master implementation by automating the management of the Spark Master and Spark Workers lifecycles, providing high availability through an Apache Cassandra®-based solution, offering distributed and fault tolerant storage for Spark Master recovery data, and simplifying configuration for client applications. In DSE 5.1, the introduction of the DSE Resource Manager adds more ease-of-use, security, and stability by replacing the Spark RPC communication mechanism between Spark Driver and Spark Master with a DSE native CQL protocol. This change bolsters the security of Spark applications and DSE while providing several significant benefits such as enabling plain password authentication, Kerberos or other future authentication mechanisms, and allowing for compression, TLS encryption, multiple contact points, and more precise control over communication between components.

The Cassandra File System (CFS) was the default distributed file system in DataStax Enterprise versions 2.0 to 5.0, designed for Hadoop and Spark workloads with temporary storage. However, it has been deprecated and replaced by the improved DataStax Enterprise File System (DSEFS), which is available as an option in DSE 5.0 and became the default distributed file system in DSE 5.1. Unlike CFS, DSEFS separates data storage from metadata storage, with metadata stored in Cassandra tables and file data stored directly on the node's local file system. This design offers several advantages, including better cache efficiency, reduced context switches, and lower connection costs. DSEFS also supports a modern, efficient, scalable, and highly available distributed file system, making it unnecessary to use CFS any longer.

This blog post discusses the management of compile-time and runtime dependencies for Apache Spark applications that connect to DataStax Enterprise (DSE) or open source Apache Cassandra systems. It provides a set of predefined bootstrap projects in Java and Scala languages, using various build tools such as sbt, gradle, and maven. The post also explains how to handle dependency conflicts with shading techniques.

DataStax has released its first non-beta versions of the Fluent APIs for DSE Graph, bringing full compatibility with Apache TinkerPop GLVs and additional functionalities to make graph application development faster and easier. The Fluent API allows users to interact with DSE Graph via the Gremlin Traversal API, providing a more familiar interface than the existing String-based queries interface, allowing compile-time checking, and easy navigation through the Traversal API within an IDE client-side. This release also exposes additional traversal features made especially for DataStax Enterprise Graph, such as DSE Search predicates and Geometric/Geographic-based search predicates.

Apache Solr-based DSE Search and Apache Spark-based DSE Analytics can be combined to enhance indexing capabilities in DataStax Enterprise (DSE) 5.1. This integration allows for improved performance in certain scenarios, such as count queries and filtering result sets. By enabling the spark.sql.dse.solr.enable_optimization configuration option, DSE Search can transform Catalyst predicates into Solr query clauses, optimizing analytics queries like "SELECT COUNT(*) where Column > 5" to be executed in near-real time. The performance benefits of using DSE Search are significant for count queries and filtering result sets, especially when retrieving a small portion of the total dataset. However, it is essential to note that these optimizations may not always be beneficial depending on data layout and hardware configurations.

The DseGraphFrame package provides a Spark base API for performing bulk operations and analytics on DSE Graph. It is inspired by Databricks' GraphFrame library and supports a subset of Apache TinkerPop Gremlin graph traversal language. Key features include reading DSE Graph data into a GraphFrame, writing GraphFrames from any format supported by Spark into DSE Graph, and tying together DSE Analytics and DSE Graph components more strongly than before. The package currently supports both Java and Scala Spark APIs.