April 2017 Summaries

Continuous Paging (CP) is a new feature in Datastax Enterprise that optimizes streaming bulk amounts of records from Datastax to the Java Driver. It's an opt-in feature and can be enabled by setting "spark.dse.continuous_paging_enabled" to true as a Spark configuration option. CP increases read speed by having the server continuously prepare new result pages in response to a query, reducing communication cycles between the DSE Server and DSE Java Driver. However, it uses more Cassandra resources and may not be suitable for all use-cases. It's integrated into DSE Server and the DSECassandraConnectionFactory, so it can only be used with DSE. CP is automatically disabled if the target of the Spark Application is not DSE or is not CP capable. Errors related to Continuous Paging often manifest as tasks failing in the middle of a Spark job, which are immediately retried and usually succeed on a second attempt. These failures can cause jobs to take longer due to some tasks needing to be redone. The feature provides significant speed improvements over normal paging methods used by DSE, but only when reading from Cassandra is the bottleneck in the pipeline.

Property graphs are a type of attributed, multi-relational graph where edges are labeled and both vertices and edges can have any number of key/value properties associated with them. They are more complex than standard single-relational graphs due to the presence of different types of vertices and edges. This complexity affects how graph algorithms are defined and evaluated. Property graph software often supports common centrality algorithms, but their implementation may vary depending on the specific structure of the property graph. The power of property graph algorithms lies in the ability to calculate numerous eigenvector centralities for the same property graph instance by considering all the graphs that exist within the graph.

Graph computing is unique in its application to structured data and statistical algorithms from graph theory and network science. Two canonical uses are "Friend-of-a-Friend" (querying) and PageRank (statistics). Statistical techniques analyze the shape of a graph, focusing on vertex contributions to overall structure. A graph-centric statistic typically yields a single descriptive number. Eccentricity is a vertex-centric statistic that maps every vertex to a single number representing the longest shortest path from it to every other vertex. Betweenness centrality measures for each vertex the number of shortest paths it exists on between every pair of vertices in the graph, with a computational complexity reaching O(|V|^3). In large-scale graph computations, it is important to avoid geodesics if possible. Identifying centrality correspondences must be determined on a case-by-case basis. The self-similar aspect of scale-free graphs makes it possible to calculate a correlation matrix on a smaller subgraph and maintain confidence that the subgraph correlations will generalize to the full graph.

MyDrive, an Insurance Telematics company, uses DataStax Enterprise (DSE) for its data analysis services to motor insurers. The company collects sensor data from smartphones or telematics hardware, processes and analyzes it to score drivers' performance and help insurers assess risk. Before using DSE, MyDrive used MongoDB but switched to Apache Cassandra due to increased data volumes and then migrated to DataStax Enterprise for its tested set of compatible components and reduced operational overhead. The company primarily stores time-series data in DSE and uses OpsCenter, Spark, and plans to explore DSE Graph. MyDrive advises startups considering DSE to focus on their product by leveraging the ease of use and reduced operational headaches provided by DSE.