July 2013 Summaries

Cassandra 2.0 introduces several enhancements to the CQL front, including the ability to drop columns and support for lightweight transactions using IF keyword in INSERT, UPDATE, and DELETE statements. It also offers conditional schema modification statements and experimental support for triggers. Secondary indexing is now allowed on non-primary key CQL3 columns, but not yet on collection columns. Additionally, Cassandra 2.0 introduces the second version of its native protocol for CQL, which brings improvements over the first version.

Anti-entropy repair in Cassandra can be challenging due to its potential impact on disk IO and the need for running it before gc_grace expires. Reliable hints can help alleviate this issue, but if a node is lost, repair must still be performed. The process of repair involves two phases: building a Merkle tree and comparing differences between replicas. To mitigate disk IO issues during the first phase, compaction throttling can be used. However, using -pr option to repair only the primary range for a node may not fully resolve the problem as other replicas still have to perform the Merkle tree calculation. The -snapshot option can help by taking a snapshot of data and sequentially repairing from it, allowing only one replica at a time to perform validation compaction. Overstreaming is another issue that can occur during the second phase of repair when many partitions are sent due to fixed precision in Merkle trees. Subrange repair, available since Cassandra 1.1.11, allows repairing only a portion of data belonging to a node, effectively increasing precision and eliminating overstreaming behavior.

Domain Specific Languages (DSLs) are used by programmers to increase the expressivity of their code within a specific domain, making it easier for non-programmers to communicate with developers. DSLs abstract away low-level programming constructs in favor of higher level ones that better link to the elements of the domain itself. In the context of Pearson's OpenClass platform, a Gremlin-based DSL is used to extend the graph model into a programmatic construct, allowing engineers and analysts to ask questions of the graph in their educational domain language. The OpenClass DSL defines the graph schema, extends the Gremlin graph traversal language into the language of education, provides standalone functions that operate over these extensions, and exposes algorithms developed from those extensions and functions. This coarsely-grained API helps bring general accessibility to complex graph traversals.

The text discusses the concept of linearizable consistency in distributed systems and how it can be achieved using the Paxos consensus protocol. Linearizability ensures that operations appear to occur atomically, one at a time, even if they are executed concurrently. This is crucial for applications like user registration where ensuring uniqueness is important. The Paxos protocol allows a distributed system to agree on proposals with a quorum-based algorithm, without the need for a master node or two-phase commit problems. It consists of three phases: prepare/promise, propose/accept, and commit/acknowledge. Cassandra uses this protocol to provide linearizability in its lightweight transactions feature, which can be used for both INSERT and UPDATE statements with the new IF clause.

The text discusses the evolution of querying mechanisms in Cassandra, highlighting the transition from the first-generation thrift API to the more advanced CQL and CQL3. Initially, the thrift API was a low-level querying mechanism that directly interacted with Cassandra's baseline storage, making application development cumbersome. To address these inefficiencies, CQL was introduced in Cassandra 0.8, offering a modern query language that is backward compatible with thrift column families, thereby easing adoption for traditional database developers. CQL3 further advanced the querying capabilities by transposing wide rows into named columns and introducing features like collections and a binary protocol, which allowed for more natural paging through wide rows using classes like CqlPagingInputFormat. The second-generation Cassandra Hadoop driver leverages CQL3 as a high-level abstraction layer, simplifying the decomposition of composite keys and columns, which previously required complex client-side code. The text also addresses improvements in Pig and Hive drivers for Cassandra, where the second generation Hadoop Cassandra driver, using CQL3, provides better integration and efficiency by mapping input and output values to Hive data types and expanding default mappings to include collections. These advancements lead to more efficient and scalable data processing workflows in systems using Cassandra.

The text discusses the integration of Solr and Hadoop with Cassandra in DataStax Enterprise, a big open source technology. It explains how running MapReduce operations over data and doing searches on that same data can be done easily without needing to ETL the data between two clusters. The example used is a survey dataset from The Pew Research Center about Facebook habits and attitudes. The text demonstrates how to create a Solr schema file, upload it to DSE Solr, and then import the survey csv data for processing. It also explains how to search using SOLR's HTTP API and run MapReduce jobs over the imported data using Hive.

The text discusses an on-boarding primer for new hires in DataStax's Test Engineering organization, focusing on understanding Cassandra's data storage and handling mechanisms. It covers topics such as CQL3 music playlist example, Cassandra Writes, nodetool documentation, how Cassandra stores data, sstable2json documentation, Cassandra Deletes, gc_grace_seconds, tombstones, anti-patterns in data modeling, and using cassandra-stress for illustration purposes. The primer also highlights the importance of understanding key concepts such as compaction activity and how it affects the system under heavy writes.