November 2019 Summaries

DataStax has launched DataStax Examples, a collection of code snippets and examples designed to help users accomplish tasks faster with Apache Cassandra® and DataStax products. The open-source code is available on GitHub and includes demos for various topics. Users can contribute by asking questions in the community space or requesting new examples via email.

The Internet of Things (IoT) era is rapidly expanding, with global spend on IoT expected to increase from $212 billion in 2021 to $1.612 trillion by 2025. This growth presents new data management challenges for enterprises, as they need to analyze large volumes of data in real-time. IoT interactions are endless and can generate hundreds of terabytes or even petabytes of data over time. Legacy systems cannot keep up with these massive, fast-moving datasets, necessitating a modern approach to data management. Apache Cassandra is a powerful, flexible database designed for the modern world, helping manage hot, warm, and cold IoT data. Its built-in data tiering support ensures that hot data is stored in its current state and is the fastest to retrieve, while warm and cold data are stored in condensed states or data warehouses, respectively. Cassandra's masterless architecture allows for easy scaling without downtime, and it was built to be 100% available with automatic replication across data centers and clouds. Cassandra's versatility enables organizations to deploy the database in any environment, accommodating rapid application development, Docker, and Kubernetes. By unlocking the full power of IoT data, Cassandra ensures delightful user experiences in an increasingly connected world.

The DataStax Enterprise Kubernetes Operator simplifies the deployment and management of DataStax Enterprise (DSE) clusters on the Kubernetes platform, which is increasingly being used for hybrid and multi-cloud solutions. This operator handles the role of deploying and managing DSE clusters within k8s namespaces by using a simple YAML file to define the number of nodes and storage requirements. The rest of this blog post covers setting up and running the operator on a local Red Hat OpenShift 4 installation, including instructions for downloading and submitting a YAML file containing the resources required to run the operator. Once the operator is up and running, DSE clusters can be described as logical DSE data centers with specifications such as number of nodes, storage requirements, and configuration tweaks. The operator then provisions missing resources and manages them for the user.

The 2019 holiday season is projected to be the largest in history, with online sales expected to increase by as much as 12%. However, this growth also corresponds to an increase in fraud attempts. Graph technology can help retailers prevent fraud during this busy period. Unlike relational database management systems (RDBMS), graph databases allow organizations to analyze all their data at scale and identify patterns of fraudulent behavior. They can be used to set rules for transactions, detect trends over time, and automatically freeze suspicious transactions in real-time. By leveraging the relationships between data sources, retailers can improve their fraud detection and prevention abilities during the holiday season.

The DataStax whitepaper "Data Modeling in Apache Cassandra™" provides a guide on choosing the right data model for your Apache Cassandra application in five steps. It highlights that while CQL resembles SQL, understanding how Cassandra stores data is crucial to avoid problems. Using an example of partitioning users into groups, it demonstrates the importance of using groupname as a partition key for fast lookup. The whitepaper also discusses the challenges of normalizing database schemas and recommends creating query-specific tables or using composite partition keys to reduce the number of rows per partition. Secondary indexes are another solution but should only be used when queries return tens or hundreds of rows at most, and when the indexed column has medium cardinality.

Apache Pulsar supports multiple groups of consumers (or consumer groups) on a topic, allowing multiple applications to consume messages from a single topic simultaneously. This is achieved through subscriptions that manage consumers on topics and keep track of their positions in the topic. A topic can support many subscriptions at once, each with one or more consumers. Pulsar offers four types of subscriptions: exclusive, failover, shared, and key_shared, which vary based on how messages are sent to consumers within a subscription. This feature enables message fan-out to multiple applications by simply adding subscriptions to the topic.

Apache Pulsar supports multiple groups of consumers on a topic, allowing multiple applications to consume messages from a single topic simultaneously. This is achieved through subscriptions, which manage consumers on topics and keep track of the position in the topic from which each consumer reads. A Pulsar topic can support many subscriptions at the same time, enabling message fan-out to multiple applications. Apache Pulsar offers four different types of subscriptions: exclusive, failover, shared, and key_shared. These subscriptions vary based on how messages are sent to consumers within that subscription. By using these subscriptions, developers can implement various patterns such as competing consumers or active-standby configurations.

The text discusses data modeling in Apache Cassandra, focusing on the use of collection data types and user-defined types (UDTs). It explains that using collections can be more efficient when storing small lists of tags associated with videos. The five collection data types in Cassandra are: set, list, map, tuple, and UDT. When defining a collection, users need to provide a data type for its elements. CQL provides convenient syntax for inserting, updating, or deleting items in collections. UDTs offer flexibility by allowing multiple data fields with names and types to be attached to a single column. An example is provided where an address type is created and used across multiple Cassandra tables. The frozen keyword is required when using UDTs inside of collections, forcing Cassandra to treat the address as a single value.

DataStax Enterprise (DSE) is built on Apache Cassandra, an open source distributed database that powers applications for major brands requiring high availability, zero downtime, and massive data volume scaling. DSE provides a highly available, fault-tolerant, and scalable data management platform with tunable consistency. Companies like Capital One, Cisco, Comcast, Condé Nast, Delta Airlines, eBay, Macy's, McDonald's, Safeway, and Sony rely on DSE for their business-critical applications. Developing on Cassandra is in high demand, with over 40,000 job listings seeking engineers with Cassandra and NoSQL skills. DataStax Academy offers courses to learn how to develop, administer, or architect with DataStax and Cassandra, as well as specializing in graph, search, or analytics. The benefits of shifting from relational databases to Cassandra include a more elastically scalable and resilient data platform that minimizes downtime, transforming data, and modeling complexity.

On October 30th, a new version of the DataStax C# drivers was released, including the latest metrics feature. Metrics are disabled by default and can be enabled by providing a metrics provider implementation. The blog post demonstrates how to run a simple console application that exports driver metrics to Graphite using an extension based on App.Metrics. It also provides instructions for setting up a Grafana dashboard to query the driver metrics on Graphite. To use this feature, users need to install the DataStax C# drivers and the necessary packages. The post includes code snippets and examples of how to configure the application and connect it to Cassandra.

DataStax Accelerate is a conference where the Cassandra community discusses trends, strategies, and technologies in modern application development and data management. This year's premier Apache Cassandra conference will be held in San Diego and London. The call for papers (CFP) is open, seeking members of the community to share their experience and expertise with open source Cassandra, DataStax, and related technologies such as Kubernetes, Kafka, graph, and analytics. Successful submissions often tell a story, clearly define the audience, include sufficient detail, have a well-defined topic, and avoid being a sales pitch. The CFP closes on January 22, 2020.

This guide provides best practices for loading data using the DseGraphFrame package, which offers a Spark API for bulk operations and analytics on DataStax Graph. The package supports reading DataStax Graph data into a GraphFrame and writing GraphFrames from any format supported by Spark into DataStax Graph. Key points include handling null values during updates, managing caching levels, indexing with Materialized Views, updating vertices and edges, and tuning parameters for improved write performance during bulk loading. Additionally, users can specify which host a DseGraphFrame object should connect with to read graph contents from one cluster and write to another.

DataStax has released a free Beta for its Apache Cassandra as a Service offering, Apollo, which is the first service on their cloud platform, Constellation. During the Beta period, users can create clusters in both Amazon Web Services and Google Cloud Platform. The billing process will differ between the two platforms; AWS clusters will be billed through DataStax, while GCP clusters will be billed through the user's GCP account. Documentation for running clusters during the Beta is available on a temporary site. Apollo includes access via DataStax Studio, which allows users to configure schema, load sample data, and more. The author of this blog post successfully ported their Reservation Service microservice to work with Apollo by externalizing connection logic from the service code and updating schema management code. They recommend three key steps for porting an application to run on Apollo: configuring the DataStax Java Driver, managing keyspaces, and handling schema changes. The author is looking forward to exploring more of Apollo's features during the Beta period and encourages feedback from users.