March 2019 Summaries

Apache Cassandra is a NoSQL database that has been adopted by leading organizations such as Instagram, GoDaddy, and Netflix due to its ability to meet the demands of modern applications which have surpassed the capabilities of traditional relational databases. Originally developed by Facebook engineers in 2007 to power the social network's inbox search feature using large datasets across multiple servers, Cassandra was released as an open source project in July 2008 and became an Apache Incubator project in January 2009. Today, major enterprises worldwide use it for building and deploying powerful applications in hybrid cloud environments. DataStax has been the driving force behind this open-source project, contributing to its growth and sophistication over the years. Cassandra's masterless architecture supports linear scalability for read and write operations, making it highly scalable. In 2014, Apple reported having a Cassandra instance with more than 75,000 nodes and storing over 10 petabytes of data. With the upcoming release of Cassandra 4.0, its adoption and capabilities are expected to continue growing in the coming years.

In this tutorial, we will continue to explore the DataStax Bulk Loader (DSBulk) by discussing how to load and unload data from a CSV file into Cassandra using DSBulk. We will cover various aspects of loading and unloading data, such as handling errors, dealing with missing or extra fields, ignoring leading whitespaces, specifying the insert statement, deleting data, and more. First, let's create a new table in our keyspace called "iris_with_id" that includes an additional column for the id: ``` CREATE TABLE dsbulkblog.iris_with_id ( id INT PRIMARY KEY, sepal_length FLOAT, sepal_width FLOAT, petal_length FLOAT, petal_width FLOAT, species TEXT ); ``` Now we can load data from a CSV file into this table using DSBulk. We will use the same iris.csv file as before: ``` $ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -header true -m "0=id,1=sepal_length,2=sepal_width,3=petal_length,4=petal_width,5=species" ``` We can see that DSBulk is trying to convert Iris-setosa into a number. This is because it believes that this column is the id column, which is an INT. If we look through the file we see the same error over and over again. We’ve clearly messed up the mapping. After 100 errors (that is the default, but it can be overridden by setting -maxErrors) DSBulk quits trying to load. Whenever DSBulk encounters an input line that it cannot parse, it will add that line to the mapping.bad file. This is the input line as it was seen by DSBulk on ingest. If a line or two got garbled or had different format than other lines, DSBulk will populate the mapping.bad file, log the error in mapping-errors.log, but keep going, until it reaches the maximum number of errors. This way, a few bad lines don’t mess up the whole load, and the user can address the few bad lines, either manually inserting them or even running DSBulk on the mapping.bad file with different arguments. For example, to load these bad lines we could load with: ``` $ dsbulk load -url /tmp/logs/LOAD_20190314-162539-255317/mapping.bad -k dsbulkblog -t iris_with_id -header false -m "0=sepal_length,1=sepal_width,2=petal_length,3=petal_width,4=species, 5=id" ``` Sometimes we are only loading some of the columns. When the table has more columns than the input, DSBulk will by default throw an error. Now, while it is necessary that all primary key columns be specified, it is allowable to leave other columns undefined or unset. For example, let’s say we didn’t have the sepal_length column defined. We could mimic this with awk, such as: ``` $ cat /tmp/dsbulkblog/iris.csv | awk -F, '{printf("%s,%s,%s,%s,%s\n", $2, $3, $4, $5, $6)}' | dsbulk load -k dsbulkblog -t iris_with_id ``` This will result in an error because we have not provided the sepal_length column: ``` Operation directory: /tmp/logs/LOAD_20190314-163121-694645 At least 1 record does not match the provided schema.mapping or schema.query. Please check that the connector configuration and the schema configuration are correct. Operation LOAD_20190314-163121-694645 aborted: Too many errors, the maximum allowed is 100. total | failed | rows/s | mb/s | kb/row | p50ms | p99ms | p999ms | batches 101 |    101 |  0 | 0.00 |  0.00 | 0.00 |  0.00 | 0.00 |   0.00 Rejected records can be found in the following file(s): mapping.bad Errors are detailed in the following file(s): mapping-errors.log Last processed positions can be found in positions.txt ``` To address this, we can add the... --schema.allowMissingFields: ``` $ cat /tmp/dsbulkblog/iris.csv | awk -F, '{printf("%s,%s,%s,%s,%s\n", $2, $3, $4, $5, $6)}' | dsbulk load -k dsbulkblog -t iris_with_id --schema.allowMissingFields true ``` Similarly, we could have a situation where the input file has extra columns. ``` $ cat /tmp/dsbulkblog/iris.csv | awk -F, '{printf("%s,%s,%s,%s,%s,%s,extra\n", $1, $2, $3, $4, $5, $6)}' | dsbulk load -k dsbulkblog -t iris_with_id ``` This will load just fine, and the extra column will be ignored. However, if we wish to be strict about the inputs, we could cause DSBulk to error if there are extra fields using: ``` $ cat /tmp/dsbulkblog/iris.csv | awk -F, '{printf("%s,%s,%s,%s,%s,%s,extra\n", $1, $2, $3, $4, $5, $6)}' | dsbulk load -k dsbulkblog -t iris_with_id --schema.allowExtraFields false ``` This will result in an error because we have not said how to map the extra column: ``` Operation directory: /tmp/logs/LOAD_20190314-163305-346514 At least 1 record does not match the provided schema.mapping or schema.query. Please check that the connector configuration and the schema configuration are correct. Operation LOAD_20190314-163305-346514 aborted: Too many errors, the maximum allowed is 100. total | failed | rows/s | mb/s | kb/row | p50ms | p99ms | p999ms | batches 102 |    101 |  0 | 0.00 |  0.00 | 0.00 |  0.00 | 0.00 |   0.00 Rejected records can be found in the following file(s): mapping.bad Errors are detailed in the following file(s): mapping-errors.log Last processed positions can be found in positions.txt ``` To address this, we can add the... --schema.allowExtraFields: ``` $ cat /tmp/dsbulkblog/iris.csv | awk -F, '{printf("%s,%s,%s,%s,%s,%s,extra\n", $1, $2, $3, $4, $5, $6)}' | dsbulk load -k dsbulkblog -t iris_with_id --schema.allowExtraFields true ``` Sometimes there is whitespace at the beginning of a text string. We sometimes want this whitespace and sometimes we do not. This is controlled by the --connector.csv.ignoreLeadingWhitespaces parameter, which defaults to false (whitespaces are retained). For example: ``` $ cat /tmp/dsbulkblog/iris.csv | sed "s/Iris/  Iris/g" | dsbulk load -k dsbulkblog -t iris_with_id ``` We can check that the leading whitespaces are retained via DSBulk’s unload command (which we will discuss in a later blog post): ``` $ dsbulk unload -query "SELECT species FROM dsbulkblog.iris_with_id" | head Operation directory: /tmp/logs/UNLOAD_20190321-144112-777452 total | failed | rows/s | mb/s | kb/row | p50ms | p99ms | p999ms 150 | 0 | 392 | 0.01 | 0.02 | 9.49 | 41.68 | 41.68 Operation UNLOAD_20190321-144112-777452 completed successfully in 0 seconds. species Iris-setosa Iris-setosa Iris-virginica Iris-versicolor Iris-virginica Iris-versicolor Iris-virginica Iris-virginica Iris-virginica ``` To strip the leading whitespaces, we can run the load command again with the --connector.csv.ignoreLeadingWhitespaces set to true: ``` $ cat /tmp/dsbulkblog/iris.csv | sed "s/Iris/ Iris/g" | dsbulk load -k dsbulkblog -t iris_with_id --connector.csv.ignoreLeadingWhitespaces true ``` Again, we can check this with the DSBulk unload command: ``` $ dsbulk unload -query "SELECT species FROM dsbulkblog.iris_with_id" | head Operation directory: /tmp/logs/UNLOAD_20190321-144510-244786 total | failed | rows/s | mb/s | kb/row | p50ms | p99ms | p999ms 150 |      0 | 416 | 0.01 |   0.01 | 8.16 | 36.96 |  36.96 Operation UNLOAD_20190321-144510-244786 completed successfully in 0 seconds. species Iris-setosa Iris-setosa Iris-virginica Iris-versicolor Iris-virginica Iris-versicolor Iris-virginica Iris-virginica Iris-virginica ``` We can also specify a mapping by providing the actual INSERT statement itself: ``` $ dsbulk load -url /tmp/dsbulkblog/iris.csv -query "INSERT INTO dsbulkblog.iris_with_id(id,petal_width,petal_length,sepal_width,sepal_length,species) VALUES (:id, :petal_width, :petal_length, :sepal_width, :sepal_length, 'some kind of iris')" ``` Notice that we do not need to specify the -k or -t parameters, since it is included in the query itself. We can also specify a mapping by providing the actual INSERT statement itself: ``` $ dsbulk load -url /tmp/dsbulkblog/iris_no_header.csv -query "INSERT INTO dsbulkblog.iris_with_id(petal_width,petal_length,sepal_width,sepal_length,species,id) VALUES (?,?,?,?,?,?)" -header false ``` Notice that I needed to move the id column to the end of the list. This is because the order here matters and must match the order in the input data. In our data, the id column is last. We can also specify a mapping to place constant values into a column. For example, let’s set the species to the same value for all entries: ``` $ dsbulk load -url /tmp/dsbulkblog/iris.csv -query "INSERT INTO dsbulkblog.iris_with_id(id,petal_width,petal_length,sepal_width,sepal_length,species) VALUES (:id, :petal_width, :petal_length, :sepal_width, :sepal_length, 'some kind of iris')" ``` It may seem counterintuitive, but we can delete data using the DSBulk load command. Instead of an INSERT statement, we can issue a DELETE statement. For example, let’s delete the rows in our table that correspond to the first 10 lines (11 if you include the header) of the iris.csv file: ``` $ head -11 /tmp/dsbulkblog/iris.csv | dsbulk load -query "DELETE FROM dsbulkblog.iris_with_id WHERE id=:id" Operation directory: /tmp/logs/LOAD_20190320-180959-025572 total | failed | rows/s | mb/s | kb/row | p50ms | p99ms | p999ms | batches 10 |      0 | 32 | 0.00 |   0.00 | 4.59 | 5.96 |   5.96 | 1.00 Operation LOAD_20190320-180959-025572 completed successfully in 0 seconds. ``` We can check that those rows have been deleted: ``` $ cqlsh -e "SELECT * FROM dsbulkblog.iris_with_id WHERE id IN (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15)" id | petal_length | petal_width | sepal_length | sepal_width | species ----+--------------+-------------+--------------+-------------+------------- 10 |          1.5 | 0.2 |          5.4 | 3.7 | Iris-setosa 11 |          1.6 | 0.2 |          4.8 | 3.4 | Iris-setosa 12 |          1.4 | 0.1 |          4.8 | 3 | Iris-setosa 13 |          1.1 | 0.1 |          4.3 | 3 | Iris-setosa 14 |          1.2 | 0.2 |          5.8 | 4 | Iris-setosa 15 |          1.5 | 0.4 |          5.7 | 4.4 | Iris-setosa (6 rows) Clearly, we could do all manner of other types of queries here. We could do counter updates, collection updates, and so on. To download the DataStax Bulk Loader click here. To learn additional elements for data loading, read Part 2 of the Bulk Loader series here.

Microservices are increasingly being adopted by modern enterprises due to their ability to enable agile development and scalability of applications. In contrast to monolithic applications, microservices allow for independent deployment and upgrading of individual software components. However, they also introduce challenges related to data management and consistency across multiple services. To build high-performance applications with microservices, it is crucial to consider data management implications from the outset and ensure seamless data flow during periods of high traffic. By doing so, organizations can create transformative tools that enhance employee productivity and user experiences.

The Java driver team has released two new major versions, OSS driver 4.0.0 and DSE driver 2.0.0, which address longstanding issues with the previous 3.x line through a major redesign of the architecture. These releases include many improvements and new features, as well as additional features for enterprise customer deployments in the DSE driver. The object mapper is not yet ported but will be released with OSS 4.1.0 / DSE 2.1.0. The drivers are available from Maven Central or can be downloaded from their servers.

Open source software like Apache Cassandra® has become popular among enterprises seeking efficient scaling and cost savings. However, adopting open source technology can come with challenges such as lack of skilled developers, rising maintenance costs, and insufficient internal expertise. To address these issues, DataStax offers a production-ready implementation of Apache Cassandra called DataStax Distribution of Apache Cassandra. This solution provides compatibility with open source Cassandra while offering support, educational resources, and tools to help organizations effectively utilize the NoSQL database. By using this distribution, companies can unlock the full potential of Cassandra without incurring additional expenses or administrative complexities.

This article discusses the importance of metrics and visibility in distributed systems, specifically focusing on DSE Analytics. It highlights how DataStax Enterprise (DSE) Metrics Collector simplifies exporting metrics to monitoring solutions like Prometheus and Grafana. The integration of Spark Cassandra Connector monitoring is left for a second post. The article provides a bash script that sets up Prometheus & Grafana, hooks up spark metrics, and configures the DSE collectd directory with the necessary plugin. It also explains how to enable and configure the collectd-spark plugin and write_prometheus plugin. Finally, it demonstrates how to visualize the Spark data in both Prometheus and Grafana.

DataStax Accelerate, the premier Apache Cassandra™ conference, will take place on May 21-23, 2019 in Washington, D.C., where winners of the inaugural Stax Awards will be announced. The awards are given to DataStax customers who actively use DataStax Enterprise, DataStax Distribution of Apache Cassandra, and/or DataStax Managed Cloud. Companies can apply by April 30, 2019. Winners in three distinct categories for partners will also be announced on May 22 and May 23. The event will feature examples of customers using DataStax and Apache Cassandra to deliver exceptional services and experiences.

Cedrick Lunven presents a talk on creating API's for databases, specifically focusing on distributed columnar stores. The discussion includes conceptual data models and the shift from relational to distributed systems. These topics are relevant to the Killrvideo reference application as well.

This article provides a step-by-step guide on how to set up a development environment using Apache Cassandra™ and DataStax Enterprise (DSE) on Windows with the help of Docker. It covers setting up a DSE server, working with CQL, monitoring DSE with OpsCenter, and utilizing DataStax Studio for development purposes. The article also highlights some limitations in containerized environments and provides links to further resources for learning and using these tools effectively.

The text discusses three common pitfalls that teams encounter while getting started with graph technology. These pitfalls include overlooking the branching factor of a graph's schema, creating supernodes due to poor data modeling, and misusing graph technology for non-graph problems. To avoid these issues, it is advised to understand the branching factor, monitor and mitigate potential supernode problems, and use appropriate technologies for specific tasks. The text also highlights the role of DataStax in helping customers build large production applications on graph databases and their commitment to sharing lessons learned from real-world experiences with graph technology.

This tutorial guides users through the installation of DataStax Enterprise (DSE) using Lifecycle Manager (LCM). It begins with configuring security rules in AWS, creating virtual machines on EC2 for OpsCenter and distributed database nodes, and then proceeds to install Python, Java, and OpsCenter. The process involves adding SSH credentials, a config profile, repository, cluster, datacenter, and nodes. Finally, the LCM downloads and installs DSE software on all three nodes, creating a single DSE cluster ready for use.

Many companies are adopting microservices for both re-architecting existing applications and starting brand-new projects. Microservices break large applications into smaller pieces, offering benefits such as scalability, resilience, and agility. However, they also introduce challenges like data distribution and consistency management. It is crucial to understand access patterns and use cases before designing an effective microservices architecture. Martin Fowler suggests that microservices only make sense at a certain scale and complexity, emphasizing the importance of understanding requirements before applying them.

Snitches are a feature in the Cassandra database that determine node placement within racks and datacenters. They provide information about the network topology of the system, enabling efficient routing of requests and distribution of replicas by grouping machines accordingly. All nodes within a cluster must use the same snitch for consistent distribution logic. Key feature options include determining how snitches determine node placement. Understanding snitches is crucial in managing distributed databases like Cassandra and DataStax Enterprise, along with other concepts such as consistent hashing.

Graph databases are becoming increasingly popular due to their scalability, performance, and agility. They enable organizations to uncover complex relationships between disparate data sets, which can be used to improve operations and boost the bottom line. Three foundational use cases for graph databases include building a Customer 360 application to understand customer behavior across channels, delivering real-time personalized recommendations based on customer behaviors, and using graph data as a fraud detection mechanism by comparing real-time transactions to historical behavior. By leveraging relationships within the data, organizations can save money and improve their overall performance.

Apache Cassandra uses the terms "datacenter" and "racks" to describe its architectural elements, which can be abstracted from their physical counterparts. A datacenter in Cassandra is a group of nodes within a cluster for replication purposes, while racks are groups of servers that prevent redundant storage of replicas. The use of snitches helps determine the placement of data within racks and datacenters, ensuring efficient organization and performance.

In this text, the author discusses their experience building a Python application for KillrVideo microservice tier. They delve into advertising service endpoints in etcd, describing it as a distributed reliable key-value store used to share the location of microservices and database connection information. The author explains how they use etcd in both desktop and cloud deployments of KillrVideo, using Docker and a utility called registrator. They also discuss their custom utility killrvideo-dse-config for registering services provided by Apache Cassandra (DataStax Enterprise) cluster into etcd. Furthermore, the author shares implementation details on how to use etcd in Python application code, including connecting to DSE and advertising service endpoints. Lastly, they mention future improvements and alternatives to using etcd, such as relying on DNS name resolution or a service mesh.