May 2022 Summaries

In Part 2 of the Apache Cassandra series, the focus is on building advanced data models for successful applications using this distributed NoSQL database known for its big data scale and disaster tolerance capabilities. Key components of Cassandra's data structure include keyspaces, tables, partition keys, clustering columns, and primary keys. Data modeling in Cassandra involves understanding partitions, designing efficient data models, and utilizing Cassandra Query Language (CQL). The four objectives of the Cassandra data modeling methodology are to ensure consistency, availability, isolation, and partition tolerance. This series will continue with discussions on benchmarking your database, Storage-Attached Indexes, migrating SQL applications to NoSQL, and practical exercises.

Deepak Anand, Head of Field Engineering for Americas at DataStax, shares his journey from childhood curiosity about computers to his role in helping enterprises leverage real-time data with technology. Born and raised in India, he moved to the U.S. in 2013 and embraced American culture while maintaining his roots. Anand is excited about the potential of DataStax products to help businesses realize the value of real-time data and build high-growth applications. He emphasizes the importance of problem ownership and understanding perspectives when facing challenges, both personally and professionally.

DataStax Astra DB is a cloud-based Apache Cassandra service designed for developers to build massively scalable data applications without operational overhead or downtime. It offers five key benefits: easy deployment, compatibility with popular programming languages and APIs, pay-as-you-go pricing model, efficient scaling, and robust security features. Astra DB helps developers create faster and better apps while reducing their overall TCO.

The DataStax Bulk Loader (dsbulk) is a command line tool for loading and unloading data from Apache Cassandra® and Astra DB. It helps to load, unload, count data from various databases including DataStax Astra cloud databases, DataStax Enterprise 4.7 and later databases, and open source Apache Cassandra® 2.1 and later databases. The dsbulk tool can be easily installed on a virtual machine (VM) in the same region as your database to decrease latency and increase throughput. It also works well with Astra DB by passing a Secure Connect Bundle, client id, and client secret. Performance tuning is crucial for optimizing bulk data loading process, which can be controlled using flags like --maxConcurrentQueries, --dsbulk.executor.maxPerSecond, and --dsbulk.executor.maxInFlight. Other tips include handling errors, dealing with rate limits, and onboarding engineers for additional help.

KubeCon + CloudNativeCon Europe 2022 concluded with the cloud native community showing no signs of slowing down. The event highlighted growing projects, use cases, and a maturing conversation within the industry. Key takeaways included updates to the Cloud Native Maturity Model 2.0, which now includes Business Outcomes as an acknowledgement that organizations don't adopt cloud native technology solely for technical reasons. Additionally, there was a focus on improving security in Kubernetes environments with the Secure Software Factory Reference Architecture and increased adoption of self-managed Kubernetes clusters by users. The next North American event is set to take place in Detroit in October 2022.

The difference between a platform and a stack lies in their approach to data architectures, freedom of choice, and flexibility. Platforms provide developers with all necessary features but limit their ability to choose technologies or mix and match components. Stacks, on the other hand, consist of best-of-breed technologies that can be flexibly combined to solve specific problems. An open stack like DataStax's provides a cloud-native architecture for real-time applications with well-defined boundaries for each component, ensuring scalability and robustness. The key advantage of an open stack is the freedom it offers users in terms of choice and integration capabilities.

Feast, an open-source feature store for machine learning (ML), has gained popularity due to its flexibility and ease of use. It enables efficient scaling of ML practices in production by addressing common data engineering challenges such as managing relationships between offline and online stores. The feast-cassandra plugin allows developers to configure DataStax Astra DB or any Apache Cassandra® cluster as the online data store for Feast, offering advantages like handling big data and leveraging resilient architecture for predictions. With its Python API and compatibility with various data sources and online data stores, Feast serves as a valuable component of modern AI/ML data stacks.

Cloud-native methodologies involve understanding the role of containers and orchestration tools in automating applications, using APIs effectively, and managing data in dynamic application infrastructure. Apache Cassandra is a NoSQL database built for cloud data and is becoming popular among developers for cloud native applications. Managing distributed databases like Cassandra requires an understanding of Brewer's Theorem, which covers Consistency, Availability, and Partition Tolerance (CAP). Kubernetes has become the de-facto choice for managing container instances in microservices designs. Running Kubernetes together with Apache Cassandra can be achieved using a Cassandra Operator within your Kubernetes cluster. Combining Cassandra and Kubernetes makes it easier to scale out applications, taking advantage of distributed compute and data capabilities.

The text discusses the challenges faced in migrating workloads to the cloud due to issues with data persistence and movement. It highlights the increasing interest in data infrastructure designed for maximum advantage of cloud computing benefits, such as scalability, elasticity, resiliency, observability, and automation. A key example is K8ssandra, which packages Apache Cassandra® and supporting tools into a production-ready Kubernetes deployment. The text explores the concept of cloud-native databases running on Kubernetes and how it can be achieved by adopting key elements of the Kubernetes design paradigm. It delves into various principles like resource provisioning, separation of control and data planes, adoption of custom resources and control loops, observability through logging, metrics, and tracing, secure networking, secrets management, and declarative approach to managing resources. The text also mentions the Data on Kubernetes Community, which has hosted numerous meetups and is co-sponsored by DataStax. It concludes with an invitation to join upcoming events like DoK Day Europe 2022 and Kubecon/CloudNativeCon Europe.

Apache Cassandra is a distributed NoSQL database used by many Fortune 100 companies, including Apple, Facebook, and Netflix. It was designed to handle massive volumes of data with high speed requirements and various types of data relations. Unlike traditional relational databases, Cassandra has a leader-less (peer-to-peer) architecture that distributes data across multiple nodes within clusters, ensuring 100% uptime. The CAP theorem states that a distributed database system can only guarantee two out of three characteristics: Consistency, Availability, and Partition Tolerance. Cassandra is usually described as an "AP" system, meaning it prioritizes data availability over consistency. However, users can configure the level of consistency according to their use case. Key-based partitioning in Cassandra makes scaling easier by splitting data into chunks that are distributed across multiple servers. Data architects need to carefully design partitions for efficient query performance. With its powerful architecture and scalability features, Apache Cassandra is an essential tool for handling large volumes of fast-moving data in a reliable and scalable manner.

Running stateful workloads like Cassandra on Kubernetes was once considered difficult, but recent innovations have made it possible to create a platform for modern cloud-native applications with stateful requirements. Key resources such as StatefulSets, PersistentVolumes, and StorageClasses provide the building blocks needed to host stateful applications within Kubernetes. The development of the Cass Operator and K8ssandra has further improved scalability, reliability, and ease of management for these applications. As a result, Cassandra can now be considered the default data tier for building and running powerful, resilient, truly cloud-native data apps on Kubernetes.

Einat Orr, CEO and Co-founder of Treeverse, experienced a significant data loss incident while working as the CTO at SimilarWeb. This led her to co-found Treeverse, which developed LakeFS, an open source project that provides versioned data lake over object storage. The platform offers Git-like operations such as version control, rollback, and debugging, enabling risk-free experimentation with data. By allowing users to revert changes quickly and efficiently, LakeFS removes barriers to innovation and progress in the field of data science and engineering.

K8ssandra is an operational data platform designed to automate daily operations of a cloud-native platform, particularly those involving persistent data. It combines Apache Cassandra's power of scale and high availability with Kubernetes' capability to manage containers. Key features include automated backup creation, access to data, simple monitoring, and the ability to perform basic database operations through its unified API gateway, Stargate. Additionally, K8ssandra supports automatic repair operations on Cassandra using Reaper, which helps maintain consistent and stable nodes by combating entropy.

Software multi-tenancy refers to a single instance of software serving multiple tenants, with usage logically divided but physically shared on the same infrastructure environment. Compared to single-tenancy systems, multi-tenancy systems offer benefits such as simplified setup and maintenance, cost savings, and easy onboarding of new customers. Apache Pulsar is a true multi-tenancy system with built-in mechanisms supporting it, while Kafka has limited multi-tenancy capability. In Pulsar, tenant is a first-class citizen concept, and its security context relies on authentication, authorization, and data encryption features. Pulsar supports enabling multiple authentication providers at the same time, allowing each tenant to use their preferred method if needed. Kafka's multi-tenancy capability is limited from both security context and resource segregation perspectives, while Pulsar offers a robust multi-tenancy system with an abundance of built-in policies for data protection and fair resource utilization.