June 2020 Summaries

Apache Pulsar is an open-source, cloud-native pub-sub messaging and streaming platform that efficiently handles data movement for various use cases. Originally developed by Yahoo! and contributed to the Apache Software Foundation in 2016, it has been adopted by numerous companies due to its scalability and performance advantages over traditional messaging systems like RabbitMQ and Kafka. Pulsar's multi-layered architecture separates compute (brokers) from storage (BookKeeper), making it easy to integrate with cloud infrastructures and orchestration environments like Kubernetes. It supports various features such as schema registry, geo-replication, IO connectors, tiered storage, multiple subscription types, and advanced messaging capabilities. Pulsar is adept at handling high-volume pub-sub messaging as well as complex messaging patterns typical in a message queuing system. Its adoption rate has been growing rapidly since its open-source release, with an expanding feature set and community support.

Apache Pulsar is an open-source, cloud-native pub-sub messaging and streaming platform that has gained popularity since its open-sourcing in 2016. It combines the best features of traditional messaging systems like RabbitMQ with those of pub-sub systems like Kafka. Pulsar is highly scalable and can handle demanding data movement use cases, making it a perfect fit for distributed applications, serverless, and microservices architectures. Some key advantages over Kafka include better price and performance, as found by market research firm GigaOm. Apache Pulsar's multi-layer architecture allows new layers to be added, supporting features like tiered storage, geo-replication, and IO connectors. It also supports multiple subscription types and offers low, consistent latency at high throughput. Use cases for Pulsar include real-time messaging, event sourcing, delayed delivery, and real-time data processing with functions.

The DataStax Examples program has been launched, aiming to provide a collection of code samples and demo applications for the Cassandra community. This initiative is guest-edited by Rebecca Mills. One example highlighted in this newsletter is a REST backend built in Java using Spring Boot for use with the Getting Started with Astra UI. The application serves as the connection between the UI and an underlying Astra database, allowing users to visualize and interact with API resources. Additionally, there's an option to build a Docker image of this project for faster setup. DataStax is seeking community contributors to grow the program further. An 8-week Cassandra Workshop series will begin on July 1st, aimed at building proficiency in Apache Cassandra™ and preparing participants for free developer or administrator certification exams. Community highlights from Cassandra.Link are also shared.

The DataStax Examples program has been launched, aiming to provide a collection of code samples and demo applications for the Cassandra community. Each newsletter will feature a new example, with this week's highlight being an example of getting started with Apache Cassandra™ and Java using DataStax Astra. Contributions from the community are encouraged to expand the program. Additionally, an 8-week Cassandra Workshop series is starting on July 1st, offering participants high proficiency in Apache Cassandra™ and preparation for a free developer or administrator certification exam. Community highlights from Cassandra.Link include interesting discussions being followed by DataStax. Feedback and suggestions are welcome at [email protected] or @DataStaxDevs.

Apache Pulsar manages message life cycle using several mechanisms, including backlog quota and retention policy. In an ideal scenario with unlimited storage space, Pulsar retains unacknowledged messages in a backlog until they are acknowledged for deletion. However, real-world scenarios require additional measures to prevent the unlimited growth of message backlogs. To manage this, Pulsar uses time-to-live (TTL) parameters and subscription backlog quotas. TTL automatically moves a message to the acknowledged state after its expiration time, while the backlog quota enforces a hard limit on the logical size of backlogs in a topic. Pulsar offers three policies to prevent backlog overflow: producer_request_hold, consumer_backlog_eviction, and clear_backlog. The default broker option is producer_request_hold, which relies on consumers to drain the backlog. Additionally, Pulsar's retention policy helps retain acknowledged messages and messages on a topic with no subscription based on two configuration parameters: defaultRetentionTimeInMinutes and defaultRetentionTimeInSize in the broker.conf file. Understanding and managing message retention in Pulsar involves considering both backlog quota and retention policy configurations, as well as adjusting retention policies for any extra storage provided by features like tiered storage.

Apache Pulsar manages message life cycle using several mechanisms, including backlog quota and retention policy. In an ideal scenario with unlimited storage space, Pulsar retains unacknowledged messages in a backlog until they are acknowledged for deletion. However, real-world scenarios require additional measures to prevent the unlimited growth of message backlogs. Pulsar uses two mechanisms to manage backlogs: time-to-live (TTL) parameter for individual messages and subscription backlog quota for the backlog itself. TTL defines the amount of time a message is allowed to stay in the unacknowledged state, while the backlog quota enforces a hard limit on the logical size of the backlogs in a topic. To prevent backlog overflow, Pulsar offers three policies: interrupting message transmission, clearing existing messages from the backlog, and allowing slowest consumers to lose 10% of the oldest messages in the backlog. The default broker option is producer_request_hold, which relies on consumers to drain the backlog. Pulsar's retention policy tells it to retain acknowledged messages and messages on a topic with no subscription. It defines the limit to keep acknowledged events and marks deletion for messages over the limit. Retention policies can be specified at the namespace level, allowing teams using different namespaces to use different policies. To manage message retention in Pulsar, consider both backlog quota and retention policy configuration. Adjust retention policies accordingly when offloading closed ledgers to a lower-cost storage like AWS S3.

Apache Cassandra, a popular NoSQL database, is known for its ability to handle any workload with zero downtime and global scale. DataStax simplifies the deployment of Cassandra into various cloud platforms like AWS, Azure, and GCP through marketplace installations. This allows users to achieve faster business outcomes and ROI with trusted support options from DataStax Luna. The choice of leveraging open-source data platform capabilities across clouds offers agility, flexibility, and true data portability while avoiding vendor lock-in. Cassandra's architecture ensures 100% uptime for cloud applications by supporting multi-cloud strategies and infinite linear scale. DataStax Luna provides support options ranging from Gold to Bronze levels for open-source Cassandra users.

DataStax has introduced Vector, a tool designed to provide automated expert advice, continuous updates, hands-off management, and skills development for users of Apache Cassandra™ and DataStax Enterprise (DSE). Vector aims to help users identify issues before they become problems and turn each interaction into a learning experience. It is available as a cloud service or on-premise install and offers features such as automated expert advice, continuous updates, hands-off management, and skills development. The tool provides customized fixes for various issues related to Apache Cassandra and DSE.

DataStax Examples is a repository of code snippets, examples, and demo applications designed to help developers work more efficiently with Apache Cassandra™ and DataStax products. The collection includes open-source code available on GitHub, covering various tooling and frameworks for building better applications. Contributions are encouraged through the Example of the Week program, which rewards participants with publicity and a t-shirt if their example is selected. Examples currently exist for Java driver 3.x and 4.x, but there's also interest in expanding to Python, Node.js, and C# DataStax drivers. Developers are invited to share their demos and sample code with the community through a submission form or by emailing [email protected].

The joint report between DataStax and Arrikto reveals significant improvements in response times and transaction costs when using Amazon Web Services (AWS) with Arrikto Rok. Arrikto Rok is a revolutionary storage and data management solution for stateful applications on Kubernetes, offering enhanced local storage devices' data management capabilities without adding an additional layer of abstraction. The innovative approach results in 15x faster response times and a 22% transaction cost saving compared to the commonly used EBS model. Arrikto Rok also enables data management, versioning, and transport in a managed way, ensuring high performance for normal operations and fast recovery. Overall, using Arrikto Rok leads to substantial business benefits and technical advantages, allowing users to take a true cloud approach to containerized storage and data management.

The new default load balancing algorithm in DataStax Drivers aims to reduce long tail latency and improve overall response times by using a modified version of the Power of Two Random Choices algorithm. This algorithm selects two replicas at random and picks the one with fewer in-flight requests as coordinator, while also considering whether a node has not sent back any response in the last few hundred milliseconds. The new approach outperforms previous methods and is being rolled out across all client drivers.

The text discusses using DataStax Astra API and GraphQL to manage product data in a database. It explains how to generate an authorization token, run queries to check for existing products, insert new products, and confirm the addition of new products. The example provided is adding "Heavy Lift Arms II" to the Better Botz database. In the next post, the focus will be on creating a new table for orders that have shipped and further data modeling.