July 2019 Summaries

Building a Customer 360 solution on Couchbase involves a complex process of extracting, transforming, and loading (ETL) data from various source systems such as CRM, ERP, and even mainframes into Couchbase. The author emphasizes the importance of simplicity in this process, suggesting using Kafka streams to transfer data in JSON format directly into Couchbase, despite the inherent challenges of integrating disjointed data models from different systems. The solution includes using Couchbase's Eventing system to update and combine data in real time, creating a cohesive customer data model. Although Couchbase advocates for analytics without ETL, in this case, an ETL process is necessary, albeit potentially in a real-time context, depending on the data extraction schedule. The author argues that using Couchbase, in conjunction with existing infrastructure like Kafka, offers a simpler and more integrated approach than relying on multiple external ETL tools and systems, allowing for better performance and ease of use.

In modern business applications, data models optimized for low-latency queries often hinder the ability to analyze trends, traditionally requiring data movement and transformation that lead to issues like latency and increased costs. Couchbase addresses this by providing an Analytics service that facilitates real-time trend reporting on production data without needing complex transformations. The text illustrates this with a case study of a customer using Couchbase to analyze high-end customer activity linked to a corporate loyalty program, where the existing document model was designed for interactive applications rather than reporting. The solution involved using N1QL for Analytics to query and unnest various hierarchical elements within the data model, enabling efficient analysis of fields scattered across different document sections. This method demonstrates the power of N1QL for handling complex data structures and providing swift insights without altering the original data architecture.

Couchbase Mobile provides a robust embedded NoSQL database, Couchbase Lite, which can be used within iOS, Android, and Xamarin apps. The Couchbase Mobile stack also includes Sync Gateway, enabling secure data synchronization across clients. A developer created an app called CouchDraw to visualize the power of Couchbase Mobile by creating a collaborative drawing experience using real-time touch effects. The app utilizes Skia, a cross-platform 2D graphics library, to support drawing and store data in a document-based database, which is integrated with Couchbase Lite and Sync Gateway for seamless synchronization. The developer invites the mobile and Couchbase communities to expand on CouchDraw's functionality with new features such as color selection, line thickness adjustment, and erasing capabilities.

The text discusses the concept of Customer 360, highlighting its importance for businesses seeking a comprehensive view of customer data, which is often fragmented across various systems. It emphasizes the complexity involved in achieving this goal, despite marketing claims that suggest simplicity. The text goes on to detail the integration challenges, particularly the need for thoughtful data combination and formatting. It outlines the role of microservices and the importance of mobile integration, facilitated by Couchbase’s mobile stack, which simplifies data synchronization. The front-end development is noted as a critical, albeit challenging, component where Couchbase offers tools and training but not turnkey solutions. The text advocates for Couchbase by highlighting its scalability, high availability, ease of transitioning from SQL to its N1QL query language, and additional features like Full Text Search and Analytics. It concludes by asserting Couchbase's reliability in maintaining data integrity even when servers fail, ultimately promoting a cascade of satisfaction from executives to developers.

Cormac Hogan, a Director and Chief Technologist at VMware, explores the Couchbase Autonomous Operator for Kubernetes, which is recommended over the traditional StatefulSet method for managing Couchbase databases. He details the deployment of Couchbase within its own Kubernetes namespace on a VMware infrastructure, emphasizing the modifications needed to accommodate this setup, such as creating a new namespace, configuring service accounts, and adjusting YAML files. Hogan also highlights the operator's functionalities, including downloading Docker images, creating custom resource definitions, and monitoring events. He provides insights into handling persistent volumes and overcoming issues with the bundled load-generating tool, "pillowfight," to successfully deploy and manage Couchbase clusters. The deployment is part of a Kubernetes cluster running on VMware PKS, vSphere, and vSAN infrastructure, demonstrating the operator's effectiveness in a cloud-native environment.

The text provides an overview of integrating Prometheus and Grafana for monitoring Couchbase Sync Gateway deployments on Kubernetes, emphasizing the recent update in Sync Gateway v2.8, which now includes built-in Prometheus exporter support. This eliminates the need for a separate exporter, simplifying the monitoring process. The document outlines a step-by-step guide for setting up this monitoring system, explaining the roles of various components like the Sync Gateway Exporter, Prometheus Operator, and Grafana dashboards. The setup involves deploying the Sync Gateway on Kubernetes, configuring a service to expose it, and using the Prometheus Operator to automate the deployment of the monitoring stack. The integration enables detailed monitoring of Couchbase Mobile clusters and offers customizable dashboards and alerting rules, enhancing the ability to track and visualize system health and performance metrics efficiently.

The text outlines performance optimizations introduced in version 6.5.0 of Full-Text Search (FTS), primarily centered around the principle of "do less and do it less often." A significant change was transitioning the scatter-gather mechanism from HTTP/HTTP2 to gRPC, leveraging its streaming and request multiplexing capabilities, which resulted in a 2X throughput improvement for low-frequency term queries in a three-node cluster. The integration of new protobuf message types faced initial performance challenges due to legacy JSON message types, which were mitigated by optimizing the parsing process. Further enhancements included refining the numeric range query logic by implementing a more efficient Terms Dictionary lookup API, achieving a 77% increase in throughput during internal tests. Additionally, improvements in caching and reusability for prefix, fuzzy, and wildcard queries yielded throughput gains of 25%, 12%, and 9%, respectively, by reducing garbage creation and optimizing garbage collection cycles.

Migrating a relational database to Couchbase involves downloading the Couchbase server, using connectors or export-import tools, and considering key differences between RDBMS tables and Couchbase buckets, including document keys and schema transformation. The blog outlines strategies for transforming relational schema into Couchbase's NoSQL JSON format, such as direct mapping, denormalization, and adding child records as array fields. It emphasizes using Oracle's REST Data Services for data extraction and N1QL scripts for the transformation process, highlighting the importance of managing primary keys and document types. The document also notes the limitation of N1QL CURL() in handling large data volumes and suggests using Oracle's OFFSET and FETCH clauses for batch processing, while a comment advises using scripting languages for large migrations.

The text discusses the development of a User Profile service within the context of software architecture, particularly emphasizing its role in managing user-related data and its importance in microservices. It highlights the advantages of using Couchbase for this task due to its strong consistency, high throughput, caching, flexible data model, and scalability. The document introduces two tutorials aimed at building a User Profile system, covering both Java and .NET languages, with step-by-step instructions on setting up a project, creating repositories, querying data, and managing user events asynchronously. These tutorials are part of a growing open-source, searchable section and are designed to expand from a simple profile to more complex implementations. The text also provides contact information for further inquiries, inviting contributions and feedback through pull requests.

Couchbase's Full Text Search (FTS) team has been focusing on optimizing the performance of its search engine since the introduction of the scorch indexing format in version 6.0.0. In the upcoming 6.5.0 release, significant improvements are being made to geo and fuzzy queries. For geo queries, enhancements involve filtering mathematically generated geo terms to reduce unnecessary search iterations, resulting in up to a 6X reduction in latency. For fuzzy queries, the use of Finite State Transducers (FST) and the construction of a precomputed Levenshtein Automaton (LA) have led to faster and more efficient query processing, with the new implementation being up to 5X faster and 12X more memory-efficient. These updates demonstrate marked increases in latency reduction and throughput, preparing the way for further enhancements covered in future updates.