August 2020 Summaries

Couchbase has announced the general availability of the Couchbase Connector 4.0 for Apache Kafka, which facilitates the creation of robust, real-time data pipelines by integrating different datastores, applications, and services. Couchbase Server, an open-source, distributed JSON document database, and Apache Kafka, a distributed streaming platform, are combined by this connector to enable efficient data streaming, filtering, and transformation. Version 4.0 introduces significant enhancements, such as improved durability options, standardized configuration properties, even workload distribution, and a more flexible API for extensions, while also being packaged as a Confluent Hub component for easy installation. Additionally, the update supports upcoming features like Scopes and Collections in Couchbase Server 7.0, and provides a migration guide with tools to assist users transitioning from previous versions, highlighting changes in Source and Sink configurations and Java API updates.

Couchbase has introduced a new "External Datasets" feature in its latest Server 6.6 release, allowing users to access and analyze data stored in Amazon Web Services (AWS) Simple Storage Service (S3) in real-time. This feature enables the integration of S3-resident data with existing Couchbase data for analytics, meeting the needs of customers who store large volumes of historical data and business data on S3 for cost efficiency. By setting up an S3 link, creating an external dataset, and using SQL++ for queries, users can dynamically combine and analyze data from both Couchbase and S3 without needing extensive ETL processes. The feature supports various file formats like JSON and CSV, allowing for flexible data enrichment and dynamic access through massively parallel processing, ultimately facilitating faster and more comprehensive data analysis and decision-making.

Couchbase Flex Index is a new feature that allows N1QL queries to leverage FTS indexes without modifying the query syntax, providing flexibility and versatility for search capabilities. It enables applications to access both exact search services from a single API using standard N1QL predicates, simplifying application development and delegating more of the search processing to the back-end services. Flex Index can solve many challenges found in applications that provide searches, such as handling varied predicate sets, logical operators, hierarchical document elements, and combining SQL aggregation and JOIN with FTS capabilities. It also allows developers to use N1QL predicate syntax over FTS syntax, making it easier to integrate search capabilities into existing applications. The Flex Index feature can be used retrospectively on existing applications by adding the `use_fts` parameter to query API calls, and its benefits include reduced index size compared to GSI indexes, especially when array elements are involved. Query performance optimization is available only with GSI, but Flex Index provides pagination and JOIN capabilities, making it a powerful tool for building search-enabled applications.

In Couchbase Server 6.6, Flex Indexes offer a flexible schema that allows developers to create indexes without having to define the structure of their data in advance. This enables queries with complex predicates involving multiple fields, reducing the need for manual index management and improving query performance. By using Flex Indexes, developers can avoid the overhead of maintaining multiple GSI indexes and take advantage of advanced features such as FTS (Full-Text Search) indexing, which provides better performance for queries that involve full-text search or complex predicates. The process of setting up a Flex Index involves specifying the index name, bucket, JSON type, and document type, and providing hints to guide the query engine in choosing the most suitable index. With Flex Indexing, developers can write more flexible and efficient queries, reducing the risk of performance issues due to incorrect indexing or missing indexes. By leveraging Flex Indexes, developers can improve their overall development experience with Couchbase Server 6.6.

Couchbase Server 6.6 offers several enhancements that improve developer productivity, simplify cloud deployments, and enable operational analytics upon globally distributed data. The new features include Remote Links for real-time operational analytics, Flex Indexing to leverage FTS via N1QL queries, Index Advisor to create optimum indexes, TTL Support in Query to set document expiration times, Search within a Polygon to specify geospatial search queries as polygons, and New Timer Functionality with enhanced eventing timers. Additionally, Bucket-Level Durability Setting allows specifying the durability level for writes at the bucket level, Backup to S3 compatible object stores enable direct backup to cloud-based object stores, and Simplified Management of the Platform includes non-root installation and upgrade, import data via UI, Info Command, and Backup Examine (Developer Preview).

Couchbase 6.6 introduces a feature that allows users to import documents via the Couchbase Admin Web Console, offering a simpler alternative for small datasets compared to the more robust cbimport command-line tool. This web-based feature enables users to upload data in various formats, such as JSON List, JSON Lines, CSV, and TSV, directly into Couchbase buckets, with options to set document keys using UUIDs or field values. The blog post provides a practical demonstration of importing a small JSON dataset, highlighting the ease of use and potential issues like duplicate IDs when using the Value of Field option. It underscores the distinctions between different file formats, particularly in handling JSON data structure and null values, which can affect the integrity of imported data.

Remote links enable real-time operational analytics to obtain and analyze data from multiple Couchbase data clusters and datacenters in a separate cluster dedicated to the Analytics Service, allowing for the ingestion of data from the Data Service into an Analytics cluster. This feature enables customers to unify data from various operational applications into a centralized analytics cluster, reducing the total cost of ownership, improving resource utilization, and enabling hybrid transactional/analytical processing. With remote links, users can bring more data together in a single place, gather more insights, and perform correlation-style analyses across different datasets drawn from different clusters, saving time and processing power.

Couchbase 6.6 introduces significant enhancements to its Eventing Service, offering improved functionality such as the ability to cancel Eventing Timers using the cancelTimer() function or by creating a new timer with the same reference identifier. The update supports recurring timers, allowing for complex, repetitive logic and enabling timers to be scheduled well into the future without performance drawbacks. The OnDelete Handler now distinguishes between document deletions and expirations, enhancing logic execution based on document removal type. Additionally, key Eventing statistics are integrated with each function's lifecycle controls in the UI, facilitating easier diagnosis and implementation of robust business logic. These improvements simplify the development process and optimize the performance of Eventing Functions, making it easier to implement sophisticated business logic with fewer resources.

Couchbase 6.6 has introduced the ability to back up its document database directly to AWS S3 or any S3-compatible object store using the cbbackupmgr utility, allowing users to specify an S3 bucket as a backup destination. Object storage, characterized by flat storage systems and unique identifiers for accessing data, is becoming a standard for cloud-based data storage, offering advantages such as RESTful API access, scalability, and cost-effective, limitless storage capacity. These attributes make object storage ideal for various deployment types, including cloud-native and hybrid cloud environments, and beneficial for tasks like disaster recovery and big data analytics. AWS S3 is particularly noted for its role in disaster recovery and file sharing, while other alternatives like AWS Glacier and Nutanix Objects offer different performance and cost options. The transition to containerized workloads is also driving new approaches to file storage and access, reducing previous challenges associated with mounting filesystems.

The Couchbase Ruby SDK has been reintroduced due to popular demand, offering a simplified and high-performance API for connecting to Couchbase clusters from Ruby. This SDK supports features like Scopes and Collections, and is compatible with any MRI Ruby version from 2.5.0 and Couchbase server versions from 6.0.0. Installation is straightforward, with the beta version of the SDK available through a simple gem command, and it supports precompiled extensions for environments lacking a C/C++ compiler. The SDK enables easy connection to a Couchbase Server cluster and provides access to Buckets, Scopes, and Collections. It includes enhanced features for key-value operations, sub-document operations, and querying with the N1QL query language, as well as analytics capabilities for running complex ad-hoc queries. The SDK aims to facilitate the development of applications with improved efficiency and ease, inviting users to explore its capabilities and engage with the developer community for support and feedback.

The article provides a detailed guide on how to back up Couchbase Community and Enterprise editions on Ubuntu, emphasizing that the scripts and techniques described are not production-ready and may require customization based on specific environments. It advises using an external machine within the same network for backups to avoid performance issues and suggests cost-saving measures like scheduling cloud instances for backup tasks. The importance of a well-defined retention policy and regular recovery testing is highlighted, noting that backups depend on the last full backup and that testing helps familiarize users with restoration tools. For Couchbase Community, the article explains using cbbackup and cbrestore tools with a simple strategy of weekly full backups and daily incremental backups, while for Couchbase Enterprise, it introduces the cbbackupmgr tool with features like backup compaction and merging. Scripts are provided for automating these tasks using CronTab, and the article concludes with instructions on testing cron jobs using Cronitor to ensure successful execution.

Full-Text Search (FTS) in Couchbase is facilitated by the Bleve engine, an open-source text indexing and search library written in Go, which enables the examination and indexing of textual content within JSON documents. Couchbase's FTS engine supports a distributed system architecture that allows data partitioning across multiple nodes in a cluster, enhancing its capability to handle large-scale searches by scattering requests and gathering responses from various nodes. The engine uses an inverted index to link tokens generated from text to documents, improving search query efficiency. Key components of the text analysis process include tokenizers, character filters, and token filters, which work together to break down and refine raw text into tokens suitable for indexing. Users can configure custom analyzers by selecting and ordering these components to tailor the indexing process to specific needs, and a text analysis playground is available for testing stock and custom analyzers. Couchbase also provides guidelines on best practices for effectively utilizing its FTS capabilities.

The difference between relational databases and JSON databases lies in their ability to handle arrays. JSON databases, such as Couchbase and MongoDB, offer flexibility in terms of array type, size, depth, and elements, making them suitable for operational databases. However, this flexibility comes with limitations, including the inability to index multiple array keys simultaneously and the need for separate index entries for each element. To overcome these limitations, JSON databases use inverted indexes, which are ideal for indexing and searching array values, especially when they have duplicates. The upcoming Couchbase 6.6 release will support using FTS indexes for processing complex array predicates, improving the TCO of array handling and enabling developers to use arrays as needed without limitations.