August 2019 Summaries

The text creatively explores the evolution and coexistence of SQL and NoSQL in the landscape of data management and analysis, using a poetic metaphor to illustrate this dynamic relationship. It portrays SQL as a divine light bringing order and understanding through operations like select, join, and project, offering joy and insight into data. However, it also acknowledges the rise of NoSQL, depicted as a disruptive force advocating for more scalable and flexible data handling techniques such as map-reduce, and ultimately suggests a harmonious integration of SQL with newer data paradigms through N1QL, a query language designed to combine the strengths of both worlds.

The International Data Corporation (IDC) has released a report based on a survey of 2,500 developers worldwide, highlighting the increasing influence and importance of developers in driving digital transformation. According to the report, developers are no longer just executing plans but are now regarded as visionaries and architects of digital transformation. The survey also found that developers' familiarity with technologies like containers and microservices is on the rise, and they increasingly want to use open source tools. Matt Ingenthron and Huw Owen from Couchbase emphasize the changing role of developers in driving innovation and digital transformation, citing the need for technology selection and the importance of adopting tech designed to support agile development needs with DevOps patterns in mind. The report validates the growing trend of developer influence, with developers being seen as a critical constituency in technology who are calling the shots in many organizations.

Couchbase's primary index is a sorted list of all document keys within a bucket, maintained asynchronously and supporting various scan consistencies. It allows full bucket scans when queries don't have filters or other indexes can be used, but this comes at a high cost due to the need to fetch and filter all documents. The primary index should not be used in production unless necessary, as it's slow and unnecessary document fetches waste memory and CPU resources. However, it can be useful for development, querying specific ranges of keys, or when the full document key is known. Named primary indexes with multiple replicas offer high availability and query load distribution benefits. The Index Advisor in Couchbase 6.5 analyzes N1QL statements or workloads to advise suitable secondary indexes but not primary indexes, as they're not recommended due to performance issues. When creating multiple "named" indexes of the same type and combination, it's essential to consider the cost tradeoff, as maintaining multiple index nodes can impact performance. There is no other way to retrieve a document ID without using indexes or views if the ID is unknown in advance, but you can use the `USE KEYS` clause to fetch documents directly without consulting the primary index.

The Advisor function in Couchbase Server 6.5 extends its capabilities from advising on a single query to managing index recommendations for multiple query workloads and supporting session handling. When applied to a workload, it analyzes queries, provides index recommendations, and outputs current, recommended, and recommended covering indexes. The session handling allows users to define specific queries, collect them over time, and asynchronously analyze them for index recommendations. Users can manage these sessions with various actions such as starting, stopping, or purging sessions. While the Advisor function offers a structured approach to optimizing query performance through index analysis, there is an ongoing effort to enhance its functionality and performance based on user feedback.

The text discusses the development and utilization of a Customer 360 solution using Couchbase technology, detailing the process of integrating data from various sources into a unified customer view and leveraging Couchbase Mobile for offline access by Account Executives. It highlights the potential to use the Couchbase Sync Gateway “changes” feed for real-time data updates and notifications, such as alerting a Product Manager about customer feedback. The text further explores the use of Couchbase’s Full-Text Search (FTS) and N1QL queries for comprehensive data analysis, allowing Product Managers to identify customer trends, such as color preferences for products, and to support strategic decision-making with data-driven insights. The author emphasizes the simplicity and efficiency of using Couchbase's tools for complex data tasks, comparing the experience to a superhero-like ability to navigate and analyze data effortlessly.

Couchbase 6.5 introduces support for window functions, as defined by the SQL:2003 standard, in its Analytics service, enhancing its capability for efficient parallel query processing. Window functions allow for more concise and readable SQL queries, eliminating the need for complex self-joins and subqueries, thus improving query performance. This blog post demonstrates the application of window functions by calculating a running total, showcasing their practical benefits over traditional methods such as INNER JOIN. It highlights how Couchbase Analytics can handle complex joins and aggregations that are typically challenging in NoSQL environments, differentiating it from other NoSQL vendors like Cassandra and MongoDB. The post also provides guidance on setting up a Couchbase Server 6.5 environment to experiment with these features, emphasizing that window functions are a well-established part of SQL and now extend to document databases with Couchbase's latest release.

Modern business applications require tools that ensure speed of development, infrastructure stability, and ability to scale. Container orchestration tools like Kubernetes abstract away physical machines it manages, providing pods and containers as ephemeral units that can self-heal in case of failure. Persistent Volumes enable data recovery quickly by remounting storage volumes to a newly elected Pod. Couchbase Autonomous Operator uses this technology to make management and orchestration of Couchbase Cluster seamless on the Kubernetes platform. The operator creates persistent volumes using Storage Classes offered by different cloud vendors, making it easy to use cloud-native storage solutions. In the event of pod or container failure, Kubernetes re-instantiates a new pod/container automatically and remounts the persistent volumes, ensuring fast recovery and maintaining SLA during infrastructure failure recovery.

The document provides a detailed guide on implementing custom authentication with Couchbase Mobile, focusing on how to securely manage and sync data from cloud to mobile devices. It introduces Couchbase Mobile's components, including Couchbase Lite, an embedded database with SQL and full-text search for JSON, and Sync Gateway, a secure web gateway supporting multiple authentication methods. The guide walks through using an App Server, implemented as a node.js application, to authenticate users via an external system like OpenLDAP and manage user sessions in Sync Gateway. It includes code examples for both the App Server and a mobile app using Couchbase Lite Android SDK, explaining how to authenticate users, create and manage user sessions, and handle replication using session IDs. Additionally, it outlines setting up an OpenLDAP server and testing the authentication process. The document concludes by encouraging readers to explore further training and tutorials available through Couchbase's online resources.

Couchbase Server 6.5 introduces a highly anticipated node-to-node encryption feature, enhancing security by encrypting data exchanged between nodes in a distributed data platform. Previously, encryption was limited to client-server interactions and XDCR traffic between data centers, leaving node communication vulnerable in the absence of complex network-wide security controls. With this new feature, Couchbase natively encrypts both control and user data traffic between nodes, offering flexibility in encryption levels to balance security and performance. While node-to-node encryption strengthens security, it complements existing measures such as client-server encryption and network-level isolation. Setting up this feature involves straightforward steps during the installation or upgrade of Couchbase Server, and regular certificate rotation is recommended for maintaining secure operations.

Couchbase Analytics, since its general availability in November 2018, has been utilized in various industries for rapid insights, such as by payment providers, e-tailers, airlines, and stadiums. The system was designed to allow the Analytics service to run queries even during operations like rebalancing or failover, catering to developer productivity and allowing DevOps teams to manage clusters independently. However, a common request from customers using Couchbase for business-critical applications is to avoid non-monotonic query results, where queries could return outdated data. To address this, the upcoming release introduces "scan consistency," ensuring that query results are consistent with the latest data state. Users can set preferences like "not_bounded" for immediate evaluation or "request_plus" for ensuring data state consistency before execution, with an option to specify query timeout using the "scan_wait" parameter. This enhancement aims to provide reliable insights without disruptions from cluster operations.

Couchbase has introduced a feature called 'Collections' in its 6.5 Developer Preview, aiming to enhance data organization and processing efficiency by grouping similar documents within a single bucket. Collections, akin to tables in relational databases, provide logical data containers that allow for better data management and finer access control granularity, which is beneficial for multi-tenant and microservices applications. This feature facilitates the separation of data types, improves indexing scalability, and simplifies query construction. Additionally, the Developer Preview supports operations on collections and scopes, though role-based access control at these levels will be available in future versions. While the current integration for Couchbase Mobile is limited to the Default Collection, future updates are expected to enable synchronization of documents within specific scopes or collections.

Couchbase utilizes the Jepsen testing framework to ensure the database's durability and consistency through rigorous testing, particularly with the enhanced durability levels introduced in the 6.5.0 release. Jepsen tests Couchbase by simulating operations amidst chaos, like network partitions and process failures, to verify that acknowledged writes are never lost and that a sequential consistency model is maintained. The setup involves implementing Jepsen's abstractions, such as Database, Client, and Nemesis, tailored specifically for Couchbase, to test various failure scenarios and cluster operations. Challenges in testing arise from the vast number of potential cluster configurations and the processing time required by Jepsen's linearizability checker, leading to strategies that focus on critical parameters and frequent testing through a structured test suite hierarchy. This approach has successfully identified bugs, increasing confidence in Couchbase's ability to safeguard data, though continuous testing is crucial as Jepsen might expose issues after numerous runs.

Couchbase Server 6.5 introduces support for distributed multi-document ACID transactions, which simplifies application development by allowing the database to manage transaction recovery semantics instead of the application tier. This new feature extends Couchbase's traditional single-document ACID capabilities to multiple documents across various nodes, maintaining atomicity, consistency, isolation, and durability (ACID) guarantees. The transactions are executed through Couchbase SDK APIs, offering robust error handling with built-in retries for failures in a distributed system. The implementation avoids the limitations of the traditional two-phase commit (2PC) protocol by using smart clients that track transaction states, eliminating the need for global coordination or sharding. This approach ensures high availability and scalability without performance penalties, allowing applications to selectively use ACID features alongside NoSQL's speed and flexibility. Couchbase's architecture enables partition-agnostic transactions, supporting modern applications with the combined benefits of NoSQL and traditional database transactional capabilities. Future enhancements include N1QL support for transactions and features aiding the porting of relational database procedures to Couchbase.

In Couchbase 6.5, collections and scopes provide a new way to organize documents within a bucket, similar to tables in relational databases. Collections are the lowest level of document organization and directly contain documents. They allow grouping documents more precisely than before, enabling queries against specific collections instead of just the entire bucket. Scopes contain collections and collections contain documents, with different use cases depending on the Couchbase cluster configuration. The current version provides a Developer Preview feature, allowing developers to start working with collections and scopes right away. N1QL queries cannot currently work against collections in 6.5, but this will change in 7.0, enabling queries that span multiple scopes. The design allows for separating large or frequently changing data from user documents, improving the scalability and maintainability of applications.

The new Eventing Service framework in Couchbase Server 6.5 enables writing custom functions to process data change events such as create, update and delete/expiry events. One of the key improvements is the removal of a previous limitation that prevented functions from mutating documents on their source bucket, which now allows for more complex use cases like enriching changed documents with new attributes or performing cascade updates or deletes of dependent documents. The Eventing framework also includes a `curl` function to interact with external systems, and the ability to pause and resume running functions, allowing for mid-flight code changes without disrupting ongoing operations. Additionally, an Eventing function can now identify modifications made in a document by comparing the original and updated versions.

To connect RapidMiner to Couchbase Analytics, users need to set up a Couchbase cluster with Data and Analytic services enabled, download and install the CData JDBC driver for Couchbase, and configure the license. They also need to create shadow datasets in Couchbase Analytics, initialize them, and test the connection. In RapidMiner, users can create a new connection, select the CData JDBC driver, and configure the connection string options. They can then load data into RapidMiner using the "Read Database" operator, store the results locally, and use other operators and processes to analyze the data further. With this setup, users can extend Couchbase Analytics with RapidMiner and explore their data for insights.

XDCR, a feature used for data replication in various scenarios such as high availability and disaster recovery, has been enhanced in Couchbase Server 6.5 to include advanced filtering capabilities. This update allows users to filter replication based on keys, values, and metadata through N1QL-like expressions, providing flexibility to meet specific business needs such as geo-fencing. The advanced filtering is categorized into expression-based filtering, allowing the use of various language constructs, and deletion filtering, which enables the exclusion of deletes or documents with TTL from replication. Users can test and validate filter expressions by specifying document IDs, and filters can be edited without interrupting replication. These enhancements improve resource utilization by allowing more precise control over what data is replicated, and they enable new use cases like creating hot and cold data clusters. Users with versions prior to 6.5 can only filter based on keys, while versions 6.5 and above offer more comprehensive filtering options.

Couchbase Server 6.5 introduces Distributed Multi-document ACID Transactions, which combine ACID guarantees with the scalability, high availability, performance, and flexibility that Couchbase is known for. These transactions are easy to implement using the Couchbase SDK and offer automatic commit or rollback, retries for transient errors, and configurable duration to resolve deadlocks. They ensure that only committed data is readable by consumers and use a combination of optimistic and pessimistic locking to manage conflicts. Transactions can span multiple documents across different nodes and databases in a Couchbase cluster, providing atomicity, consistency, isolation, and durability. The new synchronous replication protocol enhances durability by allowing tunable levels of data protection against failures, and these features are also applicable to Ephemeral buckets for caching use cases. Overall, Couchbase's innovative approach to ACID transactions in a distributed system expands potential applications for NoSQL databases.

XDCR in Couchbase Server 6.5 introduces a new feature allowing users to assign priorities to replication streams, with options of HIGH, MEDIUM, or LOW, to better manage resource allocation during data replication processes. This prioritization is especially beneficial when adding new clusters or managing workloads, as it enables customers to either expedite or delay replication resources based on their specific needs and criticality of business operations. The default behavior assigns the highest priority to ongoing replications to minimize their performance impact when new replications are initiated. The MEDIUM priority allows new replications to gradually catch up with existing ones, while the LOW priority is suitable for less time-sensitive scenarios, such as setting up hot and cold clusters. The new priority setting can be adjusted without disrupting existing replications and is accessible via the CLI or REST API, providing users with enhanced control over their resource management and potential cost savings in cloud deployments.

Index advisor is a feature introduced in Couchbase server 6.5, providing secondary index recommendation to help DBAs and developers optimize N1QL query performance. It follows design rules such as leading array index key for unnest, equality predicates, IN predicates, not less than/between/not greater than predicates, less than/greater than predicates, array predicates, derived join filter as leading key for left-hand-side keyspace, IS NOT NULL/MISSING/VALUED predicates, and functional predicates. Index advisor works in two ways: providing N1QL statement to advise indexes for a single query and advising on query workload and support session handling. It generates recommended indexes based on predicates in WHERE/ON clause, along with the corresponding recommendation rule that each index follows. The feature also supports covering indexes and array indexes for single keyspace queries, JOIN operations, ANY expressions, and UNNEST predicates. Index advisor uses a naming convention to generate unique and reasonable index names, ensuring they reflect the index construction and are different from user-defined indexes. Released as a developer preview feature, it is still in its initial stages and will be further enhanced to better meet customer requirements and contribute to query optimization.

Couchbase Server 6.5 introduces significant updates to its statistics display in the Web Console, enhancing their utility for performance monitoring and troubleshooting in large production systems. The new features include intelligent line charts with labeled axes that are adaptable to various screen sizes, customizable dashboards, and the ability to aggregate node data for more efficient analysis. Users can select specific time ranges, toggle data feeds, and create personalized scenarios using the chart builder, which allows for the combination of different stats, services, and nodes. Additionally, service-specific stats are now integrated inline with corresponding lists, and the design accommodates scalability for large collections of data, particularly in services like Indexing. These improvements aim to provide greater visualization and clarity, facilitating quicker resolution of issues and better overall system management.

Couchbase has announced the release of Couchbase Distributed ACID Transactions, specifically designed for Java developers to integrate with ease into their existing projects. This new API allows developers to implement transactions without requiring additional configuration or setup on their clusters, as it is available for the Community Edition and can be integrated by simply adding a library line to a gradle or Maven project. Although the initial release is Java-only, future releases for other programming languages are planned. Developers are encouraged to consult the accompanying documentation and video guide to get started and provide feedback through forums or comments.

In this blog, the focus is on the implementation of LDAP group capabilities in Couchbase Server 6.5, which enables the integration of existing organizational structures for managing group-based privileges through RBAC roles. LDAP, or Lightweight Directory Access Protocol, organizes users and groups hierarchically, allowing simplified management outside Couchbase. With the new feature, external LDAP groups can be mapped to Couchbase groups, thereby streamlining the assignment of roles and privileges. The blog provides a practical example using an online LDAP test server, illustrating how different LDAP groups like scientists, mathematicians, and chemists can be mapped to corresponding Couchbase groups such as Executives, Managers, and Employees, each with specific privileges. This integration facilitates easier and more efficient user management in Couchbase by leveraging existing LDAP directories.

The document discusses the implementation of a Cost-Based Optimizer (CBO) for N1QL, Couchbase's SQL-like query language for JSON data, introduced in Couchbase 6.5. Unlike traditional rule-based optimizers, the CBO leverages statistical data to optimize query execution by selecting the most efficient query plan based on cost estimation. This involves collecting and organizing statistics on JSON's flexible schema, including scalars, objects, and arrays, to handle the complexities of JSON data structures. The CBO aims to improve query performance and system efficiency by addressing challenges such as mixed data types and nested structures, ultimately reducing the total cost of ownership by minimizing the need for performance debugging. The document also highlights the challenges and methodologies involved in collecting statistics on JSON data and how these are used to enhance query optimization.

Autocomplete, or word completion, is a feature that predicts the remainder of a word as a user types, enhancing the search experience by minimizing the number of characters needed to obtain results. This concept can be implemented using Couchbase Full Text Search (FTS) through two main approaches: the Edge NGram approach and the Prefix-based approach. The Edge NGram method involves creating an FTS index using edge-ngram token filters, which tokenize text into sub-tokens to provide potential completions as users type. The Prefix-based approach uses a keyword analyzer to store field values as single tokens and relies on prefix queries to match text that begins with the user's input. Both methods aim to improve user navigation by providing relevant suggestions in real time, although the Prefix-based approach limits matches to the beginning of field values.

Couchbase offers a solution for building a Customer 360 platform that simplifies data integration from various sources through its offline mobile synchronization, eliminating the dependency on REST calls. This approach allows Account Executives to access crucial customer information seamlessly, even in areas with no connectivity, by using a small database on their phones that syncs data in the background via the Sync Gateway and Couchbase Lite. The system organizes data into channels and assigns them to users, ensuring that Account Executives have the necessary data ready for use, without waiting for queries to run, thereby maintaining simplicity and efficiency. Couchbase emphasizes keeping solutions straightforward to minimize maintenance challenges and enhance usability, leveraging features like Full Text Search, Analytics, Eventing, N1QL querying, and Key/Value data service with built-in caching to provide an integrated, out-of-the-box experience.

The article provides a detailed tutorial on configuring the Public Connectivity feature in Autonomous Operator 1.2.0 for safely exposing Couchbase clusters on the public internet. This feature is vital for tech startups that are increasingly cloud-focused, offering benefits such as agility and innovation by enabling public service offerings over the internet. Despite the security risks associated with increased cloud exposure, the article emphasizes the importance of using full end-to-end encryption to protect data from malicious actors. It outlines the configuration process, including setting up Dynamic DNS and TLS, and the use of Kubernetes services to manage DNS records and ensure secure communication with Couchbase clusters. The tutorial also addresses troubleshooting tips and highlights the importance of adhering to best practices in digital security, emphasizing the necessity of TLS certificates and DNS configuration. The article concludes by acknowledging the new features of Autonomous Operator 1.2.0, which focus on upgradability and ease of use, and encourages feedback from users.

In large organizations, managing security across the enterprise can become challenging as the number of apps and users grows. A role-based access control model allows for fine-grained control over user permissions, but managing individual user permissions can be difficult when dealing with a large number of users. To address this challenge, Couchbase has introduced groups, which enable administrators to bundle users by job roles and easily control their permissions. With Couchbase 6.5, group creation is easy, and privileges can be associated directly with groups, making it simpler to manage security. The admin interface now allows for toggling between user and group views, making it easier to create and manage groups, as well as assign roles to users within those groups.

Databases are converging, making it increasingly challenging to pinpoint the best use scenarios for each without understanding their underlying mechanics. This text explores Postgres, particularly its JSONB support, and compares it with NoSQL databases like Couchbase. Although Postgres is traditionally a relational database, its JSONB feature allows for JSON storage, offering advantages over traditional table structures, yet it is sometimes mistakenly seen as a NoSQL replacement. The article delves into the differences in data modeling between RDBMS and document databases, highlighting the complexity of JOIN operations in relational models compared to simpler, single-structure storage in NoSQL. It discusses the query languages of Postgres and Couchbase, noting that Postgres's JSONB is complex and lacks intuitive query capabilities, whereas Couchbase's N1QL is more SQL-like and user-friendly. Indexing in both systems is compared, with Couchbase offering more mature and flexible indexing options. The performance of Postgres's JSONB is promising but has limitations in dynamic models, while Couchbase excels in scalability and performance for specific use cases. The article concludes that while Postgres JSONB is valuable for certain transactional and metadata scenarios, Couchbase is better suited for applications requiring scalable performance and less rigid transactional support.

Rebalance is a critical component of Couchbase architecture that allows for online cluster management operations, including adding/removing nodes, online upgrades of hardware or software, and recovery after node failure. In Couchbase Server 6.5, rebalance has been made more robust, manageable, and faster through various improvements, including automatic retries for failed rebalances, auto-failover during rebalance, and improved rebalance progress monitoring. The new features aim to reduce the impact on front-end applications and improve performance, especially with larger data sets. Additionally, a rebalance report can be obtained through the REST API, providing detailed information about the last rebalance process.

Version 6.5 of Couchbase has introduced significant enhancements to its backup and restore technology, focusing on improved performance, storage efficiency, and consistency. Key updates include leveraging server-side value compression, changing the storage engine and format, and isolating metadata, resulting in a ~4x improvement in backup performance and a reduction in required storage to about 40% of the original dataset size. The new version also features a developer preview of an "info" command for detailed backup file listings and consistency measurement tools that address the distributed nature of Couchbase. Additional improvements encompass support for alternative addresses, automatic bucket creation during restores, and enhanced error messaging. For more comprehensive details, users are encouraged to consult the full documentation and release notes.

Couchbase Server 6.5 introduces new capabilities for configuring TLS cipher suites, enhancing the security of data exchanges by allowing users to select specific ciphers. Cipher suites are crucial for securing network connections through TLS, involving a combination of protocols and algorithms for encryption, key exchange, authentication, and message verification. In Couchbase 6.5, users can specify which cipher suites to use, with the server defaulting to the strongest common option if none is specified. The blog provides guidance on configuring these settings and verifies their implementation through command-line examples. This flexibility ensures that users can maintain high security standards by adopting the latest ciphers. The blog encourages users to explore Couchbase 6.5's features and offers additional resources for further learning.