May 2016 Summaries

Docker Swarm is a more opinionated but powerful alternative to Kubernetes, offering features like easy setup, dynamic scaling, and service discovery. However, it has caught up with Kubernetes in terms of performance and is recommended as the best choice for many use cases. Docker Links are legacy and should be replaced with Docker Networking, which offers better connectivity options. Docker and Vagrant solve different problems, with Docker being a native Linux technology and Vagrant provisioning Virtual Machines independent of the host machine. It's possible to run Java applications in containers without breaking Oracle's licensing terms by using OpenJDK-based images or creating your own custom Dockerfile. To communicate between multiple containers across hosts, use Docker Networks instead of legacy Docker Links.

Hod has recently joined the developer advocacy team at Couchbase in Mountain View, California, focusing primarily on Couchbase for Mobile. With a diverse background in technology, Hod has experience ranging from programming on an HP calculator to working with Cray supercomputers and Raspberry Pi, as well as designing and building PCs. His career spans roles in scientific computing, computer security, and finance, with his most recent position at Samsung leading developer relations for mobile in North America, allowing him to witness the evolution of mobile technology. With mobile becoming the primary way to access the Internet in the U.S. around 2014, Hod is enthusiastic about working with developers to leverage Couchbase's NoSQL database technology for better mobile app development. On a personal note, he enjoys spending time working in his yard and playing with his child.

The enterprise-grade backup and restore tool, cbbackupmgr, offers a high-performance and secure solution for managing business-critical data backups and restorations, addressing the challenges of exponential data growth and the need for frequent data protection. Designed to handle large datasets and ongoing data expansion, cbbackupmgr supports an incremental forever strategy by merging incremental backups into a single full backup, thus optimizing backup efficiency and minimizing cluster overhead. This approach is particularly effective in scenarios involving massive data volumes, such as a 64 TB deployment with 1 TB daily changes, where a periodic full backup is impractical. By employing the Periodic Merge strategy, cbbackupmgr can complete backup operations significantly faster than traditional full backups, reducing the time required from 24 hours to less than 21 minutes at a speed of 800MB/sec. The tool's capability to perform incremental backups and merge them into a comprehensive full backup makes it a valuable asset for enterprises seeking reliable and efficient data management solutions.

The final installment of a blog series introduces full CRUD functionality for an ASP.NET application using Couchbase, allowing users to add, edit, and delete person documents through a web interface instead of directly in the Couchbase Console. The blog outlines the creation of a simple HTML form for adding new entries, the use of Linq2Couchbase for saving and updating documents, and the implementation of an "upsert" operation for managing existing data. Additionally, it explores the flexibility of NoSQL databases like Couchbase by demonstrating the ease of adding new properties, such as phone numbers and favorite movies, without altering a fixed schema. The blog concludes with guidance on further exploring Couchbase, including resources for the ASP.NET Identity Provider for Couchbase and Linq2Couchbase, and invites readers to engage by improving the example or participating in Couchbase's creative application challenge. The source code for the example is available on GitHub, encouraging feedback and further experimentation.

Couchbase is enhancing its PHP SDK in anticipation of the Couchbase Server 4.5 release and PHP 7 innovations, with the introduction of the third beta version that includes new features and internal improvements. A key feature is the Sub-Document API, which provides fine-grained control to save network traffic by operating on specific parts of large documents, demonstrated through PHP code examples. The SDK also introduces a more straightforward approach to index management via a new API in PHP SDK 2.2, allowing developers to list, create, and remove indexes through the BucketManager. Additionally, a change in the representation of the CAS value makes it easier to handle within PHP applications by encoding it as a string token, facilitating serialization. The release can be installed using pecl, and further beta versions are expected before the general release, with feedback and issues encouraged through the Couchbase community platforms.

The author of the text attended JFokus and met Matti Tahvonen from Vaadin, who presented an open-source web framework for rich Internet applications in Java. The author was impressed by Vaadin's capabilities and decided to explore it further. They created a simple CRUD application using Vaadin and Spring Boot, which stores data in Couchbase. The application uses the `@Document` annotation to define the entity, and the `CouchbasePagingAndSortingRepository` interface to handle database operations. The author also used Viritin, a server-side enhancement library for Vaadin, to create a form that allows users to edit customer data. The form is connected to the backend using Vaadin's UI components, such as the `MTable` and `Button` classes. The application uses Spring Boot's auto-configuration features to simplify development. Overall, the author found the process of building the CRUD application with Vaadin and Spring Boot to be relatively easy and straightforward.

Containers are not isolated VMs due to sharing a common kernel, which is the main idea behind their lightweight nature. Sharing a kernel with unknown containers can be uncomfortable and raises concerns about security, much like double dipping at a party. The level of trust in sharing a kernel varies among individuals, ranging from those who wouldn't share with strangers to others who are more tolerant or even adventurous. Understanding the implications of running multiple containers on the same kernel is crucial for effective use of containers in production environments.

SCRAM authentication, a feature introduced in Couchbase version 4.5, is utilized automatically if both the Java SDK version is 2.2.5 or higher and the Couchbase Server version is 4.5 or higher, allowing secure authentication for accessing buckets with a password. To monitor the SCRAM handshake process in Java, developers can adjust the logging level to FINEST, which provides detailed insights into the authentication exchange between the client and server. The server supports multiple authentication methods including SCRAM-SHA512, SCRAM-SHA256, SCRAM-SHA1, and CRAM-MD5 PLAIN, with the client typically opting for the strongest, SCRAM-SHA512. For older versions like Couchbase 4.1, CRAM-MD5 is used instead. Network traffic can also be monitored using tools like Wireshark, but the use of TLS is recommended to prevent man-in-the-middle attacks, as SCRAM's mutual authentication requires the server to verify its knowledge of the secret using the client's hashed password.

Couchbase has released the Beta version of PHP SDK 2.1, which now supports PHP version 7. A final version is anticipated before the General Availability (GA) release, which will include new features from the upcoming server update. Users can install the beta version via PECL or download it from the Couchbase SDK Downloads site, but they must ensure libcouchbase 2.5 or higher is installed first. Any issues can be reported on the Couchbase Communities site or through their issues tracker, and users are encouraged to stay updated on future developments.

I am excited to say that I will be at DevNation 2016 in San Francisco with my colleague, Arun Gupta. This is a huge developer conference hosted by Red Hat, where we'll be discussing full stack development using Node.js and an open source NoSQL database called Couchbase Server. Full stack development allows for scalable applications built with APIs that can be accessed from multiple front-end platforms, making it ideal for modern development. Attendees can expect to learn how to build a robust API with a NoSQL database, code driven with examples and demos. I'm looking forward to speaking at DevNation because I love discussing modern development platforms like Node.js. As a Developer Advocate for Couchbase, I have experience in various technologies including Node.js, while my colleague Arun Gupta is the vice president of developer advocacy at Couchbase with expertise in leading cross-functional teams and developing developer communities.

Couchbase 4.5 introduces a new security feature called SCRAM-SHA, aimed at enhancing user authentication and defending against brute-force attacks. This mechanism, which stands for Salted Challenge Response Authentication Mechanism, improves upon the previous CRAM-MD5 login scheme by exchanging hashed items between the client and server, which cannot be replayed. SCRAM is adaptable, supporting various hash algorithms like SHA1, SHA-256, and SHA-512, with Couchbase defaulting to the highest level of security, SHA-512. To utilize SCRAM-SHA, users need to upgrade to Couchbase 4.5, and the SDK will manage the authentication process, reverting to CRAM-MD5 for older Couchbase versions if necessary. While SCRAM-SHA offers enhanced security features, using TLS is still recommended for optimal protection.

Microsoft's recent updates to Windows Azure have enhanced the platform with new features, including support for Git deployments and node.js apps, as well as the ability to run virtual machines, extending Azure's capabilities beyond its traditional Platform as a Service model to include Infrastructure as a Service functionality. This development is particularly advantageous for developers, especially those working with .NET, who are interested in deploying cloud-based ASP.NET applications with a Couchbase cluster. Jim O'Neil, a Developer Evangelist for Microsoft, collaborated with the author to create a Couchbase-based ASP.NET application on the new Azure, resulting in an informative multi-part article available on Jim's MSDN blog. The article provides detailed guidance on setting up a Couchbase Server cluster, offering insights into network design, security, and cloud deployment management, and includes a PowerShell script for cluster management and instructions for installing the TapMap app with updated sample data.

Azure's new Marketplace experience, unveiled at TechED Europe, simplifies the deployment of Couchbase Server on the platform by offering a streamlined selection process and two support package options: Couchbase Server 3.0 with either Silver or Gold Support. This update enhances accessibility for users and complements best practices discussed in a recent online talk with Azure representatives Rafael and Jeremiah, providing a more comprehensive and efficient deployment experience for Couchbase users on Azure.

In this article, the advancements in Couchbase Server's global secondary indexing, particularly in version 4.5, are discussed, focusing on the distinction between standard and memory-optimized indexes. The introduction of "circular writes," a new write mode for standard global secondary indexes, is highlighted as a significant improvement over the previous append-only mode, reducing the frequency of compactions and thus improving efficiency. Circular writes reuse orphaned space within files, minimizing the need for new writes and significantly decreasing compaction occurrences, which conserves IO resources. The article explains configuration options for write modes and compaction triggers, emphasizing the benefits of circular writes for new clusters while maintaining append-only writes for backward compatibility. The piece underscores the importance of choosing the appropriate index storage option based on the available memory and performance requirements, with memory-optimized indexes offering superior performance but at a higher memory cost, while standard indexes offer flexibility with efficient disk IO when memory is limited.

Full Text Search (FTS) is introduced as a developer preview in Couchbase 4.5, offering a glimpse into future capabilities not yet ready for production use. Unlike basic SQL searches, FTS provides a more sophisticated, language-aware search experience, ranking results by relevance. The blog details the process of setting up a Full Text Search index on the travel-sample data bucket within Couchbase, including configuring type mappings to restrict searches to specific document types, such as "landmark." It also covers executing FTS queries in the Couchbase Console and ASP.NET applications. In ASP.NET, the author demonstrates using the Couchbase .NET SDK to perform searches, highlighting terms within search results. The blog emphasizes that FTS is currently a developer preview, inviting users to participate in its development by reviewing and providing feedback on the Full Text Search RFC on GitHub.

Couchbase Server offers two storage options for global secondary indexes: standard GSI and memory optimized GSI (MOI), with MOI introduced in version 4.5 to address the challenges of high-demand applications like travel-itinerary and fraud detection. MOI provides significantly better latency and throughput compared to standard GSI by utilizing lock-free processing and skiplist structures for in-memory storage, avoiding the slower disk speeds of standard GSI which uses ForestDB. MOI requires machines with large memory capacity since it keeps indexes in RAM, while standard GSI can spill over to disk, thus relying heavily on IO subsystem performance. The creation process for both types uses the CREATE INDEX command, but MOI offers additional statistics and alerts to aid in index placement and management. These statistics help administrators monitor memory usage, ensuring that indexes are placed where sufficient memory is available, and alerts notify them when RAM usage approaches critical levels. Future discussions will cover enhancements like the Circular Write Mode for standard global indexes.

Docker has gained popularity due to its ease of use in deploying microservices and other tasks, and it can also be used to install and configure Couchbase. A video is available that demonstrates how to set up a cluster of Docker containers running Couchbase Server, providing an easy-to-follow example. For more information on using Docker with Couchbase, one can visit the developer portal or check out the landing page for further details.

The blog post provides a detailed walkthrough of integrating Linq2Couchbase with an ASP.NET application, emphasizing the importance of adhering to the SOLID principles by moving data access responsibilities from the Controller to a repository class. It explains the refactoring process to use a more defined Person class instead of dynamic objects and demonstrates how to implement Linq2Couchbase to convert Linq queries into N1QL queries, thus facilitating database operations. The post also discusses the use of Couchbase's indexing capabilities, highlighting the need for a "type" field in documents for effective querying, and touches on the concept of ScanConsistency, which balances speed and accuracy in query execution. The article concludes by indicating upcoming topics on document manipulation with Linq2Couchbase and invites readers to engage through comments or direct contact for further assistance.

In an effort to optimize slow N1QL queries on Couchbase Server 4.1, the author shares their experience of enhancing an internal Java application's performance hosted on a low-budget Amazon EC2 instance. Initially, the queries were sluggish due to reliance on a generic primary index. To improve query performance, the author implemented a covering index specifically tailored to the query's needs, targeting the "date" and "type" properties, but noticed no performance change initially. After running an "EXPLAIN" command, it was revealed that the primary index was still being used because not all documents contained the necessary properties. By revising the query to check for the presence of these properties, the author successfully utilized the covering index, resulting in a significant performance boost.

The fourth episode of the NoSQL Database Podcast, titled "Querying with NoSQL," features a discussion with hosts and guests Keshav Murthy and Prasad Varakur about the differences in querying between NoSQL and relational databases. The episode covers topics such as indexing, MapReduce, and Couchbase's N1QL query language, offering insights into how these elements function within the NoSQL framework. Listeners can access the episode on platforms like iTunes and Pocket Casts, or listen directly through a media player provided. The podcast also invites questions via email and directs listeners to additional resources on Couchbase's developer portal for more information on transitioning from relational databases to NoSQL and querying for data.

Couchbase has announced the availability of Couchbase Community Edition 3.1.3, which builds on version 3.0.1 by incorporating key bug fixes to enhance product stability. This release is part of Couchbase's efforts to foster growth within its developer community by providing a robust platform for experimentation and learning. The Community Edition serves as an entry point for developers to familiarize themselves with Couchbase, while the Enterprise Edition is recommended for business-critical applications in production environments. The company encourages developers to download the latest version or review the release notes to explore the new features and improvements.

With Couchbase Server 3.0, the new "full ejection" option allows for more efficient use of memory by caching larger parts of the working set, rather than just keys and metadata, which is particularly useful for large databases with a smaller active working set. This feature enables servers to reserve memory for 'hotter' parts of the data that are frequently accessed, reducing memory usage and improving performance. The new option can be easily enabled per bucket in the admin console, without requiring any changes on the application side. Benchmarks show significant reductions in memory consumption, with orders of magnitude less memory used for metadata compared to previous versions. However, this feature compromises on low-latency access to keys and metadata that have been ejected, requiring additional IO operations to retrieve them back into memory. This option is particularly suitable for large datasets and applications that don't require aggressive low latencies for all data access.

Couchbase has released version 2.2.8 of its .NET SDK, which primarily includes bug fixes and Developer Previews (DP) for upcoming features in support of Couchbase Server 4.5. This version introduces a preview of Full Text Search, enabling search capabilities over JSON documents similar to Elastic Search, though it is not yet suited for production use. Another key feature previewed is AT_PLUS for N1QL Queries with MutationTokens, facilitating the "Read Your Own Writes" (RYOW) functionality to ensure up-to-date indexes. Additionally, the release addresses several technical tasks, bug fixes, and improvements, such as refactoring WebRequest/WebResponse to System.Net.Http namespace classes and enhancing vBucket Retry Logic. Further detailed discussions on these features and improvements are anticipated in future blog posts.

This blog post, part of a series on Couchbase integration, details the process of starting an ASP.NET MVC project and integrating Couchbase using its .NET SDK. The author begins by explaining how to set up a new ASP.NET MVC application in Visual Studio and incorporate the Couchbase client library via NuGet. Instructions are provided for configuring the application to connect to a Couchbase cluster by specifying a cluster node in the Global.asax.cs file and initializing the ClusterHelper. The importance of using an IoC container, specifically StructureMap, for managing dependencies is highlighted, and a guide is given for registering an IBucket instance, which accesses Couchbase buckets. The post concludes with a demonstration of using the IBucket in an ASP.NET controller to retrieve and display data from Couchbase, setting the stage for more advanced functionality to be covered in subsequent posts.

Couchbase 4.5 introduces enhancements to its indexing capabilities with features like exclusive in-memory secondary indexes using the memdb storage engine and a high-performance consistency model called "AT_PLUS." These improvements provide developers with finer control over data consistency during queries, allowing for efficient handling of mutations by using mutation sequence IDs. The AT_PLUS consistency model, which requires enabling mutation tokens, offers significant performance gains by synchronizing queries with specific data mutations, reducing latency and resource usage. Couchbase's global secondary indexes remain user-definable with the N1QL language and continue to support flexible query configurations, with options for default, request plus, and high-performance consistency. The update aims to enhance the developer experience and system performance, encouraging users to explore the new features by downloading Couchbase 4.5.

The document announces a presentation at Apps World North America, a conference focused on mobile, game, and web development with an emphasis on enterprise solutions, held at the Santa Clara Convention Center in California on May 11th. The presenter will deliver a developer-oriented talk titled "Cross Platform Storage & Sync with Couchbase and Ionic Framework," which will explore the use of Couchbase Mobile within an Ionic Framework for Android and iOS applications. The session promises to include code examples and demonstrations, offering attendees an opportunity to enhance their understanding of the subject. For those interested in getting a head start, the Couchbase Developer Portal is recommended as a resource for related mobile development topics.

Indexing and querying in big data scenarios require robust solutions to manage high velocity, variety, and volume of data without compromising on response times. Distributed database systems offer two primary types of indexing: local indexes and global indexes. Local indexes optimize for fast indexing by having each node in a cluster index its local data, but this method can lead to increased latency as the cluster grows because each node must independently determine partial results before a coordinating node compiles them. In contrast, global indexes are independently partitioned and stored separately, reducing latency by having a centralized index that can quickly provide answers without the need for scatter-gather operations. Couchbase Server utilizes both local indexing, through Map/Reduce Views, and global secondary indexes (GSIs), which can be deployed independently to handle queries efficiently. The Couchbase Server offers two storage options for GSIs: standard GSI and the newer memory-optimized GSI, introduced with version 4.5, which is discussed further in a subsequent post.

Part 2 of the blog series on Couchbase with Windows and .NET delves into the fundamental terminology and architecture of Couchbase, contrasting it with traditional relational database systems. A Couchbase cluster, a scalable collection of nodes, manages data through buckets, which store documents that can be retrieved using unique keys, N1QL queries, or Views. The flexibility of Couchbase allows for the configuration of nodes to provide various services such as data storage, indexing, querying, and full-text search, with the ability to scale out by adding servers. The blog also introduces N1QL, a superset of SQL, to query JSON documents, making it accessible to developers familiar with SQL, and emphasizes the importance of indexes in handling large volumes of data efficiently. The author, Matthew Groves, encourages readers to familiarize themselves with these concepts before diving into coding, offering further assistance and engagement through comments, Twitter, or email.

The sub-document API introduced in Couchbase 4.5 enhances the existing Memcached key-value API by allowing targeted operations on parts of a document without needing to retrieve or update the entire document, thereby improving performance and conserving resources. This API includes two main methods, `LookupIn` and `MutateIn`, which facilitate efficient data retrieval and mutation within a single document and are consistent across various programming languages such as Go, Node.js, Java, and Python, ensuring code portability. The API's functionality is exemplified through methods provided by the `MutateInBuilder` and `LookupInBuilder`, which allow for operations like adding unique values, inserting arrays, performing atomic counter operations, and checking path existence. By using these methods, developers can efficiently manage document data with reduced server load, and the article encourages trying out these new features using the Couchbase 4.5 Beta, which can be easily set up via Docker.

The blog post introduces the Sub-Document API, a new feature in Couchbase Server version 4.5, which enhances JSON document management by allowing users to specify and operate on specific parts of a document without retrieving the entire content. This feature, integrated into the Memcached protocol, enables more efficient data handling by reducing the need to transmit large documents over the network. The Sub-Document API supports both lookup and mutation operations, such as retrieving specific values or modifying parts of a JSON document, and includes error handling mechanisms for potential issues like path mismatches or non-JSON documents. The Java SDK 2.2.6 provides a fluent interface for these operations, allowing developers to efficiently manage JSON data within Couchbase, with features such as durability constraints, CAS checks, and array manipulation. The post invites users to explore the beta version and provide feedback, highlighting the potential for improved performance and flexibility in database operations.

This blog post by Matthew Groves introduces .NET developers to Couchbase, guiding them through the initial steps of downloading, installing, and setting up Couchbase Server on Windows 10. It recommends starting with the Community Edition and demonstrates how to run Couchbase as a Windows Service, access the Couchbase Console via a web browser, and configure basic settings including RAM allocation and sample data installation. The post explains that Couchbase stores data in "buckets" as key/value pairs, often using JSON documents for enhanced functionality like indexing and N1QL querying, which makes it a document database. It also touches on Couchbase's scalability through clustering and the option to enable or disable data replication. The post concludes by encouraging readers to explore the Couchbase Console and announcing future content on coding with Couchbase, while offering support through comments and direct contact.

Couchbase Server 4.5 Beta introduces several new features aimed at enhancing app development and performance for users transitioning from relational to NoSQL databases. Key updates include the Sub-Document API for granular data manipulation, a Query Workbench with auto-schema discovery for efficient N1QL query execution, and memory-optimized indexes for high-throughput performance. Additional features in the beta release encompass circular storage reuse, improved query consistency with Read Your Own Writes, and enhanced security through role-based access control and X.509 certificates. The release also features advanced capabilities like array indexing, extended join syntax, full-text search integration, and enterprise-grade backup and restore tools. While the beta version is intended for development purposes and may contain some bugs, Couchbase invites user feedback to refine the software before its general availability.