January 2016 Summaries

At a recent Fosdem conference, the discussion centered around the innovative use of Couchbase Mobile in conjunction with IoT technologies like beacons and peer-to-peer (P2P) synchronization. Beacons, which emit signals using Bluetooth LE, are highlighted for their utility in retail environments and events, such as supermarkets and music festivals, to notify users of promotions or gather crowd data. Couchbase Mobile enhances these applications by enabling offline data storage and synchronization, ensuring functionality even without an internet connection. Additionally, the potential for automatic P2P synchronization between devices using Couchbase Lite was discussed, although technical limitations prevented a live demonstration. The conference itself was praised for its organization and reliable Wi-Fi, underscoring the importance of connectivity in large-scale events.

Anti-virus software can sometimes hinder the use of Couchbase, a distributed NoSQL document database, by blocking installations, known ports, or mistakenly flagging code as malware. These issues arise due to aggressive heuristics, outdated virus definitions, or misclassifications under the Potentially Unwanted Application (PUA) category. To mitigate these challenges, Couchbase scans its binaries with popular antivirus scanners to ensure they are virus-free, and suggests verifying software safety through multiple scanners to confirm if detections are false positives. Users are encouraged to report false positives to their antivirus vendors to facilitate future whitelisting. Tools like VirusTotal are recommended for scanning files and URLs to detect malicious content and false positives. Additionally, users should verify the integrity of downloads and ensure antivirus settings do not block necessary ports for Couchbase's operation while adhering to its security best practices.

Couchbase Mobile offers a compelling solution for mobile application developers by providing a seamless offline-first experience through its components, Couchbase Lite and Couchbase Sync Gateway. Couchbase Lite, an embedded NoSQL database, allows developers to store and manage data without the constraints of a traditional schema, making it easier to handle evolving data requirements without complex upgrade scripts. The Couchbase Sync Gateway facilitates secure and efficient data synchronization across devices and platforms, mitigating the complexities of building custom solutions for data replication and conflict resolution. Unlike data services that may shut down unexpectedly, Couchbase Mobile ensures developers maintain ownership and control over their data, with easy deployment options on cloud platforms like AWS and Azure. This combination of features makes Couchbase Mobile a robust choice for developers seeking reliable, fast, and user-centric mobile applications.

Couchbase is participating in FOSDEM, a major developer-focused event attracting over 5,000 open-source enthusiasts from around the globe. At the event, Couchbase Developer Advocate Laurent Doguin will deliver two presentations on Saturday, January 30th. The first talk, scheduled for 13:30 in the IoT devroom, will focus on creating offline IoT experiences with beacons using Couchbase Mobile. Later, at 16:00 in the Embedded room, Laurent will discuss building offline embedded and mobile solutions using NoSQL databases, covering aspects such as data syncing, storage, security, and cross-platform data modeling. To enhance the FOSDEM experience, Couchbase is also sponsoring Belgian waffles and encouraging attendees to share photos with the hashtag #couchbase for a chance to win a SWAG bag.

The Sync Gateway technical series delves into configuring the Sync Function through Channels for data orchestration, focusing on data routing and access controls. Chris Anderson demonstrates using tags from Stackoverflow questions to sync specific topics to users based on their interests, generating channel names dynamically through a JavaScript function in Sync Gateway. The example illustrates how these tags, serving as document type keys, enable accessibility control, with a JSON configuration file specifying the sync function to pass tags into the channel function. In Couchbase's administration console, channels display content based on these tags, aiding in Sync Function logic development. Using Couchbase Lite's replication API, mobile clients can sync user-specific content by setting channel names, ensuring efficient data synchronization and offline functionality via the changes feed API and sequence numbers. The series highlights the importance of validating document types and providing security through user authorization, encouraging further reading for development and troubleshooting of channels in Sync Gateway.

The text discusses the process of migrating from a SQL database to Couchbase, emphasizing the need to adjust primary keys and migrate business logic to the application layer. It explains how to perform JOIN operations on Couchbase by updating foreign keys using N1QL Data Manipulation Language, which involves converting number fields to strings for key consistency. The document also highlights the challenges of migrating SQL structures like sequences, views, domains, triggers, and functions, noting that these elements require reimplementation in the application layer since Couchbase lacks direct equivalents. The text concludes by suggesting that a comprehensive migration should involve data model analysis and potential denormalization, and invites readers to share their experiences with SQL migrations.

Sync Gateway allows users to validate document types in an application by utilizing the Sync Function's implementation logic. The Sync Function is used to determine the accessibility of documents based on their type and user authentication. For example, if a document has a "friends" type, it can be processed only for the authenticated user who owns that document, while other users will not see it. Similarly, if a document has a "vehicles" or "schools" type, similar validation is applied to ensure access is granted to authorized users. The Sync Function can also be used to restrict write operations by requiring users to have access to specific channels for the documents they want to write to.

The text introduces cbq, a command-line REPL environment for N1QL that is packaged with Couchbase Server from version 4.0, noting its utility for learning and optimizing queries before application implementation. It explains that the location of cbq varies based on the operating system, providing specific paths for Mac OS X, Windows, and Linux. Additionally, it mentions the Query Workbench, a GUI tool that offers conveniences like query autocompletion and index status overviews, which is currently in developer preview for Mac, Linux, and Windows. Users are directed to download Query Workbench from the Couchbase downloads page and follow the provided guide.

Membase is gaining popularity as an open source NoSQL database technology, with increasing interest from users and developers. Membase meetups are being organized in major US cities, where project leaders and local users will lead discussions on various topics related to the technology. These meetups offer a great opportunity for attendees to learn about Membase's capabilities, hear from industry experts and users, and network with like-minded individuals. The first meetup is scheduled for next week, with more events planned for this fall and early next year in cities such as San Francisco, New York City, Boston, and Silicon Valley. Attendees can expect interactive sessions, presentations, and demonstrations of Membase's features, including use cases and customer examples.

I spoke at the London JUG and was probably over ambitious in trying to include our recent work on Views, Hadoop Connector and the Android Grocery Sync application in a single presentation besides demoing the “simple, fast and elastic” theme that Couchbase and the client libraries support. Each of these are exciting projects from a Java developer perspective. The aspect of being able to sync data between mobile clients and servers via the Couchbase Mobile Synchpoint is a sought after feature by mobile and Java developers. The Hadoop Connector provides access to the Hadoop ecosystem, enabling a balance between meeting critical time-based needs using Couchbase and mining data for intelligence via Hadoop. Views bring features like secondary indexing, filtering, ordering and aggregating data defined on the server using Javascript-based Map-Reduce functions. Overall, I am excited by the feedback from Java developers and look forward to discussing NoSQL in general and Couchbase in particular.

Couchbase is organizing a Bug Bash event to enhance the quality of the upcoming Couchbase Server 3.0 release, inviting participants to identify and report bugs in the beta version. Enthusiasts can join by downloading the Couchbase Server 3.0 Beta and its SDKs, actively searching for bugs, and submitting their findings. The initiative, which aims to recognize top bug finders, commenced immediately and will conclude on August 31st, 2014. Further details and support are available on the Bug Bash page, with additional assistance offered through scheduled Google Hangouts.

The Couch Developer Awards have shortlisted finalists in various categories after evaluating several strong submissions. For the Best Application built with Membase Server, the finalists include Tribal Crossing for Animal Party, AdAction for Delta Projects, TubeMogul for PlayTime Video Advertising, and Tune Wiki for TuneWiki Social Music Player. In the category of Best Application built with Apache CouchDB or Couchbase Single Server, the contenders are MuDynamics for Blitz.io, Edelweiss for Edelweiss, Voxeo for RemoteAgent, NEA for Robert’s Rules of Order New Business Items, Menagerie Vet for Menagerie Whiteboard, Nodester for Nodester, and 2600hz for Whistle. The Best Mobile Application category features Hexeditreality for Couchpad, Accessible Places for Accessible Places, Mobile Accord for GeoPoll, and Radical Dynamic for Group Complete. The winners will be revealed at CouchConf on July 29th, with tickets still available but selling fast.

The text outlines a process for transferring data from a SQL database, specifically Postgres, to a Couchbase database using Java and Spring Boot. It details setting up a JDBC connection with the necessary dependencies and configurations for accessing the SQL database and retrieving its schema using DatabaseMetaData. The process involves converting each table row into a JSON document, which is then stored in Couchbase. This transformation is facilitated by a custom RowMapper that maps SQL data types to JSON-compatible types. The guide also covers configuring Couchbase connections, including setting up hostnames, bucket names, and passwords, and it provides both synchronous and asynchronous methods for importing data into Couchbase. The text concludes by packaging these steps into a single project, emphasizing the importance of proper configuration and highlighting future work on transferring SQL business logic to the application layer.

Dzone.com features articles discussing the integration of JSON with SQL, highlighting how SQL extensions are adapting to the flexibility of JSON. Various databases, including operational document databases like Couchbase, MongoDB, and DocumentDB, natively support JSON, allowing fast data access and SQL-based querying, except for MongoDB which uses a find() method. Relational databases such as SQL Server, Oracle, MySQL, and Postgres retrofit JSON using a data type similar to XML, translating JSON into tables, columns, and rows. Columnar and key-value databases like Cassandra and Riak are also incorporating JSON by extending their native models. Additionally, SQL-based analytics languages like Drill, Hive, and Spark have extended their capabilities to process JSON, reflecting its popularity as a serialization format.

Couchbase Kafka Connector 1.2.0 introduces a sample Kafka consumer alongside the existing producer, designed to facilitate interaction between Kafka and Couchbase Server. This release includes a detailed setup guide for creating a local testing environment using Vagrant, VirtualBox, and Ansible, and provides sample code for a "Hello World!" project that demonstrates data flow from Kafka to Couchbase. The guide covers installation instructions, environment provisioning, and running sample applications, which include a generator to create messages, a producer to send data to Kafka, and a consumer to output messages. Additionally, it highlights customization options for filters, encoders, and state serializers to tailor data processing and stream cursor management, encouraging experimentation with the code to suit different development needs.

The NoSQL Database Podcast episode 1 is hosted by Couchbase and features interviews with industry leaders Perry Krug from Couchbase, Tim Berglund from DataStax, and Srini Penchikala from InfoQ. The podcast explores the history of NoSQL, its differences from relational databases, and how companies are using it in their technology stacks. Industry insights and common questions about NoSQL are discussed in this episode, which is now available on Pocket Casts and iTunes, and can be listened to for free on the Couchbase website.

The Couchbase Mobile Sync Gateway provides various ways to authenticate and authorize users for mobile applications. Authentication methods include HTTP Basic Authentication, Pluggable Authentication (such as Facebook), and Custom Authentication. Authorization mechanisms involve the use of 'admin_channels' feature in the JSON configuration file, which enables a list of channels where the user has access to once they are authenticated. The Sync Function allows users to submit revisions to documents by checking if the users are authorized to access the document.

A tutorial was created to demonstrate how to synchronize data between a web application and Couchbase Server using client-facing technologies such as AngularJS, HTML, and CSS, facilitated by the JavaScript library PouchDB, which connects with Couchbase Sync Gateway for data replications. Due to popular demand, a step-by-step video tutorial was developed to supplement the original blog post, aiming to address any uncertainties in the development process.

Couchbase Server and Apache CouchDB are both open-source, document-oriented database systems, but they exhibit distinct differences in architecture and features. Couchbase Server functions as both a key-value and document store, supports SQL-like queries through N1QL, and offers features such as strong consistency, integrated caching, and cross-datacenter replication. It is designed to operate as a distributed cluster with automatic failover and master-master replication, with its development led by Couchbase Inc. Apache CouchDB, on the other hand, is a pure document store that emphasizes eventual consistency and operates with individual nodes containing full copies of data, using JavaScript-based views for querying. It supports master-master replication but lacks integrated caching and automatic failover, and is developed by a community of developers using Erlang. Both systems are licensed under Apache 2.0, and while Couchbase Server provides various official SDKs and connectors for integration, CouchDB relies on community-supported libraries.

The text discusses the evolution of IT from traditional commercial software and proprietary hardware towards open source systems, fueled by the rise of Linux in data centers and the popularity of smartphones and tablets. Open source participation, such as contributing to projects like Couchbase and Apple's open-sourced Swift, is portrayed as accessible and beneficial, offering opportunities for collaborative problem-solving and skill development. The document suggests that contributing to open source can range from small code additions and improving documentation to fixing bugs and reporting issues, emphasizing the supportive community and faster resolution times. The text encourages engagement with open source projects by following GitHub guidelines, highlighting the collective advancement in software development and the value of detailed contributions for the broader developer community.

The text provides a tutorial on utilizing roles within Sync Gateway to manage user privileges in an application designed for sharing restaurant reviews. It introduces three distinct roles: level-1 users who can post reviews requiring moderator approval, level-2 users who can post without validation, and level-3 users who function as moderators with the ability to approve or reject reviews. The guide walks through configuring the Sync Gateway by defining users and roles within a configuration file and leveraging the Sync Function to manage data access and write permissions based on roles. This is achieved by granting access to channels through roles, which simplifies data management by allowing users to inherit channel access via their roles rather than direct user assignment. The tutorial includes practical scenarios demonstrating the creation and assignment of roles, as well as validating document access and write operations, emphasizing the flexibility and efficiency roles offer in managing hierarchical user privileges.

This blog post discusses how to store large binary data, such as PDFs and PowerPoint presentations, in Couchbase using a custom chunk manager. The goal is to reduce the number of systems and licenses needed, while benefiting from Couchbase's replication features. The architecture involves two buckets: `cmismeta` for metadata and `cmisstore` for blobs. When a document is inserted into a folder, metadata is stored in `cmismeta`, and the blob is chunked and stored in `cmisstore`. Each chunk has a unique key, and the number of chunks depends on a parameter set by the project. The post also describes how to retrieve the file from Couchbase by concatenating each part and sending it to a stream. The workbench tool provided by Apache Chemistry can display the document content, including its mime type. The architecture and implementation are designed to be scalable and efficient, allowing for large binary data storage in Couchbase.

Couchbase is leveraging its active developer community to improve its products by releasing early developer previews of features like the free text search tool CBFT and the query workbench, inviting feedback to refine their development. To facilitate easy testing, a new Docker image has been created that includes a preconfigured Couchbase instance with sample data and services like data, index, and query pre-activated, allowing users to run everything in a single image. The Docker setup simplifies access to Couchbase and its new features, and developers are encouraged to provide feedback to help enhance the tools further. Additionally, the full-text queries in CBFT utilize Bleve, a Go-based library, highlighting Couchbase's commitment to open-source projects.

Couchbase offers a series of free online training courses designed to introduce newcomers to NoSQL databases and Couchbase technology. These courses begin with foundational topics, such as the fundamentals of NoSQL data management and the essentials of Couchbase technology, and are structured with a combination of videos and learning materials, concluding with quizzes to reinforce understanding. For those interested in application development, additional courses delve into the N1QL query language and NoSQL application development using Java, each complete with hands-on labs and quizzes. Mobile developers can also explore specific courses for iOS and Android, learning the basics of Couchbase Mobile. These introductory courses aim to provide a comprehensive starting point for anyone looking to develop a strong understanding of Couchbase technologies, supported by further resources available through their developer portal and community forums.

The text details the process of implementing a picture upload feature for a user profile within the Touchbase application, building on previous parts of a tutorial series. It outlines the integration of front-end and back-end components necessary for uploading, cropping, and storing profile images. The process involves using Node.js modules, such as Multer for handling file uploads and GraphicsMagick for image manipulation, including cropping and downsizing, based on user input from ng-cropper. The images are then converted to a base64 string and stored in Couchbase with a document ID linked to the user's ID. Additionally, the text describes how the system retrieves and displays the stored images using a specific function, ensuring that user profiles are updated with personalized images rather than default ones. The tutorial underscores the importance of efficient image handling to enhance user experience, especially on mobile devices.

This blog post introduces Continuous Deployment, a process that automates the deployment of an application, using Jenkins as a CI/CD tool. The author sets up Jenkins to deploy a .NET Windows Service, using Visual Studio and Git version control. They install the necessary plugins, configure MSBuild, and add a build step for NuGet package restore before building the project. The source code is provided in a GitHub repository, where users can clone and modify it to suit their needs. The post explains how Jenkins automates the deployment process, including compiling and installing the Windows Service application, stopping and starting the service, and logging its activity to Couchbase Server.

Three new or updated books about Couchbase were published in late 2015, covering the topic from different angles. Pro Couchbase Server 2nd ed., published by Apress, provides 349 pages of comprehensive information on the subject. Learning Couchbase by Packt Publishing offers a practical approach to learning the platform. Meanwhile, Pro Couchbase Development: A NoSQL Platform for the Enterprise by Apress focuses on the development aspects of Couchbase.

Couchbase Chennai recently held its inaugural meetup, which proved to be a successful event featuring a series of talks and welcoming new community members. The presentations included an introduction to Couchbase Mobile by Anish K Mathew and Kadhambari Anbalagan from RedBlackTree, as well as an introduction to Couchbase by Karthik Babu Sekar, a Solution Architect at Couchbase. These sessions were recorded and shared with the wider Couchbase Community, encouraging those interested in participating in future meetups or contributing to the organization to join local groups or take on tasks fitting their interests.

The tutorial covers the use of Couchbase Lite in a React Native application, including creating a database, persisting documents locally, using Sync Gateway for bi-directional synchronization and server-side user authentication. The code is written in JavaScript and uses the React Native framework to build a simple Todo application with multiple users. The tutorial includes step-by-step instructions and explanations of key concepts such as Couchbase Views, replications, and design documents. It also provides a starter project that can be used as a basis for building more complex applications.

Schemaless databases offer flexibility and low cost, allowing enterprises to quickly elicit actionable insights from their data in today's data-centric world. They eliminate the need for rigid schema design, data type limitations, and pre-setting data types, reducing complexity and time spent on database design and normalization. Schemaless databases store data as key-value pairs or JSON documents, providing flexibility in data representation and allowing users to closely model entity-relationship models. Benefits include reduced time spent on database design and normalization, simplified ETL/ELT processes, ongoing change management, real-time analytics on streaming data, and on-demand scalability to meet extreme volumes, velocity, and variety of data.

A recent change in Couchbase Server 4.1 regarding the way N1QL returns results for star queries has affected the deserialization process in the .NET SDK, resulting in the Query<> method yielding the correct number of results but with properties set to their default values. This issue occurs because the query SELECT * FROM default now returns JSON results with each document nested under the bucket's name, contrasting with the earlier format where results were returned as an array of JSON document bodies, which the .NET SDK was designed to handle. To address this, it is recommended to modify the query to SELECT default.* FROM default, which adjusts the output to the expected format and ensures that objects are returned with correctly deserialized values.

The author of the text is excited to report that N1QL, a new dialect of SQL, has been developed in Couchbase Server 4 to tackle the challenges posed by big data. Big data has velocity, variety, and volume, but SQL can only handle two of these "v"s - velocity and volume. However, N1QL can also handle variety by allowing it to easily traverse complex structures in JSON documents. This allows for powerful extensions such as querying over arrays embedded within JSON documents, using operators like ANY/EVERY, and commands like UNNEST & NEST that can flatten or unflatten nested arrays. One of the key advantages of N1QL is its ability to understand array types, which makes aggregation functions like MAX() more powerful. The author uses TSQL (the SQL dialect used by Microsoft) as an example to demonstrate how N1QL can outperform it in terms of performance and simplicity. Specifically, N1QL allows for a single FETCH operation that can perform the MAX() function without needing multiple FETCH operations, resulting in significant performance gains.

This blog post by Roi Katz provides a detailed guide for setting up LDAP authentication for Couchbase using Active Directory on a Linux environment, specifically for version 4.1. The instructions cover installing the necessary software, configuring saslauthd, and adjusting permissions to ensure Couchbase can authenticate users via LDAP. Katz outlines the process of configuring the LDAP settings, including specifying the LDAP server, search base, and admin credentials. The guide also emphasizes testing the connection with tools like JXplorer and ensuring that Active Directory ports are open. The article concludes with steps to enable LDAP authentication in the Couchbase console and offers a brief note on an upcoming update for version 5, where the process has changed, suggesting using the couchbase-cli for LDAP settings in the absence of the "LDAP Auth Setup" tab.

The text discusses the implementation of bulk operations in Couchbase using the Go programming language, highlighting the importance of batching to enhance network performance by increasing throughput and reducing latency. It explains the process of building a simple application with Couchbase by using the "gocb" package to manage connections and perform CRUD operations on documents through the bucket API. The example given describes inserting and retrieving 10 documents using the BulkOp interface, which facilitates batch operations by pipelining requests. The text emphasizes the simplicity and efficiency of using bulk operations with Couchbase, supported by the Go SDK, and mentions additional operations like prepend, remove, replace, and upsert. It encourages users to refer to the Couchbase Developer Guide and the accompanying GitHub repository for further insights and code examples.

Couchbase offers multiple ways to access and manipulate data, including reading and writing data by object ID, reading data from views, using N1QL for querying, and full text search via Solr or Elastic. Accessing data via object ID is the fastest method, but requires more intelligence and advanced data modeling, while views provide easy queryability over larger amounts of data and can be programmed with JavaScript. N1QL offers flexibility in getting at data, scalability through independently managed services, and performance comparable to traditional querying methods. Full text search capabilities are provided via Solr or Elastic plugins, enabling applications to use each tool for what they're best at and get the best performance from each. The choice of access method depends on the application's functional and performance needs, with a mix-and-match approach often being the most effective solution.

In the exploration of data modeling with Couchbase's N1QL, key considerations include representing document types and modeling relationships between documents. While existing JSON data can be queried without changes, optimizing the data model for N1QL can enhance query efficiency. Document types play a crucial role by moving responsibilities of identification and retrieval into the document itself, allowing for more efficient indexing and querying. Unlike SQL's use of "FROM" to narrow query scope, N1QL uses document types within a bucket to distinguish data. Relationships between documents, previously managed by embedding, can now be handled using JOINs in N1QL, facilitating one-to-one, one-to-many, and many-to-many relationships. Overall, while N1QL introduces some new practices, it allows developers to maintain much of their existing data structures, with the potential for improved document shapes emerging through further exploration and community sharing.

The 2016 Couchbase meet-up calendar in London begins with a hands-on N1QL workshop on January 20th at the Couchbase London office in Shoreditch. The event will cover the fundamentals of N1QL, its differences from standard SQL, handling nested and heterogeneous data, and data modeling for N1QL. Participants are encouraged to bring their own laptops with Couchbase Server 4.1 pre-installed to start their N1QL journey. The workshop will also offer food and drinks, providing a cozy environment for attendees on a typical cold and rainy January evening. For information on future sessions, attendees can check their local Couchbase meet-ups and the @couchbasedev Twitter feed.

Hashing passwords before storing them in a database is crucial for security, as it protects sensitive information from being exposed in the event of a data breach. Unlike encryption, which can be decrypted, hashing transforms data into a format that cannot be reversed, thereby enhancing security. The bcryptjs library for Node.js is a popular tool for implementing password hashing, as it allows the generation of a hash using a password and a salt, making it unreadable to unauthorized parties. Additionally, bcrypt provides functions to compare plain text passwords with hashed ones, enabling secure user login verification without revealing the original password. An example involving bcryptjs, Couchbase, and uuid demonstrates how to create a secure system for storing and validating hashed passwords, emphasizing that plain text passwords should never be stored in databases, regardless of their security level.

The text addresses the question of whether Couchbase, a database platform, is suitable for storing images and other binary objects, ultimately advising against using databases for such purposes. The author, an experienced architect, argues that storing large, static objects in databases is inefficient due to high operational costs and performance issues, as databases are not optimized for this type of storage. Instead, the recommendation is to use databases like Couchbase for storing metadata and small, frequently accessed data such as thumbnails, while leveraging services like Amazon S3 or HDFS for larger, static objects. This approach combines performance, operational efficiency, and cost-effectiveness by using each tool for its intended purpose. An exception is noted where a specific use case involving streaming audio files into Couchbase exists, highlighting the importance of aligning technology choices with specific application requirements.