November 2012 Summaries

Marky is a simple application that uses Markov chains to generate nonsensical tweets based on the user's Twitter history. It utilizes Couchbase Server for data storage and processing. The application splits text into smaller chunks using a sliding window over two consecutive words, regroups these chunks in correct order to form sentences based on statistical weights. To generate a word, it queries the view using the last word output, and uses the built-in reduce function to add together values for each group of word pairs. Marky picks the word to output after a given word randomly but weighted by frequency of appearance in input text. The application is set up with a simple Clojure wrapper and can be launched by creating a configuration file pointing to the Couchbase Server cluster and Twitter account, adding seed data, and setting up jobs such as sending tweets and fetching atom feeds.

Couchbase supports JSON documents, and importing them into Couchbase Server involves using the cbdocloader tool, which streamlines the process by first placing data into an in-memory cache before asynchronously writing it to disk. The tool requires specific command-line parameters, such as username, password, node IP address, and bucket name, with options for setting RAM quota and loading data from directories or zipped folders. An example given is the Vancouver Tree Dataset, which includes details about boulevard trees in Vancouver, and was split into individual JSON files for each tree using a Python script before being loaded into Couchbase using cbdocloader. The blog encourages users to explore the dataset further by creating views and offers insight into the application's real-world utility.