August 2013 Summaries

Faceted search functionality can be implemented in MongoDB without requiring the use of external search engines. For this example, let's say we have three facets on which we wish to search – Subject, Publisher and Language – and consider how to search efficiently, and how to generate the faceted navigation meta-data to present to the user. The performance characteristics of different approaches to implementing faceted search in MongoDB are discussed, including schema #1 where all facet types and values are stored in subdocuments in an array, schema #2 where all facet types and values are concatenated into a single string value, and schema #3 where each facet is associated with an array of tags. The benefits of these different approaches include smaller index sizes for schema #3, but the need to perform more aggregation queries when using this schema. An additional approach is discussed which involves pre-computing faceted navigation meta-data and storing it in a separate collection, which can be updated quickly by specifying that the map-reduce job operate only on the recently updated items. Third-party tools such as Solr and ElasticSearch are also available for providing faceted search capabilities.

The authors of this blog post used MongoDB's aggregation framework to analyze a domestic flights dataset, which contains information on commercial flights from 1987 onwards. They answered various questions about flight delays, aircraft ages, and the effects of Hurricane Sandy on air transportation in New York. The authors started by importing the data into MongoDB, removing unnecessary fields, and creating indexes on relevant fields to improve query performance. They used simple aggregations, queries, and visualizations to answer their questions, and also explored more complex topics such as cascading delays and the effects of Hurricane Sandy. The results showed that morning flights have the least average arrival delays, summer and Christmas seasons have the most delays, and certain states experience the worst delays. The authors also discussed challenges and lessons learned from the project, including the importance of indexing and handling issues with date fields.

The MongoDB Web Shell is a web-based application designed to emulate some of the features of the mongo terminal shell, providing a responsive command line interface similar to the desktop shell. The user interfaces with the backing mongod instance through a RESTful interface implemented on top of Flask, allowing for sandboxing capability and preloading data into resources. A key challenge was allowing blocking paradigm code without blocking in the web browser, which was addressed by using a Javascript evaluator that can pause execution at any time. The shell will be used to replace the existing browser shell on try.mongodb.org, provide better support for JavaScript syntax and MongoDB features, and make its debut on the 10gen Education platform, where users can complete assignments entirely within the education platform without needing to download and install their own copy of MongoDB or sample data. Additionally, it will be embedded into the API reference pages as part of the database's documentation, allowing users to test commands and evaluate effects against provided sample data sets. The project is Apache licensed open source software, freely available through a Github repository.

The MongoDB Connector for Hadoop is a tool that enables users to use MongoDB databases or backup files in `.bson` format as input sources or output destinations for Hadoop Map/Reduce jobs. This allows for efficient processing of large datasets, and also supports Pig and Hive languages for building complex workflows. The connector works by examining the data, calculating splits, assigning them to nodes in a Hadoop cluster, and then having each node pull data from MongoDB or BSON, process it locally, and merge results before streaming output back to MongoDB or BSON. This integration enables users to leverage both MongoDB's strengths in storing operational data sets and Hadoop's capabilities in batch processing tasks.

Big Data Is The New Normal` highlights the growing importance of big data in business and how it's becoming a norm for decision-making. Big data refers to the vast amount of information that businesses are now dealing with, which can be used to gain valuable insights and make informed decisions. While not all data will be considered big data, its impact on various industries, such as retailing, is significant. Companies like Netflix, which uses big data to understand customer behavior and preferences, are already seeing the benefits of harnessing this information. As a result, big data is expected to mature into an everyday business tool by 2016, with organizations already demonstrating its value in achieving superior results. The future of big data seems distant now, but it will likely become a crucial aspect of marketing and design decisions in the near future.