May 2016 Summaries

The MongoDB team rewrote their database backup utility, mongodump, to support high-throughput concurrent workloads while maintaining its serial output. To achieve this, they utilized Golang constructs like reflection and channels to safely permit multiple goroutines to concurrently feed data into the archive. The new archive format interleaves slices of data from collections, allowing for concurrent backups of multiple databases and collections to a single file. A multiplexer is used to manage the concurrency by selecting input sources based on their readiness, using a control channel to signal completion or changes in the select cases. This design enables efficient streaming of backups over networks or into another instance as part of a load operation. The new tool is faster, simpler, and more featureful than its predecessor, thanks to Golang's powerful concurrency features.

The new native MongoDB Connector for Apache Spark offers higher performance, greater ease of use, and access to more advanced Spark functionality than any existing connector. This allows users to operationalize results generated from Spark within real-time business processes supported by MongoDB, enabling organizations to unlock valuable insights from their data quickly and act on them in real-time. The new connector is designed for developers and data scientists building modern applications incorporating sophisticated real-time analytics, and provides a more natural development experience as it's written in Spark's native language. It also offers support for advanced Spark features such as DataFrames, Datasets, Machine Learning, GraphX, Streaming, and SQL APIs, as well as data locality awareness and MongoDB secondary indexes to filter input data. The new connector is now available for early access evaluation and can be integrated with a variety of storage and messaging platforms including Amazon S3, Kafka, HDFS, relational databases, NoSQL datastores, and more.

The rise of the data lake is driven by the need to capture and analyze unprecedented volumes of data from digital transformation initiatives. However, exposing this data to operational applications without proper integration is a challenge, as users struggle to maximize returns on their Hadoop investments. The traditional Enterprise Data Warehouse (EDW) is overwhelmed by the sheer volume and variety of data pouring into businesses, making it difficult to store in a cost-efficient way. As a result, organizations have turned to Hadoop-based data lakes, which provide levels of performance, efficiency, and low Total Cost of Ownership (TCO) unmatched by EDWs. However, these data lakes are not designed to provide real-time access to operational applications, which need millisecond latency query responsiveness, random access to indexed subsets of data, and support for expressive ad-hoc queries and aggregations. Integrating a highly scalable and flexible operational database layer is essential to address these challenges and help companies act on the insights and intelligence created by their data lake.

D3.js`, a popular data visualization library, can struggle with rendering large datasets on a single thread, leading to "jank" or unresponsiveness. To address this issue, the author of the blog post delves into D3's internals and develops a custom batched rendering approach to optimize performance. By breaking down the rendering process into smaller chunks and using timeouts to manage the execution order, the author creates a responsive visual profiler that can handle large datasets without freezing the browser. The solution involves modifying D3's selections to work with batches of data, rather than relying on the library's native functionality.