May 2017 Summaries

The WiredTiger logjam was resolved by Bruce Lucas's wait-free solution, which eliminates the need for threads to wait before copying their payloads. This is achieved by using a single register to multiplex a slot's state along with bookkeeping about joined and released bytes, allowing threads to write into the slot as soon as they receive their write offset from their join operation. The new implementation uses masking and bit-shifting to perform joins and releases atomically, without compromising any other workloads or introducing a "leader" thread. This solution has been successfully integrated into the WiredTiger storage engine in MongoDB, resulting in nearly tripled performance of the journal algorithm and almost entirely eliminated negative scaling at high thread counts.

In an effort to address performance challenges in the WiredTiger write-ahead log, which was obstructing MongoDB's aim of making it the default storage engine in version 3.2, Senior Technical Service Engineer Bruce Lucas uncovered a critical logjam issue caused by busy-waiting when the number of threads exceeded a certain threshold. By rethinking the atomicity constraints and using a novel approach that allowed the use of two separate counters within a single int64 register, Lucas and his colleague were able to eliminate the need for threads to wait by enabling them to write into slots immediately after joining. This optimization resulted in nearly tripling the journal algorithm's performance, resolving the negative scaling at high thread counts, and making WiredTiger a more efficient storage engine. Their work underscored the importance of having a fresh perspective when tackling entrenched technical issues and highlighted the satisfying impact of significant, user-visible improvements in code optimization.

Deep learning is a subset of artificial intelligence that focuses on building machines capable of intelligent behavior by using algorithms to sift through data, learn from it, and make predictions or take autonomous actions. Machine learning is a branch of AI that involves training algorithms to independently learn, reason, and perform specific tasks, with supervised and unsupervised learning being the two main approaches. Supervised learning involves using labeled data to train models and generate predictions, while unsupervised learning relies on unlabeled datasets to uncover structures and patterns in the data. The differences between machine learning and deep learning are significant, with deep learning focusing on neural networks that can learn complex patterns in data. The next part of the series will delve deeper into deep learning and evaluate key considerations when selecting a database for new projects.

The text covers two main topics related to MongoDB. Firstly, it details the challenges and optimizations in the WiredTiger storage engine's write-ahead log, which encountered bottlenecks due to assumptions that became outdated when WiredTiger became MongoDB's storage engine. The text explains how the write-ahead log, crucial for data durability, was initially optimized for specific conditions that changed, causing issues like negative scaling when tested with high thread counts. This led to a reevaluation of the log's design, particularly its handling of "no-sync" writes and thread coordination, to improve performance under new conditions. Secondly, the text announces a significant leadership transition at MongoDB, with CEO Dev Ittycheria deciding to retire and Chirantan "CJ" Desai being appointed as the new CEO. The transition is presented as a strategic move to guide MongoDB through its next phase of growth, leveraging Desai's experience in scaling technology companies. Ittycheria expresses confidence in Desai's capabilities and outlines the reasons for the transition, emphasizing the company's strong position and future potential in the context of emerging technologies like AI.