April 2019 Summaries

The MongoDB team conducted an experiment to improve the repeatability of performance tests on EC2 instances by investigating CPU options and their impact on noise in system performance. They recorded data about their current setup, found that EBS instances were stable for IO operations, and then focused on tuning CPU-related knobs to minimize noise. The investigation involved testing various CPU options, including idle strategy, kernel boot, CPU frequency, and hyperthreading, which revealed that disabling these features resulted in lower noise levels. The team also ran tests with different configurations, comparing results across client-only runs and connected server runs, finding that the c3.8xlarge instance configuration yielded consistently low noise levels when using all tuning options. Furthermore, they discovered that CPU tuning had a significant impact on IO-heavy tests, which were previously thought to be less affected by CPU tuning. The experiment demonstrated the importance of fine-tuning CPU options for achieving stable and repeatable results in performance testing.

The MongoDB team set out to reduce variability in performance tests on EC2 by analyzing and controlling for environmental noise in their test configuration. They used a scientific approach to measure and analyze the noise, applying the "Assume nothing. Measure everything." principle to their experiments. By repeating tests multiple times across different EC2 clusters, they were able to identify factors contributing to variability, such as disk performance, CPU tuning, and instance types. The team found that using Provisioned IOPS and EBS disks significantly reduced test results' variation, while disabling CPU options also stabilized performance. They established canary benchmarks to monitor EC2 performance and detect potential issues before they affect MongoDB's performance. By reducing system noise, the team improved the reliability of their System Performance tests, enabling them to detect changes in MongoDB's performance, diagnose issues promptly, and address them before releases.

EC2 instances were initially assumed to have good or bad performance based on the cluster they were run on, but a series of experiments and data analysis revealed that there is no correlation between test results and the clusters. The variance in performance does not correlate with the clusters, suggesting that EC2 instances are neither inherently good nor bad, and that other factors such as configuration, instance type, and usage patterns may be more significant contributors to variability. By measuring their system and analyzing data in different ways, the MongoDB Performance team was able to disprove one of their assumptions and gain a better understanding of what contributes to performance variability on EC2 instances.

In an effort to enhance repeatability in performance tests on EC2 instances, the MongoDB Performance team sought to minimize noise by focusing on I/O performance, ultimately discovering that using Provisioned IOPS with EBS provided more stable performance compared to ephemeral SSDs. The team's experiments revealed that while i2 instances delivered impressive results in non-MongoDB benchmarks, they were less effective in practice due to MongoDB's handling of dirty data in RAM, leading to unstable performance. Consequently, the production configuration was switched to EBS disks, resulting in more consistent performance. Additionally, MongoDB announced a leadership transition, with CEO Dev Ittycheria stepping down and Chirantan “CJ” Desai being appointed as the new CEO. CJ Desai's experience, particularly in scaling companies, positions him to guide MongoDB through its next phase of growth, capitalizing on the rise of AI and data-intensive applications. Dev Ittycheria expressed confidence in MongoDB's future and emphasized the importance of timely leadership changes to strengthen the company, while also sharing personal reflections on his journey with MongoDB and plans to remain involved as a board member.

The MongoDB Blog has outlined 12 different data modeling patterns for building with patterns in MongoDB. These patterns are designed to solve specific problems and provide benefits, such as improved performance or reduced writes to the database. The patterns include Approximation, Attribute, Bucket, Computed, Document Versioning, Extended Reference, Outlier, Pre-allocation, Polymorphic, Schema Versioning, Subset, and Tree. Each pattern has its own pros and cons, and some can be used together to further enhance performance. By understanding these patterns and how they can be applied to specific use cases, developers can harness the flexibility of the MongoDB document model and build more efficient and scalable applications.

The Schema Versioning pattern in MongoDB allows for the easy updating of database schemas without downtime, making it ideal for situations where application downtime is not an option. This pattern utilizes MongoDB's support for polymorphic documents to store multiple versions of a schema in the same collection, with each version distinguished by a `schema_version` field. The pattern involves creating and saving new schema versions, which are then used by the application to handle different document types. With this approach, developers can better decide when and how data migrations will take place, resulting in less future technical debt and improved flexibility. The simplicity of the data model and the ability to combine with other patterns for extra performance make the Schema Versioning pattern particularly powerful in MongoDB.

The Document Versioning Pattern is a design pattern used to keep the version history of documents available and usable in MongoDB databases. This pattern addresses the problem of wanting to keep around older revisions of some documents instead of bringing in a second management system by adding a field to each document to track the document version, resulting in two collections: one for the latest data and another for all the revisions. The pattern is useful in highly regulated industries that require a specific point in time version of a set of data, such as financial and healthcare industries, and can be used in scenarios where there are few revisions and most queries are done on the current version of the document. The pattern is relatively easy to implement and can be applied to an existing set of documents, but it may not be suitable for use cases with frequent changes or high write volumes to the database.

The Preallocation Pattern is a design approach in MongoDB that involves creating an initial empty structure to be filled later, balancing the need for simplification with the additional resources consumed. This pattern can simplify code and improve performance in certain scenarios, such as representing two-dimensional structures or tracking accessible seating. However, its use has been limited by the deprecation of the MMAPv1 storage engine, but it still holds value in specific situations, requiring a trade-off between simplicity and performance.

Oakbrook Finance, a UK-based consumer finance business specializing in providing loans to people with poor credit history, has automated its IT operations using Microservices and MongoDB Atlas. The company faced challenges managing various data sources, creating an Intelligent Scorecard for loan applicants, and ensuring security and compliance of sensitive customer data. By adopting MongoDB as their database in 2013, they were able to handle the variety of data effectively and build sophisticated machine learning models for risk assessment. However, managing the infrastructure manually became a challenge as the business scaled. To address this issue, Oakbrook upgraded to MongoDB Atlas, which automates many operational tasks such as backups, monitoring, and performance optimization. This move not only helped them scale but also strengthened their security posture in compliance with regulations like GDPR. The company is now transitioning from a monolithic code structure to a microservices architecture, enabling faster development and deployment of new features while maintaining customer focus.

Matt Bettinson, an infrastructure engineer at Oakbrook Finance, has been instrumental in addressing the company's data management challenges by leveraging MongoDB's capabilities to streamline their loan services for individuals with poor credit histories. Oakbrook, based in Nottingham, specializes in using technology and machine learning to assess credit risk, and has achieved significant growth, ranking 16th in The Sunday Times Hiscox Tech Track 100 list. Facing time constraints and complex data source integration, the team chose MongoDB for its ability to handle diverse data types, enabling them to create precise risk profiles and automate IT operations. This shift to MongoDB Atlas facilitated a transition from monolithic to microservices architecture, improving efficiency and security, especially under GDPR regulations. This technological evolution allowed Oakbrook to focus on customer experience and scalability, with a modern stack including Terraform and RabbitMQ, ultimately helping the company maintain its customer-centric approach.