August 2019 Summaries

SingleStore is a relational database system designed to handle large datasets and provide low-latency access to data, making it suitable for real-time AI applications. It allows users to store and retrieve data efficiently, with features such as instant queries and high-performance computing. SingleStore integrates well with popular machine learning frameworks like Spark and Python packages, enabling users to build and deploy models quickly. The system is designed to be scalable and can handle large amounts of data, making it suitable for applications such as fraud detection, financial analysis, and image recognition. With the ability to push TensorFlow and PyTorch models directly into SingleStore, users can run these models in production, delivering great user experiences by building smart apps.

SingleStore is a database that supports in-memory rowstore tables and disk-backed columnstore tables, with features such as SQL support and MySQL wire protocol compatibility. The database's primary feature is the in-memory rowstore, which allows it to circumvent issues with disk-based databases. SingleStore also has columnstore tables with unique features, and its architecture is designed for scalability and low latency. The database is designed for companies with large datasets and fast query requirements, such as those with shifting data or historical data sets. Its cluster consists of two types of nodes: leaves that store data and aggregators that coordinate data manipulation language (DML). The master aggregator coordinates DDL and is responsible for handling failures and implementing a two-phase commit mechanism. SingleStore's interface is MySQL compatible, and it uses code generation to optimize queries and reduce runtime costs. The database also collects metadata on various columns and tables, which allows it to make efficient decisions at runtime.

A new report from RT Insights highlights the importance of real-time transaction processing in banking and financial services, where traditional database architectures hinder the movement of data. To achieve benefits such as improved portfolio management, fast credit card fraud checks, and others, banks need to use a translytical database that combines transactional and analytical data processing capabilities in a single system. A translytical database is designed to be both fast and scalable, eliminating intermediate steps and providing real-time support for critical applications. By adopting real-time databases, organizations can improve customer experience, better manage risk, and create a "data culture" across all functions and levels. The report emphasizes the need for banks to move away from traditional data structures and adopt a more modern approach to data management.

SingleStore is a database platform used by financial services institutions to power real-time analytics and fraud detection applications. A major US bank created a new streaming data architecture with SingleStore at its core, enabling them to move from batch fraud detection to real-time detection using machine learning models. This case study presents a reference architecture that can be used for similar use cases in financial services and beyond, highlighting the benefits of SingleStore's high-performing data platform, speed, scale, and SQL capabilities. The platform allows customers to add new features to their model scores using standard SQL, enabling agile updates without re-engineering or lengthy change management processes. This results in significant cost savings and improved customer experience, with potential tens of millions of dollars in lost fraud events avoided. SingleStore's distributed scale-out architecture, lock-free ingestion technology, and flexible data types make it an attractive option for financial services institutions looking to power their real-time analytics and fraud detection applications.

SingleStore is a cloud-native operational database designed to power portfolio analytics and other business-critical applications in financial services institutions. It combines the requirements of online transactional processing (OLTP) and online analytical processing (OLAP) into a single workflow, known as operational analytics. This allows for faster data-driven actions, interactive experiences with live market data, and the ability to apply historical data to real-time market trends. SingleStore supports large numbers of simultaneous users, scalable storage, and high availability, making it suitable for financial services companies looking to modernize their portfolio analytics and reduce risk. The database also provides features such as MVCC (multi-version concurrency control) and skiplists, which enable fast ingest, parallel stream processing, and lock-free semantics. With a focus on speed, scale, and SQL, SingleStore is designed to meet the requirements of financial services companies, including trade analytics, risk management, and fraud detection. The company also offers a managed service and plans to expand its offerings in the future, including Universal Storage and automated management capabilities.

This case study highlights a major financial services company's successful replacement of Oracle Exadata with SingleStore to power portfolio analytics, resulting in significantly improved responsiveness and the ability to easily incorporate machine learning models into their applications. The company replaced its legacy database technologies with a combination of Kafka, Spark, and SingleStore, enabling them to achieve real-time data processing, reduce costs by 3x, and improve performance. This new architecture allows for continuous development and evolution of machine learning algorithms, making the organization more agile and better equipped to handle changing market conditions.

The author of the blog post is upgrading their SingleStore cluster and replicating data from PostgreSQL to achieve high performance for analytical queries while keeping transactional data in PostgreSQL. They have a 1TB cluster, which is not justified, but they need the performance of SingleStore's columnstore for queries. The author uses a replication service to replicate rows from PostgreSQL to SingleStore's rowstore in real-time, allowing them to use the columnstore for analytics and speed up queries. However, updating rows is slow due to the way data is stored in a columnstore. The author tries different solutions, including using ActiveRecord callbacks and replicating the whole table from PostgreSQL once in a while, but these methods are not efficient. They eventually compress the file before loading it into SingleStore, which speeds up the process and makes storage less of an issue. The author creates a Rails module to replicate any query from PostgreSQL to SingleStore easily, including truncating old data and minimizing downtime during replication.

SingleStore's real-time fraud detection capabilities enable banks to deliver a better customer experience while protecting their assets. The company's distributed scale-out system and innovative lock-free architecture support fast analytics, making it an attractive solution for financial institutions. SingleStore's ability to deliver speed, scale, and SQL all in one package sets it apart from other data warehouses, including Cassandra and Hadoop. By integrating with Kafka, customers can leverage high throughput and exactly-once semantics, while also using stored procedure logic to score streams. The company's software is designed for flexibility, allowing customers to deploy it on any cloud or on-premises infrastructure, making it a top choice for modern cloud-based platforms. With SingleStore, banks can move from legacy architectures to the cloud in a flexible and adaptable way, enabling continuous data-driven actions and decision-making.

Web Workers are a powerful feature of JavaScript that enable parallel execution of code in the browser, allowing for expensive computations to be performed without blocking the main thread. A fully client-side web application can leverage Web Workers with React and Redux to create a seamless user experience. The key to integrating Web Workers with React and Redux is to use a custom API, middleware, and observables to communicate between the worker thread and the main thread. This allows for efficient data exchange and population of the Redux state from the worker thread.

The financial services industry is highly competitive, requiring high concurrency, low latency, and access to vast amounts of current and historical data in real-time. To address this, banks and other institutions have used in-memory databases and streaming data, such as SingleStore, which improves the wealth management experience for users and enables institutions to stand out in a crowded market. By replacing traditional Hadoop/HDFS data stores with SingleStore, banks can deliver more responsive updates to clients, support tens of thousands of simultaneous users, analyze five times more historical data, and avoid processing delays that occur when news events drive sudden surges in usage. Digital transformation is critical for wealth management, with 80% of retail bank profits generated by high net worth individuals, and SingleStore's solution enables banks to speed up the delivery of data to their wealth management dashboards, reducing event-to-insight latency and providing a seamless user experience.

Real-time analytics is necessary to enable fast decision making and deliver enhanced customer experiences. To build a real-time application, organizations need to rapidly ingest, transform, store, and make data easily accessible at sub-second speed. SingleStore is a database that supports real-time analytics as a real-time database, scalable, and uses the proven SQL syntax. It has been ranked as the number one database for operational analytics. The mission of SingleStore is to enable every company to build a real-time data pipeline and have real-time analytics on it. A typical real-time data pipeline includes ingesting application data through a distributed messaging system, transforming information, storing data in an operational data warehouse, and querying with SQL to power real-time BI dashboards. To create real-time analytics, organizations need a real-time backing store that speaks the same language as business intelligence tools, which is usually SQL. SingleStore meets these requirements, providing a scalable, SQL database that supports in-memory and solid state storage, distributed architecture, data source connectivity, and flexible deployment. The company uses Pinterest's and an energy company's use cases to demonstrate its capabilities, including serving gigabytes per second of ingest, performing ad hoc queries, and providing real-time analytics from streaming data. SingleStore also offers a graphical dashboard for looking at the SingleStore clusters, which can be used to view insights based on real-time data coming into the pipeline.