January 2018 Summaries

Machine learning is a method of analyzing data using an analytical model that is built automatically from training data. The goal is to enable the algorithm to improve over time as more data points are fed into it. Machine learning has two distinct steps: training and operationalization. Training involves exploring the data set to find patterns, while operationalization involves deploying the model to a production system where it runs to score new data and returns results to users. To accomplish these steps, various tools such as data ingestion tools, data cleansing libraries, development frameworks, testing platforms, and deployment environments are used. SingleStore is a distributed database platform that excels at doing calculations typically found in machine learning models, making it an ideal environment for storing training data and operationalizing algorithms. It meets key requirements for effective operationalization, including fast calculations, scalability, compatibility with existing libraries, and powerful programming languages, allowing organizations to build efficient ML applications. SingleStore can be used in various ways, including as a fast service layer that stores raw data and serves results, pipelines that execute the ML scoring algorithm on incoming data, or even within the database itself using a language to encode the algorithm. By leveraging these approaches, organizations such as Thorn and Nyris.io have successfully implemented machine learning applications with improved performance and efficiency.

SingleStore is an adaptable database designed for real-time applications that combine transactions and analytics in a single high-performance platform. It provides fast data ingestion with low latency, serving large numbers of simultaneous users. SingleStore leverages both memory and disk for optimal performance, automatically partitions data for distributed ACID consistency, and supports various development IDEs. The database can replace traditional workloads and reduce costs by utilizing industry-standard hardware, making it suitable for petabyte-scale data querying. However, storing large images or binary objects may require object stores like S3, and the DBA must manually manage data transfer between row store and column store. SingleStore offers a free developer edition with unlimited scale, supports JDBC and ODBC connectivity, and can run on any cloud platform or on-premises.

AWS is a large and growing market for database solutions, with revenue of $18 billion and 42% annual growth. The data landscape can be simplified into four categories: Databases, Data Warehouses, NoSQL, and Data Lakes. AWS offers various solutions in each category, including Aurora, Redshift, DynamoDB, S3, Hadoop (Elastic MapReduce), and Athena. However, many of these solutions have limitations, such as slow ingest for Redshift or limited concurrency for DynamoDB. SingleStore is a new solution that combines the benefits of databases and data warehouses, providing fast ingest, high concurrency, and SQL support. It can be used when dataset size exceeds single server capacity, performance requirements outpace single server capabilities, or simultaneous read and write workloads are needed. SingleStore can also integrate with AWS services like S3 to provide a pipeline for real-time data ingestion. The combination of SingleStore and AWS provides a compelling platform for building real-time applications that combine transactional and analytical requirements.

Traditional data warehouses were designed for workloads 20 years ago, struggling to meet modern business demands of volume, velocity, and user demand. Companies invested heavily in legacy systems but now face challenges to leverage these investments for real-time insights. The "flying car dilemma" highlights the need to modify a traditional system to adapt to changing requirements, which is unlikely to work long-term. Instead of replacing or rip-and-replacing a legacy system, companies can take an approach similar to FedEx's drone delivery model, where they isolate high-demand workloads on modern real-time data warehouses and continue using legacy systems for lower-demand tasks. This pragmatic approach allows for incremental adoption and simplification over time, reducing the need for costly ETL tools and complex processes.