June 2024 Summaries

This article highlights the challenges and benefits of buying a real-time analytics database, drawing from five years of experience at SingleStore, where over 500 database proof-of-concepts (POCs) were conducted to evaluate customer needs. The key components that customers consider when evaluating databases include data size, data ingestion, query execution, query complexity, and concurrency. Several case studies are presented, showcasing how SingleStore met or exceeded the requirements of various companies, including IEX Cloud, Heap, Adobe, and GoGuardian, with notable successes in areas such as performance on JSON data, scalability for large datasets, and handling complex queries and concurrency demands. The article concludes that if a customer needs three or more of these criteria, they are likely looking for the right database solution.

SAS has revolutionized model performance by collaborating with SingleStore, boosting model performance by 50% while nearly halving costs. The company's innovative solution seamlessly fuses data infrastructure with model computation, optimizing deployment for efficiency in generating outcomes and intelligence. This effort is validated by OpenAI's acquisition of Rockset, a strategic move recognizing the critical role of real-time analytics databases. SAS foresaw this shift three years ago, aligning with SingleStore to pioneer an integrated solution that stores data in a secure, high-performance framework, minimizing data movement and complexities at scale. The integration promises an exciting future, setting a new standard for enterprise ML, AI, and intelligent applications.

This technical blog provides a step-by-step walkthrough on how to integrate DynamoDB with SingleStore using Apache Kafka and SingleStore Pipelines. The integration combines the fast, NoSQL OLTP layer of DynamoDB with the scalable, multimodel OLAP layer of SingleStore, allowing for seamless streaming and batch loading of data from DynamoDB into SingleStore tables. The blog emphasizes the simplicity and power of SingleStore's real-time data pipelines, providing an ideal solution for customers migrating from Rockset to SingleStore. Key steps include setting up Kafka topics, streaming data from DynamoDB to Kafka, creating pipelines in SingleStore to ingest data from Kafka topics, and bulk loading remaining data from Rockset into SingleStore using SingleStore Pipelines. The blog also highlights alternative methods for replicating DynamoDB data to SingleStore, including AWS DMS and SQS/Lambda functions.

SingleStore has made significant advancements in its capabilities, expanding beyond data management to a versatile real-time data platform with features such as ANN vector search, native data integration services, and a compute service for AI and data prep workloads. The company is now offering four new platform capabilities: bi-directional integration for Apache Iceberg, faster vector search and enhanced full-text search, Autoscaling for dynamic scaling of applications, and the ability to deploy SingleStore Helios in your own VPC. These enhancements aim to simplify application development, improve performance, and provide more control over database resources, making it easier for developers to build intelligent applications.

The era of open data lakehouses has begun, with Apache Iceberg emerging as the de facto standard. SingleStore is now integrating with Apache Iceberg, offering a zero ETL solution that enables seamless ingestion and querying of vast amounts of structured and unstructured data in real-time, without requiring additional tooling or manual intervention. This integration provides automatic schema consistency, performance and scalability, bi-directional integration, and data discovery and metadata management capabilities, making it easier to power low-latency apps and analytics on Snowflake, Cloudera, or any open data lake. The solution also enables external Iceberg tables for direct query access, egress data to Iceberg, and a low-latency operational engine, fast ingest from Iceberg data lakes, and bi-directional data flow, simplifying the architecture and enhancing data management and interoperability.

The acquisition of Rockset by OpenAI has led to a shift in the market, forcing developers to reassess their analytics applications and consider alternative solutions. SingleStore is positioning itself as a rich alternative to Rockset, offering improved performance, scalability, and ease of use. With its patented Universal Storage, SingleStore provides a flexible and efficient way to manage structured and unstructured data, eliminating the need for complex indexing capabilities and reducing storage overhead. Additionally, SingleStore's Pipelines feature enables fast and parallel ingest from various databases, making it an attractive option for developers looking to power their real-time analytics applications.

OpenAI has acquired Rockset, a real-time general-purpose SQL database, marking a significant shift in the company's focus towards developing an enterprise-ready stack for AI applications. This move bridges the gap between model providers and enterprises' data estates, addressing the need for a repeatable stack to integrate structured and unstructured data across operational and analytical stores. The acquisition suggests that OpenAI is prioritizing a real-time database over vector databases, which have become synonymous with building AI apps. Instead, the company is likely to develop a platform that can utilize cheap and efficient compute from a database to offload expensive and slow compute of AI models, enabling more robust and scalable AI applications. Ultimately, this partnership aims to enable new workflows in enterprise data, where humans and AI collaborate to generate hypotheses and proposals, rather than simply replacing human DBAs with AI alone.

SingleStore is a real-time data platform that provides powerful operational and transactional capabilities for intelligent applications and analytics, with its patented Universal Storage engine offering ultra-low latency and high concurrency for real-time ingestion and querying of data. The platform's hybrid transactional-analytical processing (HTAP) capabilities enable seamless integration of transactional and analytical workloads, allowing for faster and more efficient data processing. SingleStore's lack of foreign keys in DDL is mitigated by the use of filters in insert...select statements and pipelines, which can be used to enforce referential integrity and uniqueness checks with inserts or stored procedures. The platform's scalability and performance make it well-suited for applications that require sub-second latency and high concurrency, such as those involving real-time streaming analytics and AI workloads.

A modern SQL database with vector search capabilities is the best platform for building intelligent applications that incorporate generative AI enhancements such as summarization of content and semantic search. These applications are already built on top of existing SQL databases, which provides a natural fit for integrating gen AI capabilities. The data is already in a relational format, making it easier to manage and integrate with other features like transactions, high availability, and disaster recovery. SingleStore, an emerging platform, has made significant progress in vector search capabilities and is well-positioned to support gen AI applications, offering improved performance, scalability, and cost-effectiveness compared to specialized vector databases.