July 2024 Summaries

With Rockset's acquisition by OpenAI, customers have until September 30, 2024, to offboard their applications and migrate to a new platform. SingleStore is a reliable real-time analytics database that offers better performance than Rockset without sacrificing connectivity or unstructured data support. SingleStore provides real-time analytics, low-latency queries, horizontal scalability, hybrid search, and support for unstructured data, making it an ideal alternative to Rockset. The company is also offering a one-time credit to customers who migrate to Google Cloud before September 30. Additionally, SingleStore integrates seamlessly with Google Cloud BigQuery for comprehensive data needs and the industry-leading AI platform Vertex AI, providing users with a powerful combination of tools to handle demanding workloads.

SingleStore offers a native API for MongoDB called SingleStore Kai that enables seamless and performant execution of aggregation queries without any application code changes or ETL data movement. This allows developers to take advantage of SingleStore's multi-model database capabilities, which support strong aggregations on JSON of any kind, including seekable JSON, sparse JSON compression, and vectorized processing for $count, $match, and $lookup operations. By using SingleStore Kai, users can see significant improvements in performance compared to MongoDB, with 142x, 82x, and 299x improvements seen for $count, $match, and $group queries respectively.

We are thrilled to announce that SingleStore Kai, a MongoDB wire-compatible API, now offers integrated vector search capabilities, enabling 100x faster analytics and empowering gen AI use cases without code modifications, as the demand for intelligent AI applications surges with advancements in vector data utilization across various domains. With this update, Kai extends its powerful capabilities to include vector search, allowing developers to store and query vector data alongside operational data, combining features like full-text search and operational analytics within a single database without additional complexity. SingleStore provides multiple indexing options, including IVF_PQFS, which offers lower memory usage and faster index build times, outperforming competitors in recent benchmarks. To use vector search through SingleStore Kai, developers can create a collection with an embedding field, insert documents with embeddings, set up a vector index, and perform vector searches using MongoDB's familiar syntax. With enterprise capabilities and integrations like workload isolation and fast ingest of vector embeddings, SingleStore is a comprehensive solution suited for enterprises, supporting different MongoDB integrations and offering a free Starter workspace to explore these capabilities.

In the modern data stack, real-time insights and seamless integration are essential for organizations aiming to stay ahead. SingleStore, a database that powers real-time intelligent applications, has added Change Data Capture (CDC) support to build such applications with enhanced agility and efficiency in operations. This feature promises to revolutionize how businesses capture and utilize data by exposing an event stream of every alteration made to data within a specified database in real-time. With CDC support, organizations can enhance decision-making, ensure data consistency, and facilitate real-time analytics. By enabling real-time data synchronization from SingleStore databases to external systems, organizations can access up-to-date data across systems for faster and more informed decision-making processes, maintain consistency between operational databases and analytics platforms, and empower analytics and business intelligence tools with live data feeds. The need for CDC is typically found in scenarios where data needs to be moved from one location to another or consumed as it's being created, making it a valuable tool for businesses looking to stay ahead in the modern data stack.

The retail business is one of the most challenging ventures in today's market landscape, requiring systems to provide real-time recommendations and instant offers to meet heightened customer expectations. Leveraging data-driven strategies is crucial for survival and growth, with Confluent Flink and SingleStore offering powerful integrations that combine real-time data streaming with robust stream processing capabilities. These solutions enable organizations to build sophisticated data pipelines, process large volumes of data with low latency, and drive innovation and competitive advantage in a data-driven world. By harnessing the power of advanced data processing technologies, retailers can unlock the full potential of their data, enhance operational performance, and stay ahead of the competition.

The text discusses the application of data analytics in motorsports, specifically in the context of collecting and processing vast amounts of real-time data from vehicles equipped with hundreds of sensors. The Gaucho Racing Formula SAE team developed a cutting-edge data acquisition system using SingleStore, which achieved an impressive average latency of 270 milliseconds from data transmission to insight delivery. The system involved collecting data from CAN bus nodes, aggregating it into byte arrays, and transmitting it to a cloud-based ingest service for processing and uploading to SingleStore. The guide walks through the process of building a similar high-performance data acquisition and analysis system using SingleStore's technology, highlighting its capability to ingest and process data even under suboptimal conditions.

Our Portal platform has recently undergone a significant UI and navigation revamp aimed at improving user experience. The initial change was met with panic from users who were unable to find certain features, such as notebooks and deployments, which had been moved to new locations on the sidebar. To mitigate this, visual cues like small icons and informative text were introduced to guide users through the changes. However, some users struggled to adapt to the new layout, exhibiting old habits that die hard, and required additional guidance. The experience highlighted the importance of accounting for extreme user reactions when updating UI, providing clear visual cues, and monitoring user behavior to inform future decisions.

The SingleStore team is launching a new navigation in the SingleStore Portal, designed to better serve their evolving business and improve user experience. The existing navigation struggled to scale with the addition of new features, and the team conducted research and gathered feedback from customers to identify pain points and redefine their core user audience. The new navigation aims to create a scalable scheme that is flexible and prepared for future product releases, while also augmenting core user workflows for admins and developers, establishing clarity on objects and their relationships, and providing a more intuitive interface. The team will launch the redesign slowly, allowing users to switch back and ease into the changes, and encourages users to rethink how they use SingleStore as a whole.

SingleStore has re-implemented its full-text search engine based on the widely used Apache Lucene (JLucene) platform, providing fuzzy search, proximity search, boosting, BM25 scoring, and support for JSON data type. The new implementation runs JLucene in a co-process next to the SingleStore service process on each node, minimizing data copying for excellent performance. Full-text search is now available over JSON fields directly, with examples provided for fuzzy search, proximity search, and boosting. Vector search enhancements include filtered ANN support and a new vector range search feature. The system supports boosting of terms, BM25 scoring, and proximity search, allowing developers to rank documents with favored terms higher. SingleStore's JSON data type has become popular since its introduction, and the new release further enhances its capabilities. The system is designed to help developers build modern, intelligent apps that combine transaction processing, analytics, search, and gen AI without the need for multiple data stores or complexity.

With the acquisition of Rockset by OpenAI, SingleStore has seen increased demand from customers who were previously using Rockset as their real-time analytics database. A company called ConveYour, which provides a platform for companies to attract, train, and retain sales teams, has migrated to SingleStore after comparing it favorably to Rockset in terms of features such as low-latency streaming writes, nested JSON support, and SQL support. ConveYour chose SingleStore over other alternatives like Apache Pinot, StarTree, Clickhouse, and CrateDB due to its complex queries performance, approachable and intuitive UX, end-to-end support, and low-latency queries capabilities, making it a great Rockset alternative for companies that need real-time analytics and operational data layer capabilities.

Databases are crucial for storing and processing data efficiently, but single-node databases like MySQL can become apparent limitations as businesses grow, particularly with larger datasets and real-time analytics. MySQL struggles with slow ingestion and analytical performance, lacks advanced features such as column-store indexes and materialized views, and can incur higher costs due to inefficient resource utilization. In contrast, SingleStore offers high throughput parallelized data ingestion and retrieval, supports both transactional and analytical workloads seamlessly, and eliminates the need for multiple databases, reducing complexity while ensuring superior performance and faster data processing. Migrating from MySQL to SingleStore can be done in five easy steps using SingleStore's native Change Data Capture (CDC), which allows for real-time data processing and effortless scalability.

This blog post explains how to set up Change Data Capture (CDC) from an AWS DynamoDB database to a SingleStore database using DynamoDB streams and a Lambda function. The process involves five steps: enabling DynamoDB streams, setting up the SingleStore database, creating a Lambda function to consume the stream and write to SingleStore, connecting the DynamoDB stream to the Lambda function, and testing out the connection from DynamoDB to SingleStore by inserting, updating, and deleting data in DynamoDB and verifying that it flows through to SingleStore. The blog post provides examples of Python scripts for each step, including generating synthetic transaction data, writing it to DynamoDB, and then reading it back into SingleStore. It also notes the importance of using persisted computed columns in SingleStore to parse out individual key-value pairs from JSON blobs.

SingleStore's Job Service allows users to schedule notebooks on a time-based schedule to repeatedly update tables with aggregated data, similar to Rockset's scheduled query lambdas. To achieve this, users create a notebook that uses Table-Valued Functions (TVFs) and executes them via SingleStore's Data API. The TVF queries the desired table, and the results are then inserted into another table. Users can schedule these notebooks using the Job Service to run at specific frequencies, such as once an hour, to maintain up-to-date aggregated data in their database. This functionality is useful for tasks like data preparation, ML flows, transforms, caching results, sending automated alerts, ingesting data from various sources, and creating and managing workspace groups.

We are thrilled to announce the latest advancements for SingleStore Kai with the 8.7 release, which has continuously evolved since its initial GA release with version 8.5, introducing powerful new capabilities designed to deliver ultra-fast performance for MongoDB applications. With this release, we've added support for building intelligent applications through vector search and increased compatibility with the MongoDB command set, further optimizing query performance. SingleStore Kai enables seamless integration of NoSQL/MongoDB apps with robust vector search capabilities through any MongoDB client or driver, allowing users to harness AI capabilities into their applications. The latest updates also make processing massive data volumes faster and more efficient, especially for transactional operations like document lookups and updates, while optimizing BSON storage type and functions as native engine functions. Additionally, SingleStore Kai's new multi-key index feature accelerates queries filtering individual array values, and our native CDC capabilities enable effortless data migration and replication from MongoDB.

SingleStore customers benefit from Helios' cloud-based managed SaaS solution that provides a distributed, scale-out architecture with decoupled storage and compute, as well as fast scaling capabilities. To address dynamic usage patterns, SingleStore has released autoscaling, a feature that automatically adjusts compute resources based on workload demand, ensuring high availability, optimal performance, and cost-effective solutions. Autoscaling allows customers to set guardrails and monitor resource utilization using Grafana, with the ability to scale up or down as needed. Additionally, SingleStore offers alternative scaling mechanisms, such as scaling storage-intensive workloads or entire workspaces through full resize operations. Customer examples demonstrate the effectiveness of autoscaling in powering SaaS applications and AI-driven demand forecast tools.

Following OpenAI's acquisition of Rockset, engineers building intelligent, generative AI applications are seeking robust, scalable alternatives. SingleStore provides a powerful option to keep your applications running and offers advanced vector operations crucial for generative AI, including indexed ANN search and k-Nearest Neighbor searches via dot product, Euclidean distance, and cosine similarity. Its scalable architecture ensures that your generative AI apps can handle growing data volumes and increasing inference requests without performance degradation. SingleStore seamlessly integrates with leading AI platforms like OpenAI, AWS Bedrock, and Hugging Face, allowing for easy development and deployment of sophisticated generative AI applications. The database supports both Inverted File (IVF) and Hierarchical Navigable Small World (HNSW) indexing for efficient ANN searches, reducing search time compared to kNN, and offers robust full-text search capabilities integrated into its database engine built on the open-source Lucene technology. Additionally, SingleStore provides performance optimization techniques such as benchmark queries and inference requests, scalability through distributed architecture, and hybrid search to enhance context retrieval. Transitioning your intelligent, generative AI applications from Rockset to SingleStore opens up new possibilities for enhancing your AI capabilities with its comprehensive AI ecosystem integration and robust features, supported by expert engineers who will guide you through the migration process.

SingleStore has introduced automatic schema and pipeline DDL inference, simplifying the initial setup process for customers to quickly leverage their data and reduce time to insights.` This automation enables developers to accelerate setup times for new databases, minimize errors, and tailor schema and pipeline setups, enhancing developer productivity. The input_configuration may be a configuration for loading from Apache Kafka, Amazon S3, a local filesystem, Microsoft Azure, HDFS, and Google Cloud Storage. Users can choose to review, modify, and apply the suggestions provided by schema inference in SingleStore. The new functionality involves simple commands that enable users to connect to data files and generate schema definitions. Schema detection is just the beginning of SingleStore's investments to developer productivity and simplifying data ingestion problems, with plans for expanded support for additional file types like JSON and Parquet.

To create parameterized queries using SingleStore's Table-Valued Functions and execute them via the Data API, first set up a SingleStore database and tables, then create a Table-Valued Function (TVF) that accepts input parameters and executes a single SELECT statement. The TVF can be accessed via the SingleStore Data API, allowing for automation of data ingestion and query execution through scripts or ETL tools. This approach enables efficient management and execution of complex queries within the SingleStore ecosystem, providing a powerful solution for data management and analysis.