June 2023 Summaries

Stage is a feature in Singlestore Helios that enables users to load files into a staging area attached to every Workspace Group, simplifying the process of getting data into the database. With Stage, loading local files can be done quickly and efficiently, and it also supports schema inference, which speeds up the creation of tables and pipeline mapping. Additionally, users can export query results from SinglestoreDB into Stage, allowing them to share data with other users or download large result files locally. The feature is designed to help users quickly get their data into SinglestoreDB, making it easier to work with data in the database.

Vector databases are designed for efficient storage, retrieval, and similarity search of high-dimensional vector data. They use a process called embedding to represent vector data in a continuous and meaningful high-dimensional vector space. This allows for efficient similarity searches, including cosine similarity or Euclidean distance, and can be used in applications such as generative AI, image classification, recommendation engines, and natural language processing. Vector databases offer advantages like efficient similarity search, scalability, support for high-dimensional data, and native support for vector operations, but also have limitations such as limited SQL support, no full CRUD capabilities, indexing being time-consuming, and questionable enterprise features. Pure vector databases are designed specifically for storing and retrieving vectors, while full-text search databases, vector libraries, and vector-capable NoSQL databases offer alternative approaches to handling vector data. SingleStoreDB is a robust and full-context vector database that allows users to store and query vector data alongside traditional structured data, providing a unified platform for various types of queries and analysis. It offers benefits like simplicity, lower costs, and the ability to mix and match metadata, SQL, and JSON, making it suitable for enterprise gen AI use cases.

This company, SingleStore (formerly memSQL), was founded in 2011 as an on-prem database provider but has shifted its focus to a cloud-first approach, offering Singlestore Helios as a managed service on major clouds including AWS, GCP, and Azure. By leveraging AWS's support, SingleStore can help customers avoid administrative tasks, focusing instead on developing their applications. The collaboration between SingleStore and AWS enables the company to accelerate its business, strengthen its relationship with AWS, and provide customers with a range of choices in terms of database services. Additionally, this partnership facilitates product-led growth through free trials and proof-of-concepts, reduces acquisition costs, and offers financial resources for marketing efforts. Furthermore, SingleStore benefits from innovation credits, integrations with AWS services, and participation in the Workload Migration Program to support customers migrating to its database on AWS. The relationship also enables co-selling opportunities, transparent sharing of sales data, and access to AWS subject-matter-experts for customer success management.

Oracle GoldenGate now offers support for SingleStoreDB, enabling seamless integration and real-time data replication. SingleStore is now a provider on the Terraform Registry. The partnership between SingleStore and Qlik enhances SAP data migration, combining real-time data extraction and processing for improved insights. SingleStore released its JDBC connector on Tableau Exchange, allowing businesses to quickly start using the solution and improve their performance and decision-making capabilities. SingleStore has been announced as a launch partner in Informatica's ISV Partner Program, and undertook a TPC-H benchmark with Intel and AWS, showing up to 19% performance improvement from Ice Lake to Sapphire Rapids. This partnership aims to help organizations capture, process and analyze large volumes of SAP data in real time, leading to timely and accurate business insights.

SingleStore Kai is a MongoDB-compatible API that introduces 100x faster analytics on JSON data and adds vector embeddings for JSON data, empowering users to easily migrate to a more feature-rich and performant database without changing their existing MongoDB queries or API calls. This feature enables the use of semantic search capabilities such as dot_product to perform powerful searches on JSON data, useful for building accurate product recommender systems, AI-powered chatbots, and improving customer support. With SingleStore Kai, users can utilize these features on their JSON data through a simple migration from MongoDB, making it an attractive option for those looking to upgrade their database without significant changes.

The migration from RDS MySQL to SingleStore Helios was successful, completing in under nine hours, despite the application downtime requirements of less than 24 hours. The company MarTech had strict migration requirements due to its highly OLTP-based workload, and they chose a weekend with lower-than-normal app traffic to minimize downtime. A combination of in-house features and third-party tools, such as Arcion, was used to streamline the migration process. The migration involved transferring data from four RDS instances into SingleStoreDB Cloud, utilizing a combination of columnstore tables and rowstore tables for optimal performance. The cluster was scaled up to S-256 to accommodate the parallel replication process, reducing overall migration time. After validation, the SingleStoreDB cluster was scaled down to its original size, and scripts were run to sync auto-increment values and drop replicate-created tables. The results showed a 95,000% performance improvement in single query execution and a 15x total performance improvement at peak application concurrency. Additionally, there were significant cost of ownership savings.

The integration of SingleStoreDB and Redpanda enables the creation of a modern clickstream analytics system that can efficiently process and analyze large amounts of real-time data. To build such a system, one must first set up a Redpanda cluster to receive clickstream data from various sources, which is then ingested into a SingleStoreDB cluster running on AWS. This integration allows for fast ingestion and processing of data, providing low-latency access to large datasets and enabling businesses to make informed decisions based on real-time insights. The combination of SingleStoreDB's native Pipelines feature and Redpanda's API compatibility with Kafka provides a seamless solution for customers building analytical applications, making it easier to manage and scale the system.

Full-text search primarily relies on keyword matching to retrieve documents or web pages containing specific keywords. In contrast, semantic search aims to understand the user's intent behind queries by analyzing context, relationships between words or concepts, and overall meaning. While full-text search typically treats each keyword independently, semantic search considers these factors to provide more relevant results. The article debunks common misconceptions about the superiority of one method over the other, highlighting that both can be effective depending on the use case. It also emphasizes the importance of selecting the right approach based on the dataset size, query complexity, and desired outcome. For large datasets, full-text search is often a reliable and efficient option, but for more complex queries or keyword searches with nuances, semantic search may be necessary. The article concludes by suggesting that a combination of both methods could potentially offer the best results, although this has not been implemented in mature distributed database systems yet.

SingleStoreDB allows developers to create external functions that can interact with external services, such as OpenAI's vector embedding API. These external functions can be used to get vector embeddings for text data directly from SQL queries, simplifying programming and reducing the need to round-trip data through an application. An example of how this works is shown in the article, where a Flask-based web application hosts an external function that calls OpenAI's vector embedding API to generate embeddings for given text strings. The generated embeddings can then be used to update a vector column in a SQL table in a single UPDATE statement. This approach allows developers to easily incorporate vector data into their applications and take advantage of the power of SQL.

The results of a TPC-H benchmark conducted by SingleStore on Amazon Web Services (AWS) showed a significant up to 19% performance boost from Ice Lake to Sapphire Rapids Intel Xeon processors, with the latest generation delivering substantial improvements in processing power and efficiency for data-intensive workloads. The benchmark, which simulated complex business-oriented queries, highlighted the promising potential of Sapphire Rapids for real-time applications and decision support systems. SingleStore's collaboration with Intel and AWS enabled the evaluation of this cutting-edge technology, demonstrating its capabilities and potential for performance optimization.

SAP is a leading enterprise resource planning system used by businesses worldwide, and it's becoming increasingly difficult to manage and analyze SAP data effectively as systems become more complex. SingleStore and Qlik Replicate are two technologies that can help overcome these challenges, offering fast and scalable database solutions and real-time change data capture and data replication capabilities respectively. When combined, they provide a complete solution for managing and analyzing SAP data, enabling businesses to get the most out of their SAP data. The integration of SingleStore and Qlik Replicate offers several key benefits, including improved performance, scalability and flexibility, real-time insights through data streaming, ease of use and reduced complexity, and robust security and compliance. By using these technologies together, organizations can capture, process, and analyze large volumes of SAP data in real-time, making informed decisions that can drive business growth and success.

We're excited to announce the general availability of SingleStoreDB release 8.1, featuring 100x faster analytics for MongoDB apps, six-times faster transactions for wide tables, and power-packed features across observability, scalability, performance, security, and more. This release includes real-time analytics, cloud-native capabilities, distributed SQL, developer experience enhancements, and new partnerships with Oracle GoldenGate and Tableau. SingleStoreDB's AI assistant, SQrL, provides immediate responses to product-related questions, while its Notebook extends Jupyter Notebooks capabilities for data professionals. The release also introduces column groups to speed up row retrieval in wide tables and improves security with OpenSSL 3.0 support and Private Link Automation.

The SingleStore database has improved its support for JSON data, using a columnar format called Parquet2 to store and query JSON documents efficiently. This enables the use of built-in functions such as JSON_EXTRACT, which can extract values from JSON documents at specific keypaths without having to read the entire document. The JSON_MATCH_ANY function allows filtering on properties within JSON arrays, while the JSON_INCLUDE_MASK function constructs a sub-JSON containing a subset of keys. These optimizations are made possible by the design of SingleStore-encoded Parquet, which achieves high compression and efficient data access through techniques such as striping and bit-packing. Performance tests demonstrate significant reductions in query latency for queries that would otherwise require reading and reconstructing entire JSON documents.