January 2024 Summaries

SingleStore Helios cloud has launched its public preview for an advanced self-service authentication service, providing support for OpenID Connect (OIDC) and Security Assertion Markup Language (SAML) authentication protocols. This authentication proxy adds a layer of security and flexibility for users accessing the SingleStore database, allowing customers to configure identity provider connections themselves and test before going live. The platform serves as an authentication gateway for users accessing various SingleStore database services, facilitating authentication with external Identity Providers using either OIDC or SAML. While both protocols have their advantages, OIDC offers enhanced security and ease of setup, whereas SAML provides fine-grain control over information exchange. Configuration and fine-tuning involve exchanging XML configuration blobs and configuring attributes on both the IdP and SP sides. The platform ties authentication sessions to access tokens granted by customer's IdPs, with session duration determined by the lifespan of these tokens or their ability to be refreshed.

We're building a modern real-time AI app that provides recommendations on which Large Language Models (LLMs) are most suitable for specific use cases based on the latest data, sentiment, and scores. The app uses SingleStore's Notebooks and Job Service to bring in real-time data from various sources, enrich it with vector embeddings, and ensure it's queryable by downstream LLMs and applications. We've used Twitter, Reddit, and GitHub APIs to gather raw data for each model, and then searched for this data using the SingleStore Elegance SDK and Vercel. The app queries this data using RAG (Real-Time RAG) and provides recommendations based on user input. This demonstrates how a single developer platform can host hybrid free database + cloud services, making it easy to build modern AI apps with real-time capabilities.

The World Economic Forum's Annual Meeting in Davos, Switzerland, saw SingleStore CEO Raj Verma lead the company's delegation for the second year. At the event, Verma shared his insights on Artificial Intelligence (AI) and its potential to transform humanity. He emphasized the importance of using AI responsibly, prioritizing data privacy, reducing biases, and adopting a transparency framework to mitigate risks. Verma also introduced his book, "Time is Now: A Journey into Demystifying AI," and hosted an event called "A Journey Into AI" to encourage relaxed conversations about AI's potential benefits and limitations. The experience reinforced Verma's conviction that data companies will play a critical role in developing AI technologies, and he expressed SingleStore's commitment to delivering accurate data in real-time to support businesses and organizations in making consequential decisions.

SingleStore has shipped a public preview of indexed approximate-nearest-neighbor (ANN) search and support for a native vector data type, making it a modern SQL database with vector search capabilities. This allows developers to build transactional, analytical, full-text, and vector search applications on one platform, saving implementation time, data transfer time, code, and money compared to using specialized vector databases. The new features include indexed ANN search, which provides orders-of-magnitude faster vector search, and a native vector data type, making it easier for developers to build vector-based applications. With the addition of these features, SingleStore is now a leader in the market for modern SQL databases with built-in vector search capabilities.

SingleStore Pro Max is the ultimate real-time data platform, built for all data, applications, and data professionals. It offers faster hybrid vector search, native data integration services, and a new Free Shared Tier to empower startups and developers everywhere. The platform provides a unified architecture with SingleStore's high-performance SQL engine, Hybrid Search, cloud services like SingleStore Helios, Kai API, Native Data Integration Services, Compute Service, and more. These innovations enable power with simplicity, real-time data processing, and scalability for enterprise workloads, generative AI, transactional applications, petabyte-scale analytics, front-office analytical applications, and all kinds of data formats. The platform is designed to serve all makers and data professionals, from developers to data scientists and end-users interacting with data-driven systems.

We're excited to introduce SingleStore Job Service, a native job scheduler that simplifies the process of operationalizing Python and SQL code with data stored in SingleStore, empowering developers to build apps and logic with seamless data access, development, and production capabilities. With its familiar interface, Job Service ensures robust security and governance while streamlining compute management, making it an ideal solution for modern enterprises. Our goal is to bridge the gap between prototype and production apps, enabling developers to share insights, perform ETL, and build real-time embeddings with ease. We're now rolling out a preview of Job Service, inviting users to try it out and provide feedback, as we continue to innovate and improve this exciting feature.

SingleStore Kai™ is an API designed to enhance analytics on MongoDB® applications without requiring code changes or data transformations. It allows developers to utilize both MongoDB® and SQL APIs on SingleStore, providing a unified solution for real-time analytics that eliminates the need for multiple data stores. The API supports seamless integration with MongoDB® through Change Data Capture (CDC), enabling data replication without complex tooling or schema definitions. SingleStore Kai leverages a Universal Storage system and a massively parallel processing (MPP) database engine to handle both transactional and analytical workloads, offering significant performance improvements. With its native support for BSON, SingleStore ensures full compatibility with MongoDB® document types, and it provides an interactive terminal via the cloud portal to enhance developer productivity. The platform allows MongoDB® applications to harness the power of SQL for advanced queries, integrate with relational tables, and perform high-speed analytics, making it an attractive option for applications needing fast JSON analytics and scalability. As demonstrated by CanPR, a popular immigration app, transitioning to SingleStore Kai resulted in increased query speeds and eliminated performance bottlenecks, showcasing the API's potential to transform application experiences.

A robust data integration service is essential for efficient data movement both into and out of the platform, enabling real-time updates and seamless integration of diverse data sources while supporting the extraction, transformation and loading (ETL) of data for downstream consumers. Data integration poses several challenges, including handling diverse data formats and sources, meeting real-time processing demands, scalability issues, data quality and governance concerns, security and compliance requirements, complexity in hybrid environments, interoperability between various data platforms and applications, and costs associated with data integration tools. To address these challenges, SingleStore is announcing a native suite of data integration services designed to ETL/CDC data between SingleStore and various sources/targets, including Change Data Capture (CDC) capabilities that support seamless integration with MongoDB, MySQL, OLAP systems, and other databases like PostgreSQL in the future. These native capabilities aim to provide continuous, automated, and cost-effective integration solutions, eliminating much of the costs, complexity and burden of data pipelines, making zero ETL a reality.

The 54th World Economic Forum meeting in Davos-Klosters focused on the theme of artificial intelligence as a driving force for the economy and society. The forum aimed to address questions such as how AI can be used for the benefit of all, how regulatory landscapes are balancing innovation with societal risks, and how AI will interact with other transformative technologies like 5G, quantum computing, and biotechnology. SingleStore CEO Raj Verma discussed the importance of democratizing access to data and AI, emphasizing that information and context are crucial for making informed decisions. He highlighted the challenges presented by generative AI and the need to ensure users have access to accurate data and context. Verma also expressed his optimism about the future of AI, predicting it will be a transformative technological evolution with the potential to change civilization for the better, powered by data and context.

Thoughtspot's SingleStore connector is now generally available, allowing customers to tap into the power of real-time data and actionable insights. This partnership enables seamless access to real-time data, enabling businesses to make informed decisions. SingleStore offers lightning-fast data ingestion and querying, while ThoughtSpot provides an intuitive interface for users to engage with live data. The combined strength of these tools can be harnessed across various industry-specific use cases, including financial analytics, customer analytics, and predictive maintenance. Getting started is effortless, with a 30-day free trial available through the ThoughtSpot website, where SingleStore can be easily connected as a data source to begin visualizing data promptly.

SingleStore 8.5 introduces a new VECTOR data type designed to simplify building generative AI and RAG-based applications. The vector data type allows for easier insertion, loading, and querying of vector data with improved data validation and automatic conversion between JSON arrays and packed vectors. Infix operators have been added for DOT_PRODUCT and EUCLIDEAN_DISTANCE, making it cleaner to query vector similarity. Casting between BLOB and VECTOR has also been implemented for ease of use. The new type is designed to work seamlessly with existing SQL queries and applications, providing a modern distributed SQL DBMS platform for building vector-oriented AI applications.

SingleStore's query event tracing is a powerful tool that enables users to capture, understand, and optimize real-time applications by extracting query workloads as they occur with full query text, execution time, query owner, and database information in real-time without impacting performance. The feature allows developers to enable and configure an event trace to analyze query execution details and visualize performance profiles, helping them diagnose and address unexpected cluster slowdowns, optimize query execution, and improve resource utilization. By providing a valuable asset for organizations seeking precision and agility in managing their real-time workloads, SingleStore's query history empowers developers to proactively detect and resolve issues, leading to optimized query execution and resource utilization.

The SingleStore Management API provides a suite of Python functions that simplify various tasks such as creating, resuming, terminating, and resizing workspaces, as well as uploading files to staging areas from S3. These functions are designed to be user-friendly, timesaving, and customizable, allowing developers to automate repetitive tasks and focus on more important aspects of their work. The API offers robust capabilities for managing database environments, including file management tasks such as local and S3 file uploads, and provides easy access to billing information. By leveraging these Python functions, developers can streamline workspace operations and enhance their productivity in managing their database environment.

SingleStore Helios Cloud now offers a robust security solution through Role-Based Access Control (RBAC), which simplifies fine-grained access control and reduces the risk of human error in managing access rights. RBAC allows administrators to define and manage access at a granular level, ensuring users have precisely the permissions they need to perform their tasks. The solution includes predefined roles, teams, integration with database RBAC, and role inheritance, which simplifies the administration of roles across multiple resources. By implementing RBAC, organizations can ensure users have the right level of access, minimize security breaches, and enhance the overall security of their database.

Today, SingleStore is launching its Free Shared Tier on the cloud, making its powerful engine available to everyone, everywhere. This multi-model HTAP database powers some of the world's leading companies and applications with its versatility to handle transactional workloads, demanding analytical queries, and hybrid search. The new offering democratizes access to this powerful engine for all apps and ideas, providing a free starter workspace with limited storage and access to Pipelines, UDFs, and Stored Procedures. SingleStore has maintained its performance at any scale, ingesting real-time streaming data, running large aggregate queries, and performing vector search over millions of embeddings. The Free Shared Tier offers shared compute resources, full security, and isolation of data and workloads. SingleStore is excited to see what the community builds with this new offering, which can be tried out through a preview, and is investing heavily to bring its platform to everyone for their applications, regardless of scale.

The SingleStore Helios cloud service has been developing its new Shared Tier for cost efficiency and fast deployment. The goal was to keep hundreds of users in the same underlying cluster while supporting existing APIs, improving free trials and developer experience, and increasing exposure to those who find the normal tier too expensive. To achieve this, a custom shared MySQL proxy was built on top of SingleStore's architecture, which detaches idle databases for cost efficiency and uses a proxy to automatically attach them. The proxy handles packet parsing, resource creation, and routing information, simulating interactions with existing database features. Despite some challenges, such as identifying a bug related to the MySQL protocol, the new tier is now available in preview, and design partners are encouraged to provide feedback.

Structured data is organized in a format that is easily understandable by both humans and machines, making it efficient for storage, retrieval, and analysis. It typically appears in relational databases, spreadsheets, and other tabular formats with well-defined schema or models governing relationships and properties of elements. In contrast, semi-structured data has some structure but leaves room for non-structured data, often appearing in formats like XML, JSON, Avro, Parquet, HTML code, and emails. Unstructured data lacks organization, including text files, PDFs, images, video, and audio files, which can be stored as Binary Large Objects (BLOBs) or in file systems/object storages with metadata and vector embeddings associated with them. SingleStore supports processing structured, semi-structured, and unstructured data using SQL queries, JSON columns, and external machine learning models for applications like customer information, JSON datasets, and AI-driven use cases.

AI is rapidly becoming more accessible and exciting for people to use, with 2023 being a breakout year for generative AI. As we move into 2024, we can expect to see significant advancements in AI technology, including increased productivity, real-time data processing, and the adoption of multiple Large Language Models (LLMs). Hybrid human-AI models will also become more prevalent, allowing humans to guide AI systems and mitigate bias. These developments aim to leverage AI to enhance human productivity and improve lives, while ensuring responsible use of this powerful technology.

The SingleStore (née MemSQL) team has had an incredible year, shipping numerous improvements and features to their cloud-first database as a service. They've expanded monitoring and alerting capabilities, added a new API called Singlestore Kai that allows customers to accelerate queries on MongoDB data with no changes to their data or application code. The team has also made significant investments in developer experience, introducing features like Notebooks, which enable users to write custom Python code securely connect to their database and collaborate on interactive documents. Additionally, they've improved the cloud portal UI for faster performance, introduced automated rollouts and deployment rings, and enhanced manageability and operations with features like Auto suspend, Resize progress, and forwarding audit logs. The team has also focused on security, introducing role-based access control, improved SSO capabilities, and a modernized billing platform. Overall, the SingleStore Cloud platform has matured significantly, becoming more stable and user-friendly, with plans for even further evolution in the coming year.