December 2023 Summaries

In 2023, DataStax played a significant role in enabling developers to participate in the generative AI revolution by adding vector capabilities to their offerings. The company acquired Kaskada's expertise and launched Vector Search for Astra DB, which has been well-received by customers. Developers are now able to build AI-driven apps with ease of use, relevance, and fast production times. DataStax is proud to have many customers in production with interesting use cases, making the company optimistic about the future of generative AI and its potential impact on businesses.

DataStax Astra DB and Apache Cassandra have released Neighborhood Watch (nw), a configurable GPU-powered ground truth KNN dataset generator, to address limitations in existing KNN datasets. The tool is designed for generating ground truth datasets for high-dimension embeddings vectors that are more representative of what people are actually using today. It incorporates GPU acceleration and supports multiple embedding models (both open source and proprietary). Neighborhood Watch can be used to test the quality of Approximate Nearest Neighbors (ANN) by ensuring it returns a large, representative, ground truth KNN dataset.

A recent study by analyst firm GigaOm compared the performance of DataStax Astra DB vector search to Pinecone, a widely used vector database. The results showed that Astra DB performed significantly better than Pinecone in terms of throughput, latency, F1 relevancy, and total cost of ownership (TCO). These metrics are crucial for enabling accurate generative AI and retrieval augmented generation (RAG) applications in production. Throughput testing revealed that Astra DB indexes data up to 6x faster than Pinecone, while latency tests showed that during active indexing, Astra DB delivered up to 6x faster query responses versus Pinecone. F1 relevancy was also higher for Astra DB compared to Pinecone, with recall ranging from 90% to 94% during active indexing. Finally, GigaOm found that Astra DB had up to an 80% lower total cost of ownership compared to Pinecone over a three-year period.

The Astra Assistants API now supports Google's new Gemini Pro model, which is available via Vertex AI. This integration was made possible by leveraging file storage capability in the Assistant API and adding a new auth purpose. Users can now reference the uploaded file by ID in the google-application-credentials-file-id HTTP header when creating their OpenAI client. To use Vertex AI, users must also pass a vertexai-project header with their project ID and an embedding-model for file retrieval/RAG. The Astra Assistants API already supports various LLMs and providers, including Cohere, AWS Bedrock, Perplexity, Anthropic Claude, Llama, and Mistral.

SwiftieGPT is a chatbot built using DataStax Astra DB and powered by conversational AI. It was designed to answer any questions about Taylor Swift and is constructed in two parts: a data ingest script that creates the knowledge base, and a web app that provides the conversational experience. The application uses Next.js, LangChain.js, Cohere, OpenAI, and DataStax Astra DB for its development. It's deployed to Vercel and can be accessed at https://www.tswift.ai. The source code is available on Github.

Retrieval augmented generation (RAG) is an efficient deployment method for enterprises to launch large language model (LLM) applications. RAG enables AI teams to build applications on top of existing open-source LLMs or those provided by companies like OpenAI, Cohere, or Anthropic. This approach allows the introduction of time-sensitive and private information not possible with foundation models alone. DataStax Astra DB and Fiddler's AI Observability platform have partnered to enable enterprises and startups to quickly put accurate RAG applications into production. The partnership combines Astra DB's real-time vector capabilities for building generative AI applications with Fiddler's monitoring capabilities, addressing safety, accuracy, and control requirements for deploying RAG applications in production.

Recent market developments indicate a growing demand for end-to-end retrieval augmented generation (RAG) solutions, as evidenced by software vendors like DataStax introducing new offerings in this space. LangChain, an open source framework for developing applications powered by language models, has seen massive adoption among startups and large enterprises alike. Approximately 87% of DataStax's vector search production customers use LangChain. The growing popularity of LangChain is due to its ability to enable advanced RAG techniques that reduce hallucinations and leverage both structured and unstructured data, making it easier for businesses to move from prototypes to production. As the ecosystem evolves, LangChain is becoming more modular with separate packages for integrations, allowing companies like DataStax to independently add features without breaking the project. DataStax aims to contribute technical expertise to the project and help conduct benchmarking to determine which RAG techniques work best in practice.