June 2024 Summaries

In May, Team Weaviate held an event in New York City with over 100 AI enthusiasts attending. The event included expert talks and hands-on training for building AI applications. A panel discussion was hosted by the CTO of Weaviate, discussing leveraging AI technologies in production while maintaining data security and user trust. Instabase and Astronomer shared their experiences using AI to enhance efficiency across various industries. Key takeaways from the discussion included integration of AI for data processing, importance of trust and reliability, and adaptability and continuous improvement of AI systems.

OpenAI has released two new embedding models, text-embedding-3-small and text-embedding-3-large, which are trained using the Matryoshka Representation Learning technique. This technique allows for a trade-off between performance and cost in embeddings by shortening them while maintaining accuracy. The embeddings can be used with Weaviate vector database to store more information at lower costs and search faster.

The Weaviate TypeScript client has been released after a successful beta phase. This stable v3 client offers improved features such as faster gRPC-powered API, first-class Typescript support, and numerous quality-of-life improvements. It is now available for installation via npm or yarn. The release includes changes like updated client connection methods, multiple vectorizers to a collection definition, refreshed way of reconfiguring collections, and backward compatibility with v2 clients. Future plans include improving error messages, releasing a web version of the client, and enhancing the batching experience.

Vector embeddings are crucial in modern search and Retrieval-Augmented Generation (RAG) applications as they capture semantic meaning of data objects like text and represent them in numerical arrays. Weaviate, a vector database that stores these embeddings and retrieves data based on vector search, has numerous integrations with various model providers and their embedding models. To choose the right embedding model for your application, you need to consider several factors such as use case, baseline model selection, task-specific performance, model size and memory usage, embedding dimensions, max tokens, and evaluation of the model on your specific dataset. The Massive Text Embedding Benchmark (MTEB) Leaderboard can be a useful starting point for this process. Once you have selected an appropriate model, it's crucial to evaluate its performance on your own dataset as benchmarks may not accurately represent the data you are dealing with. You can do this by preparing a small hand-labeled dataset and using Weaviate vector database to generate and store corresponding vector embeddings. Then, run queries and compare results against desired outputs for each query. Finally, after building an initial pipeline with a small baseline model, experiment with different embedding models to see if you can improve performance beyond your initial choice. Fine-tuning the model is optional but may be necessary for last bits of performance enhancement.