August 2023 Summaries

Weaviate offers hybrid search capabilities that combine vector and keyword search results using two fusion algorithms: rankedFusion and relativeScoreFusion. The newer algorithm, relativeScoreFusion, is likely to be the better choice for most users as it retains more information from the original searches than rankedFusion. Hybrid search provides semantically rich results while respecting the precision of keyword searches. Weaviate also introduced AutoCut in v1.20, which can intelligently retrieve groups of objects from a search and works well with relativeScoreFusion.

Weaviate 1.21 introduces new features such as ContainsAny and ContainsAll operators for complex queries, multi-tenancy improvements with experimental tenant deactivation, and performance enhancements in search, indexing, and backup processes. Additionally, two new vectorizer modules are added: text2vec-gpt4all for fast transformer inference on CPUs and multi2vec-bind for multimodal data from up to 7 modalities.

Vector databases like Weaviate use machine learning models to analyze data and calculate vector embeddings, which are stored together with the data in a database for later querying. Various distance metrics can be used to judge how similar or dissimilar two objects are based on their vector values. These metrics include Cosine Similarity, Dot Product, Squared Euclidean (L2-Squared), Manhattan (L1 Norm or Taxicab Distance), and Hamming. The choice of distance metric depends on the data, model, and application being used. Weaviate supports five different distance metrics and allows users to create their own.

Vector databases are a type of database that indexes, stores, and provides access to structured or unstructured data alongside its vector embeddings. They allow for efficient similarity search and retrieval of data based on their vector distance or vector similarity at scale. Core concepts around vector databases include vector embeddings and vector search, which enable efficient vector search by leveraging approximate nearest neighbor (ANN) algorithms. Vector databases are essential for efficiently managing and searching high-dimensional vector embeddings, enabling real-time accurate similarity searches that perform a critical function in the AI native app stack.