December 2022 Summaries

This article discusses the process of importing a large dataset into Weaviate using Apache Spark. The author provides detailed information on the hardware and software setup used for this task, including the use of Google Kubernetes Engine (GKE) nodes and the text2vec-huggingface vectorizer module in Weaviate. The article also covers performance metrics during the import process, such as batch duration and LSM store size. Additionally, it mentions future developments to improve memory usage at scale, including Vamana and HNSW+PQ technologies.

Weaviate 1.17 introduces replication for improved database resilience and performance, hybrid search combining dense and sparse vectors, BM25 for keyword-based search ranking, faster startup times and more efficient imports, as well as various other improvements and bug fixes. Replication enables multi-node configuration, improving availability, throughput, and enabling zero downtime upgrades. Hybrid search combines the strengths of sparse and dense vectors to deliver better search results. The BM25 operator performs case-insensitive keyword search with a ranking function based on term frequency and inverse document frequency. Faster startup times are achieved by persisting computations on disk, adjusting memtable size dynamically, and improving batch import latency.

Natural Language Processing (NLP) has significantly impacted how humans interact with computers, largely due to advancements in Large Language Models (LLMs). Cohere is an AI platform that provides developers and businesses access to its LLMs for NLP implementation. In combination with a vector database like Weaviate, complex NLP queries can be executed at large scale on millions of text objects, returning results quickly. A new Weaviate module called text2vec-cohere allows users to directly run complex NLP queries using Cohere's LLMs from within Weaviate. This integration enables efficient search across documents written in different languages and supports continuous data modification while maintaining operational efficiency.

Meta has released an open-source dataset called Sphere, which consists of 134 million documents broken up into 906 million 100-word snippets. It is one of the largest knowledge bases that can help solve knowledge-intensive natural language tasks such as question-answering and fact-checking. The dataset aims to act as a "universal, uncurated and unstructured source of knowledge." However, accessing and using Sphere in its current open-source format is challenging for the average developer due to its enormity. To make this resource more accessible, Weaviate now offers Sphere as JSON or Parquet files that can be easily imported with Python and Spark.

Vector Search uses vector embeddings generated by Machine Learning models to represent data objects such as text, images, or audio. Both Vector Databases and Libraries use Approximate Nearest Neighbor (ANN) algorithm for efficient search through vectors. Key differences between them include the ability of databases to store and update data, support CRUD operations, real-time search, and additional features like sharding, replication, automated backups, deployment ecosystems, SDKs, execution methods, communication with apps, multi-tenancy, arbitrary aggregations, and module ecosystem. Vector Libraries are commonly used for applications that do not have changing data while vector databases are great to use for applications where data is constantly changing.