December 2024 Summaries

During a recent event in London, the Weaviate team shared insights and real-world applications of AI, highlighting its transformative impact across various sectors. Key discussions included Etienne Dilocker's exploration of vector databases for e-commerce and email processing, emphasizing that solutions should be tailored to specific data and problems. Andy Hoare from OLIVER demonstrated their AI Sandbox, which revolutionizes client-agency collaboration and content transparency. In the nuclear sector, Willem van Asperen and Derek Van Gelderen showcased AI innovations at Sellafield Limited, such as SLComply.ai and DANI, which significantly enhance compliance and document processing efficiency. Amir Huieh of Unbody.io discussed building AI-native software and announced their transition to open source, inviting community participation. The event underscored themes of simplifying AI integration, experimenting boldly, and leveraging AI for practical problem-solving, reinforcing the collaborative and dynamic future of AI development.

Weaviate emphasizes the importance of robust data protection and security practices, particularly as more community members transition from development to production using their Open Source product. The company has introduced an OSS Hardening Guide to help users enhance the confidentiality, integrity, and availability of their Weaviate installations by leveraging built-in tools and adopting risk-based security measures. This involves assessing potential risks, implementing mitigations such as Role-based Access Control and encryption, and performing thorough testing before going live. Ongoing maintenance is crucial, with regular updates and risk reassessment recommended to address new vulnerabilities and ensure system resilience, underscoring the necessity of a proactive approach to information security. Users are encouraged to engage in comprehensive testing and consider third-party penetration testing for critical services, while staying up-to-date with Weaviate's latest releases to benefit from security improvements and new features.

Weaviate 1.28 introduces significant enhancements to the vector database, focusing on enterprise-grade security, faster indexing, and improved multilingual support. The release features role-based access control (RBAC) as a technical preview, allowing granular user permissions for better security management, although it's not recommended for production use yet. Async indexing improvements make the system more robust and performant by reducing lock contention and switching to an on-disk queue system, enhancing the handling of large datasets. Conflict resolution strategies for deletions are introduced to maintain data consistency across replicas, offering options like DeleteOnConflict, TimeBasedResolution, and NoAutomatedResolution. The update includes a new Japanese tokenizer, kagome_ja, for enhanced keyword and hybrid search capabilities, contributed by a community member. Experimental features like BlockMax WAND aim to improve the efficiency of BM25 and hybrid searches. Additionally, Weaviate continues to support Voyage AI's Multimodal model and has introduced Weaviate Embeddings for scalable, secure embedding generation. The release is available as open-source on GitHub, with upcoming availability on Weaviate Cloud for various deployment options.

Hybrid search combines multiple search algorithms, enhancing precision and relevance by integrating keyword and vector searches, which leverage the strengths of both methods to provide users with an effective search experience. Weaviate, a vector database, utilizes both sparse and dense vectors, with the former requiring tokenizers like the newly introduced KAGOME_JA for Japanese text in Weaviate v1.28. This approach allows for the integration of the benefits of keyword searches, which excel in precise matches, and vector searches, which understand query context, creating an ideal search system for scenarios needing both capabilities. The current implementation in Weaviate uses algorithms like BM25/BM25F for sparse embeddings, while dense embeddings are generated from machine learning models like GloVe or Transformers. Hybrid search, exemplified by queries such as "spring dress," effectively combines vector representations for context (e.g., interpreting "spring" as floral patterns) with keyword precision for terms like "dress." By adjusting the alpha parameter, users can balance the weight between keyword and vector search results, offering flexibility in search systems. This functionality is further enhanced by the ability to use different tokenizers for Japanese text, providing nuanced search capabilities in Weaviate's hybrid search framework.

Unbody, an AI-native development stack, initially adopted Weaviate, an AI-native vector database, to unify AI and product development workflows, but faced limitations with its generative capabilities as demands evolved. While Weaviate's generative module, including Retrieval Augmented Generation (RAG), initially served user needs, issues arose with static configurations, limited syntax flexibility, and text-only input constraints. In response, Unbody developed a custom Generative API, enhancing flexibility by decoupling the generative module from the database, allowing dynamic model selection, multi-modal input, and advanced data transformation capabilities. This solution introduced a sophisticated templating engine, enabling richer interactions and custom workflows, and set the stage for future enhancements such as structured output, streaming support, and broader model and provider integration, aimed at improving performance and functionality.

Weaviate Embeddings, a new service in Weaviate Cloud, aims to simplify the creation of vector embeddings for AI developers by offering model flexibility and eliminating common challenges like rate limits and self-hosting complexities. In the face of a rapidly evolving landscape of AI models, Weaviate Embeddings allows developers to run embedding models directly within the cloud service, providing access to open-source models like Snowflake’s Arctic-Embed, with plans to integrate commercial multilingual and multimodal models. The service promises high throughput and no artificial constraints on processing, supporting large-scale data operations with a pay-as-you-go pricing model based on tokens consumed. Designed for enterprise readiness, Weaviate Embeddings also emphasizes security and offers developers the flexibility to optimize embedding dimensions to meet specific use cases, setting the stage for future advancements in hosted models and promising enhancements in retrieval quality across various domains.