October 2023 Summaries

Jacob Lee, a JS/TS maintainer at LangChainAI, explores building a web app using only local models and browser-compatible technologies in response to the growing interest in machine learning among JavaScript developers. The project focuses on creating a web app that performs Retrieval-Augmented Generation (RAG) to allow users to interact with unstructured data locally on their machines, emphasizing cost efficiency, privacy, and potential speed benefits. Using JavaScript-compatible tools like LangChain for document processing, a HuggingFace embeddings model for vector representation, and Voy for vector storage, Lee demonstrates how to set up a pipeline for querying data. However, the challenge of running large language models (LLMs) in-browser remains, leading to the integration of the Mistral 7B model via Ollama for local processing. The project highlights the rapid advancement of open-source models and the potential for future web apps to leverage local hardware, although a new browser API may be necessary for non-technical users to access locally running LLMs easily.

Ollama has been released as an official Docker-sponsored open-source image, enabling users to efficiently operate large language models within Docker containers while ensuring data privacy by processing interactions locally. For Mac users, Ollama runs with GPU acceleration as a standalone application, whereas on Linux, it can be executed inside Docker containers with Nvidia GPU support. The platform offers both a command-line interface and a REST API for application interaction, and getting started involves simple installation steps. Users can run models like Llama 2 inside the container using specific Docker commands, and Ollama provides an online library for more models. The community is encouraged to engage through Discord and Twitter for updates and discussions.