October 2023 Summaries

A recent survey of over 350 AI professionals shows that adoption of large language models (LLMs) is accelerating, with 61.7% of developers and machine learning teams planning to have an LLM app in production within a year or faster. OpenAI remains the dominant player, but alternatives like Meta's Llama 2 are gaining popularity. Concerns about data privacy and responsible deployment are decreasing, while barriers such as "require on-prem" and "accuracy of responses and hallucinations" are increasing. Prompt engineering is the most common implementation method for LLMs, and retrieval augmented generation (RAG) is the most popular use case among teams planning to leverage LLMs. The survey indicates that LLM adoption is not a passing trend and highlights the growing need for tools like LLM observability to ensure companies can maximize the benefits of these models.

The benefits of model observability are significant, with high returns on investment (ROI) due to its ability to preemptively detect and fix model issues impacting business value. Model insights can be automatically detected through Arize monitors and then root-caused through data exploration in interactive guided workflows. The study found that 95% of teams can find a valuable insight when first exploring their data in Arize, with users uncovering an initial insight within the first 24 hours. Proper monitoring coverage allows for automatic detection of model insights, which can be detected once a month and resolved within 24 hours. Observability initiatives provide cost savings by correlating improvements back to business metrics, enabling easy calculations of the value of observability and individual insights for each project and model. Arize offers training and guides on monitoring best practices, and custom metrics allows users to define any metric using model data and metadata to track personalized metrics.

In this paper, the authors propose RankVicuna, an open-source model for document re-ranking that achieves comparable performance to proprietary models like GPT-3.5 and GPT-4 while being significantly smaller in size (7 billion parameters compared to 175 billion). The model is deterministic, ensuring consistent output format and rankings across different runs. RankVicuna uses a teacher-student paradigm for data augmentation, generating query-document pairs from a larger model and shuffling the input order of documents to provide more examples for training. The authors also highlight the importance of prompt engineering in achieving stable results. Overall, this paper showcases the potential of open-source large language models for document re-ranking tasks and emphasizes the role of data augmentation and prompt engineering in improving model performance.

In this paper, the authors propose RankVicuna, an efficient and deterministic reranking model for large language models (LLMs). The model is based on Vicuna, which has been fine-tuned using instruction data from Open Assistant. The main advantage of RankVicuna is its smaller size compared to proprietary models like GPT-3.5 and GPT-4, while still achieving comparable performance in terms of ranking metrics such as NDCG@10 and MAP@100. The authors also highlight the importance of data augmentation for ensuring stability in document reordering. They demonstrate that using a teacher model to generate training data examples from a larger model can improve performance, especially when dealing with smaller datasets. Additionally, they showcase the effectiveness of prompt engineering in achieving stable outputs and reducing hallucinations. Overall, RankVicuna offers an open-source alternative for reranking LLM outputs, which could be particularly useful for teams that do not have access to proprietary models like GPT-3.5 or GPT-4. However, further research is needed to evaluate the model's speed and scalability in production settings.

Microsoft Presidio is an open-source project aimed at ensuring proper management and governance of sensitive data, including PII (personally identifiable information). It uses mechanisms like entity recognition, regular expressions, rule-based logic, checksum with relevant context in multiple languages, and external PII detection models. The two main components are AnalyzerEngine, which scans text to identify PII, and AnonymizerEngine, which replaces identified PII with anonymized values. Presidio can be used to anonymize conversations in a chatbot system by importing necessary dependencies, initializing the analyzer and anonymizer, creating a function that finds and redacts important PII, and running this function on each row of a pandas dataframe to create a new column with anonymized data.

Explaining Grokking Through Circuit Efficiency" is a research paper exploring novel predictions about grokking in neural networks, providing significant evidence in favor of its explanation. The authors demonstrate two surprising behaviors: ungrokking, where a network regresses from perfect to low test accuracy, and semi-grokking, where a network shows delayed generalization to partial rather than perfect test accuracy. The paper discusses the concept of "circuits" within neural networks, which refer to modules that can learn multiple different ways of achieving low loss in parallel. The authors argue that efficiency is independent of training size and that there is a crossover point beyond which the network's performance improves dramatically. They also propose a novel prediction about grokking, which they show is supported by their analysis. The paper highlights the importance of understanding generalization and the challenges associated with it, particularly in the context of large language models like GPT-4. The authors discuss potential applications and open questions related to grokking and efficiency in neural networks.

Arize Phoenix is an open-source library that provides visualizing datasets and troubleshooting large language model (LLM) application development by making it easier to debug and troubleshoot LLM framework applications. It offers flexible data frameworks to connect private data to LLMs, enabling developers to gain visibility into their system with LLM orchestration frameworks like LlamaIndex, Microsoft's Semantic Kernel, and LangChain. Arize Phoenix provides a comprehensive view of the inner workings of an LLM application by breaking down the process into spans and categorizing each span with a common interface across frameworks, making troubleshooting and optimization easier and more effective. The library offers various features such as tracing, evaluating, and analyzing LLM applications to surface problems at different levels of the system, including prompt templates, token usage, runtime exceptions, retrieved documents, embeddings, LLM parameters, tool descriptions, and LLM function calls. It also supports all common spans and has a native integration into LlamaIndex and LangChain, enabling developers to get started with Arize-Phoenix in a few minutes.