May 2024 Summaries

Large Language Model (LLM) summarization is a technique that uses advanced natural language processing methods to generate concise and informative summaries of longer texts. It leverages LLMs to comprehend the content of source documents and produce abridged versions that capture key points and main ideas for an LLM system. The benefits of summarization include streamlining information processing, enhancing efficiency in information retrieval, promoting better retention and understanding of materials, leading to improved learning outcomes. There are three primary approaches to LLM summarization: extractive, abstractive, and hybrid. Extractive approach involves selecting and assembling specific sentences or passages from the source document to create a summary. The abstractive approach aims to understand the underlying meaning and concepts expressed in the text, emulating human comprehension. Hybrid approach combines elements of both extractive and abstractive techniques, leveraging their advantages while mitigating their limitations. Challenges in LLM summarization include recursion issues, refine issues, better chunking for summarization, and evaluation. Evaluation generally consists of an evaluation of LLM outputs by using a separate evaluation LLM. The fundamentals of LLM evaluation for production include benchmarking with a golden dataset, leveraging task-based evals, and running across environments. A code walkthrough demonstrates how to perform summarization classification tasks using OpenAI models (GPT-3.5, GPT-4, and GPT-4 Turbo) against a subset of the data from a benchmark dataset. The results show that there is a significant increase in the quality of predictions with each model, with GPT-4 Turbo providing the best performance.

In this paper review, we discussed how to create a golden dataset for evaluating LLMs using evals from alignment tasks. The process involves running eval tasks, gathering examples, and fine-tuning or prompt engineering based on the results. We also touched upon the use of RAG systems in AI observability and the importance of evals in improving model performance.

GetYourGuide powers millions of daily ranking predictions by leveraging Production AI, a machine learning system that meets such demands and maintains performance. The company faced several challenges while building its search ranking system, including diverse feature types, running real-time feature pipelines, cost-efficient serving, A/B testing, drift detection, and data quality monitoring. To tackle these challenges, GetYourGuide adopted Tecton as its feature platform and Arize for model observability, which fit nicely with the organization's existing tech and helped create new features for user personalization while offering a clearer view of how models perform in production and whether any changes in features or model behavior need addressing. The team uses Airflow to orchestrate dataset generation, automate model training, and deploy a fresh model on a daily basis. Tecton's offline store enables GetYourGuide to easily fetch point-in-time accurate feature values for each unique entity at the exact time of historical ranking events. Arize is used to monitor model performance and track Normalized Discounted Cumulative Gain (NDCG) as a primary performance metric, allowing the team to identify areas of improvement and compare different datasets.

Arize AI has partnered with Microsoft Azure to enhance the deployment of large language models (LLMs) in enterprise applications. The collaboration integrates Arize's LLM evaluation and observability platform with Azure's Model as a Service, offering users access to popular open-source models curated by Azure AI. This partnership aims to speed up the reliable deployment of LLM applications while ensuring robust ML and LLM observability for Fortune 500 companies using Azure along with Arize. The integration also provides tools for collecting evaluation data, troubleshooting search and retrieval, and tracing to see where an LLM app chain fails.

Generative AI can be used to evaluate bias in speeches by analyzing the language and content for potentially discriminatory remarks. A custom prompt template was created using OpenAI's GPT-4 model, which identified a section of Harrison Butker's commencement speech as "misogynistic" due to its perpetuation of gender stereotypes. The LLM classified another section of the speech as "homophobic" after identifying derogatory comments and references to Pride Month. These results highlight the potential for generative AI to monitor and mitigate harmful language in various contexts, including online conversations, customer call centers, and public speeches.

The evaluation of large language models (LLMs) is crucial to ensure their reliability and effectiveness in various applications. However, the process of evaluating LLMs can be challenging due to the subjective nature of some criteria and the need for human judgement. In this paper review, we discuss a study that explores the use of LLMs as judges for evaluating other LLMs. The study presents a framework called EvalGen, which aims to improve evaluation metrics by incorporating human feedback and iteratively refining evaluation criteria. The EvalGen framework consists of four main steps: pretest, grading, customization, and implementation. In the pretest step, users define their evaluation criteria and create an initial set of examples with labels. The LLM judge then evaluates these examples based on the defined criteria. In the grading step, human evaluators grade the LLM's performance on the same set of examples to identify any misalignments between the LLM's judgement and human expectations. The customization step involves adjusting evaluation criteria based on feedback from human evaluators. This can include adding or removing criteria, modifying existing criteria, or changing their weightage. The final implementation step incorporates the refined evaluation criteria into the LLM application for continuous monitoring and improvement. One key takeaway from this study is the importance of iterative evaluation and refinement of evaluation criteria to ensure accurate and reliable results. Additionally, the use of golden data sets can help users better understand their evaluation criteria and identify any misalignments between human judgement and LLM performance. While there is some skepticism around using LLMs as judges for evaluating other LLMs, particularly in production environments, this study demonstrates that with proper customization and iteration, LLMs can be effective tools for evaluating LLM applications.