June 2024 Summaries

Researchers have developed a method to determine when an AI large language model's response comes with uncertainty. They distinguish between two categories of uncertainty: epistemic (lack of knowledge) and aleatoric (irreducible randomness). By employing an information-theoretic metric, they can consistently identify occurrences where epistemic uncertainty is elevated, suggesting that the model's output may be unreliable or even a hallucination. The main idea is to capitalize on the diverse behavioral patterns observed when an LLM is presented with repeated potential responses. An information-theoretic metric measures epistemic uncertainty by assessing the sensitivity of the model's output distribution to the iterative addition of previous (potentially incorrect) responses to the stimulus. The paper introduces a hallucination detection algorithm based on scores, determining a "pseudo joint distribution" over multiple responses and using mutual information as a score that denotes the degree of conviction that the LLM hallucinates for the specified query. Experiments show that MI-based method exhibits comparable performance to semantic-entropy baseline on predominantly single-label datasets and significantly outperforms simpler metrics such as probability of greedy response and self-verification methods.

The Align AI Research Review discusses a paper that explores understanding and interpreting the cognitive processes of AI using sparse autoencoders to extract interpretable features from large language models like Claude 3 Sonnet. By training these autoencoders on substantial datasets, thousands of features utilized by the model to process information were identified. The researchers discovered a systematic correlation between the overall incidence of a concept in the training data and the dictionary size required to resolve a corresponding feature. They also demonstrated that manipulating the activations of specific features could consistently induce the model to exhibit or refrain from specific behaviors, providing valuable insights into the model's internal representations and behaviors.