December 2023 Summaries

In the field of large language models (LLMs), there has been a shift from dense architectures, where all neurons participate in processing each piece of information, to mixture-of-experts (MoE) architectures, which allow for more efficient use of resources. MoE architecture involves a gating network that decides which "experts" to route tokens to based on their content. This enables the model to focus its computational power on relevant areas while reducing overall compute time and cost. Mistral 8X7B is an example of an LLM utilizing MoE architecture, with a total of 46.7 billion parameters distributed across eight "experts." The non-feedforward blocks are executed for each token, resulting in only two out of the eight experts being utilized per token. This allows for more efficient use of resources and faster inference times compared to dense models like Llama 2 70B. However, there are limitations to this approach, particularly when it comes to knowledge compression within the model. Due to having fewer parameters than some other LLMs, Mistral may not perform as well on tasks that require extensive knowledge storage and retrieval. Further research is needed to optimize MoE architectures for various applications and improve their overall performance.

The adoption of generative AI tools, particularly large language models (LLMs), is rapidly increasing among enterprise engineering teams. Many early adopters are facing challenges such as evaluation, hallucinations, and abstraction issues, but those successfully deploying LLMs are adopting an agnostic approach to connect with major foundation models and tools, operationalizing scientific experiments through independent evaluations, and quantifying ROI and productivity gains by implementing systems for detecting performance issues and proactively addressing them.

In the paper "How to Prompt LLMs for Text-to-SQL," Shuaichen Chang and his co-author investigate the impact of prompt constructions on the performance of large language models (LLMs) in the text-to-SQL task. They focus on zero-shot, single-domain, and cross-domain settings and explore various strategies for prompt construction, evaluating the influence of database schema, content representation, and prompt length on LLMs' effectiveness. The findings emphasize the importance of careful consideration in constructing prompts, highlighting the crucial role of table relationships and content, the effectiveness of in-domain demonstration examples, and the significance of prompt length in cross-domain scenarios.

This blog post benchmarks OpenAI's GPT models with function calling and explanations against various performance metrics, focusing on correctly classifying hallucinated and relevant responses. The results show trade-offs between speed and performance for different LLM application systems. GPT models with function calling tend to have a slightly higher latency than LLMs without function calling but perform on par with them. For model predictive ability on relevance, GPT-4 performs the best overall, while for hallucinations, GPT-4 correctly identifies more often across precision, accuracy, recall and F1 than GPT-4-turbo. The use of explanations does not always improve performance. When deciding which LLM to use for an application, benchmarking and experimentation are required, considering the latency of the system in addition to the performance of relevant prediction metrics.