October 2023 Summaries

Portkey's collaboration with Hasura aims to enhance the development of production-grade Retrieval Augmented Generation (RAG) applications by unifying private data sources and providing secure data query capabilities. Hasura facilitates the creation of robust data APIs that streamline data retrieval and maintain data synchronization, while Portkey brings DevOps reliability principles to the LLM ecosystem, adding observability and reliability layers to RAG apps. The integration of Portkey's platform ensures efficient management of LLM operations through features like automated fallbacks, load balancing, and semantic caching, which collectively improve application reliability and performance. The joint effort addresses challenges such as secure data access, synchronization complexities, and the need for efficient query management, ultimately saving developers time and effort in constructing secure and scalable LLM applications.

Recent observations indicate that GPT-4 is rapidly closing the latency gap with GPT-3.5, showing a significant decrease in processing delays over the past few months. Although specific prompt characteristics can affect response times, the overall trend demonstrates improving speed for GPT-4, with median request latencies for both models remaining under 1 ms per token. Notably, in the 99th percentile, GPT-4's latency has more than halved in just three months. While GPT-4 is still more expensive, it no longer lags behind in speed for most requests. Additionally, there's an ongoing investigation into whether latency increases as users approach their rate limits, suggesting a potential deliberate slowdown by OpenAI.

The paper introduces LLMLingua, a method designed to compress prompts in large language models (LLMs) to speed up model inference and reduce associated costs. This approach integrates a budget controller for determining compression ratios, a token-level iterative compression algorithm, and an instruction tuning-based method aimed at distribution alignment. The research demonstrates that LLMLingua achieves state-of-the-art performance, enabling up to 20x compression with minimal performance degradation. The method was tested on four datasets from diverse domains, showcasing its effectiveness across multiple scenarios. LLMLingua significantly reduces LLM inference costs while maintaining the semantic integrity of prompts, is compatible with black-box LLMs accessible only via API, and requires no gradient flow through the models, making it suitable for various LLM applications.