April 2024 Summaries

Portkey has been actively updating and expanding its features over the past month, with 154 pull requests resulting in 30 distinct updates, including new features like distinct IDs for feedback updates, saved filters for frequently used combinations, a "Do Not Track" option, a templating engine using MustacheJS, and Okta SSO support for enterprises. They hosted their first "LLMs in Prod" meetup in NYC, offering engaging conversations and attracting attendees from as far as Spain. A significant highlight is the partnership with F5 to integrate Portkey's observability suite and AI gateway with the F5 Distributed Cloud. Additionally, Portkey claims a potential 98% cost saving on LLMs with their FrugalGPT implementation guide and has expanded their documentation and feature set, which now includes support for custom LLMs and the latest Llama 3. They invite users to join their community to prioritize new features and engage with leading LLM practitioners.

Portkey.ai is advancing its mission to empower developers and businesses by integrating Google's Vertex AI into its AI gateway, thereby expanding access to over 250 powerful language models, including the Gemini model series, through a single, consistent OpenAI-like API. This integration simplifies the process of integrating AI models into applications, allowing for exploration and experimentation with state-of-the-art models tailored to specific use cases. Developers can easily install the Portkey SDK and initialize it with their Vertex virtual key to start utilizing the models, and security is prioritized through the support of OAuth2 authentication. The platform also offers a Prompt Library for managing and refining prompts, enhancing the effectiveness of AI integrations. For those needing more control, Portkey.ai supports direct requests to Vertex AI with detailed configurations. With comprehensive model support and observability, Portkey.ai aims to streamline AI integration, enabling developers to focus on creating innovative solutions.

FrugalGPT, a framework developed by Lingjiao Chen, Matei Zaharia, and James Zou from Stanford University, offers strategies to reduce costs and enhance the performance of large language model (LLM) APIs. The framework focuses on three key techniques: prompt adaptation, LLM approximation, and LLM cascade. Prompt adaptation involves using concise prompts to minimize processing costs, while LLM approximation employs caching and model fine-tuning to avoid repeated queries to expensive models. The LLM cascade dynamically selects the optimal set of LLMs based on input, allowing for cost-effective querying. These methods have demonstrated potential for significant cost savings, with FrugalGPT achieving up to a 98% reduction in costs while maintaining or even improving performance compared to individual LLMs like GPT-4. Practical implementation advice, including code examples, is provided to help developers apply these strategies effectively, ensuring efficient and cost-effective LLM-based applications. As LLMs advance, the FrugalGPT framework remains critical for balancing accessibility, cost, and performance in AI applications.

Portkey is increasingly addressing real production challenges by providing modularity that users demand, without adding complexity, as evidenced by the recent need for a JSON mode to configure prompt variables en masse. This trend of users requesting JSON modes for various features, such as Logs, Playground, and Configs, indicates a pattern where developers working at scale require precise tooling to fit their workflows. The company proposes a novel metric for assessing product's production-readiness, linked to the necessity for JSON views. Portkey plans to explore these production workflows further at an upcoming event in New York, aimed at sharing insights from practitioners who have successfully scaled LLM-based applications to production.