We made an open source LLM Performance Tracker

Building AI features presents unique challenges due to their stochastic nature and non-deterministic costs, making observability crucial for performance monitoring and iteration in production. The LLM Performance Tracker is an open-source solution designed to help AI engineers and developers track the performance of large language models (LLMs) across applications, offering real-time visualization of key metrics like cost, time-to-first-token, and total requests. Built with a tech stack including Next.js, Tinybird, and Vercel, this tool allows for multi-dimensional filtering, drilldowns, and multi-tenancy support, enabling users to analyze usage patterns, evaluate model efficiency, and assess the impact of parameters like "temperature" on response quality and costs. The app is customizable and extendable, supporting integration with observability platforms for alerting on usage anomalies, while also providing a template for deploying analytics dashboards with advanced features, such as user authentication and multi-tenant dashboards, using Clerk and Tinybird. Users can easily fork and adapt the project to fit specific needs, and for those seeking more advanced observability options, Dawn offers comprehensive AI monitoring solutions.