The rapid rise of AI has introduced a new paradigm in digital product development, significantly changing how products are evaluated for quality and effectiveness. Traditional evaluations, or "evals," which assess AI performance through predefined benchmarks, often fall short in capturing the qualitative aspects of AI outputs. Instead, product analytics, focusing on user behavior and engagement, offer a more reliable measure of an AI product's success. By analyzing how users interact with AI-driven products, companies can gain insights into user satisfaction and product effectiveness, similar to how mobile apps were previously assessed. This approach not only helps identify areas for improvement but also guides iterations of AI models by linking user behavior data to product performance. Ultimately, while model quality remains important, understanding and leveraging user engagement through product analytics becomes crucial in determining an AI product's real-world value and success.