Convex Evals: Behind the scenes of AI coding with Convex

AI coding is revolutionizing the productivity of developers, as they increasingly leverage AI models to enhance their coding workflows. At Convex, the focus is on evaluating how well large language models (LLMs) can perform specific coding tasks using their reactive database product. This involves understanding the models' ability to write Convex code and addressing issues such as the "knowledge cutoff problem," where LLMs prefer older tools due to pre-training data biases. Convex uses a systematic evaluation method called "evals," which involves tasks, data, and scoring functions to quantitatively measure LLM performance on Convex-specific tasks. The creation of a test suite encompassing fundamentals, data modeling, and other categories has enabled the improvement of AI's coding abilities through prompt engineering, which involves crafting prompts to optimize model output. This approach has shown significant progress in mitigating knowledge cutoff issues without costly fine-tuning. Convex's guidelines, which include specific prompts for using Node.js modules and storage APIs, have improved model performance, highlighting how different models, like Claude and GPT-4o, respond uniquely to prompt tuning. Overall, these evaluations demonstrate the potential of prompt engineering in enhancing AI coding performance, making Convex an attractive platform for building full-stack AI projects.