March 2024 Summaries

Supersimple has been using fine-tuned OpenAI models since the availability of GPT-3 Davinci for domain-specific use cases in natural language data questioning. The company recently gained access to the GPT-4 fine-tuning API and found that a fine-tuned GPT-4 outperforms fine-tuned GPT-3.5 by more than 50% in their specific use case. They have been using these models for answering users' natural language questions about data, aiming to provide an effective starting point for further deep dives into the data. The performance comparison shows that fine-tuned GPT-4 is slower and costlier than fine-tuned GPT-3.5 but offers significant improvements in accuracy. Despite these enhancements, models still struggle with broad and open-ended queries. To address this issue, Supersimple employs a mix of various specialized models, prompts, and heuristics to improve both accuracy and response time.

The modern data stack has seen significant investment in recent years, focusing on the initial stages of handling data and improving cloud data warehouses. However, utilizing this data to accomplish something meaningful has received less attention. One exception is the semantic layer, which serves as a central definition layer for "what things mean" and "how we calculate anything." A solid semantic layer offers numerous benefits, including increased flexibility, reliability, reduced maintenance costs, and more context for wherever data is used. Prioritizing efforts and achieving small early wins can be helpful in creating an accurate and useful semantic layer. Some modern data analytics platforms offer tight integration with a semantic layer, ensuring all metrics, ad-hoc explorations, and alerts rely on the same gold-standard definitions.