February 2025 Summaries

We developed a method for performing API enrichments in the warehouse using Snowflake external functions, allowing us to cut down on API usage, keep our data fresh, and maintain consistency across all apps. However, this process was time-consuming and grew in complexity with every provider. We've now made it easier by introducing HTTP columns in Census, which enables users to set up enrichments from any API endpoint directly in the platform, without lengthy setup or SQL maintenance. This allows users to sync enriched datasets directly to any connection, and can be used for free or low-cost APIs such as MaxMind, US Postal Service, or internal services.

Gemini Flash and Pro are now available within Census AI columns, offering a range of models with varying performance and pricing. Gemini's pricing is based on the length of the prompt, with costs ranging from $0.075 to $10 per million tokens. The Gemini 1.5 Flash model is optimized for speed and has a context window of 1 million tokens, while the Gemini 1.5 Pro model is more performant but also more expensive. Benchmarking results show that Gemini models consistently perform close to par for their GPT counterparts in most applications, but Claude still outperforms them in many areas. Lightweight models like Flash and mini are suitable for internal applications, while more complex models like Pro and GPT 4o are better suited for human-like tasks or externally-facing tasks that require high performance.

The benchmarking landscape for Large Language Models (LLMs) is complex, with various testing styles and metrics used to evaluate their performance. The MMLU benchmark tests models across multiple subjects, including humanities, STEM fields, and medicine, while the BIG-Bench Hard test assesses reasoning capabilities on challenging tasks. The DROP test evaluates discrete reasoning over paragraphs, and HellaSwag tests common sense reasoning through sentence completion tasks. Math benchmarks like GSM 8k and MATH assess reading comprehension and logical problem structuring, with scores varying widely depending on the model's performance. Code benchmarks like HumanEval evaluate LLMs' coding capabilities, while leading models in each benchmark consistently outperform others, with some showing significant gaps in their performance. Understanding these benchmarks is crucial to making informed comparisons and optimizing model selection for specific use cases.

The company Census transformed an overwhelming stream of product events into actionable updates that their teams actually looked forward to reading. They achieved this by organizing essential events, turning customer events into AI-ready data, and bringing in AI magic. They captured moments like creating new models, setting up syncs, and adding destinations, and bundled them into a neat JSON array for GPT analysis. The output provides a structured report with key activity, growth signals, feature usage breakdown, service connection type concerns, suggested next steps, and sales talking points. This system has transformed how their teams work, providing context for every call, spotting fires before they start, making account managers look like mind readers, and everyone working from the same playbook. They are constantly tweaking it but find it beats staring at dashboards or hoping for the best.