Experimentation in the wild: Google’s AI upsell

Generative AI features are being prominently promoted by companies like Google, which has been experimenting with its Bard AI tool by placing upsell promotions at the top of search results, potentially impacting ad revenue and user interactions. This approach is similar to Instagram's launch of Stories, where valuable screen space was dedicated to a non-monetized feature, resulting in tradeoffs with existing revenue-generating activities. Google's experiments include variations in promotional banner colors and placements, akin to classic A/B testing, to assess the impact on user behavior, such as the usage of Bard versus competitive products like ChatGPT. The process raises questions about experiment design, such as whether assignments are based on search queries or user accounts, and how these experiments could influence user engagement with Google's services. Companies like Statsig offer tools to help manage such experiments, allowing businesses to control feature launches and assess their impact with data-driven insights.