AI deep dive: LLM jailbreaking

In 2023, Chris Bakke tricked a Chevrolet dealership's chatbot into selling him a $76,000 car for one dollar using a special prompt to always agree with the customer. This incident is an example of LLM jailbreaking, where malicious actors bypass an AI model's built-in safeguards and force it to produce harmful or unintended outputs. Jailbreak attacks can result in models forcing a legally binding $1 car sale, promoting competitor products, or writing malicious code. To mitigate against these threats, companies must take proactive steps to safeguard their AI infrastructure from exploitation.