Tool Calling with LLMs: How and when to use it?

OpenAI's introduction of function calling for language models a year ago has evolved into a robust feature known as tool calling, which allows for structured data responses in JSON format, enhancing communication with AI models without explaining JSON structures in prompts. Tool calling simplifies prompt engineering by leveraging model-understood idioms to invoke external actions, enabling structured, consistent outputs and facilitating complex model routing architectures. Unlike OpenAI's JSON mode, which merely forces a format, tool calling integrates a language for structured communication, offering advantages such as prompt injection protection and scalability for building intricate LLM systems. Real-world applications, such as building financial advisors or SQL chatbots, demonstrate the effectiveness of tool calling in handling structured queries and responses, reducing the need for manual parsing and enabling models to self-correct in case of errors. While the implementation details vary between providers like OpenAI and Anthropic, tools like PromptLayer facilitate easy management and iteration of tool call schemas, making it a favored approach in AI application development.