With the rise of Large Language Models (LLMs), it's increasingly important to monitor their performance as they can be unpredictable and have varying outputs. Observability tools like OpenTelemetry can help analyze inputs and outputs of complex software, including LLMs, providing multiple signals needed for troubleshooting in production. However, LLMs are limited by the information they were trained on and cannot access or retrieve real-time data. Frameworks like LangChain extend their power by integrating them with external databases or APIs through retrieval-augmented generation (RAG). RAG allows models to first retrieve information from a database or search engine before generating a response, providing up-to-date and relevant information. OpenTelemetry can be used to collect and export metrics, logs, and traces from LLM applications, enabling tracing, error tracking, cost management, and A/B testing of performance. By leveraging these tools, developers can build reliable systems with guide rails, even with the use of unpredictable technologies like LLMs.