In 2024, as LLM-powered applications increasingly enter production, LangSmith introduces new features to assist AI engineers in optimizing these applications post-deployment. These features focus on production logging and automations, enabling engineers to monitor real user data, identify application performance trends, and implement improvements efficiently. The advanced filtering capabilities allow for detailed inspection of data based on various attributes, while the monitoring feature offers aggregated statistical insights over time. The Threads feature provides a comprehensive view of chat-based interactions, facilitating easier debugging. Automations empower users to act on specific data points through actions like adding to datasets or annotation queues, and a novel online evaluation feature leverages language models to assess data based on customizable criteria. An illustrative use case demonstrates how these tools can optimize applications by learning user preferences over time, highlighting the importance of leveraging user interactions and feedback for continuous improvement.