New Dagster Integration: Include OpenAI Calls Into Your Data Pipelines

We are pleased to announce a new integration that will allow data practitioners to easily include OpenAI API calls as part of their data pipelines, providing insights to optimize API calls and credit consumption. This integration enables the use of generative AI in various scenarios such as document summarization, sentiment analysis, and language translation. Dagster Labs has successfully implemented this integration to build a pipeline that handles complex support requests, provides first-stab answers to user questions, auto-categorizes issues, and generates learning summaries on a weekly basis. To manage costs effectively, the platform introduces features like OpenAIResource and with_usage_metadata, ensuring uniform resource utilization across the platform. These features are integrated seamlessly with Dagster Insights for enhanced analytical capabilities, allowing users to monitor and optimize their data pipelines while maximizing the benefits of generative AI.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.