A conversation with StackAI's CTO: how Temporal cut indexing failures from 20% to below 0.1%

StackAI, an enterprise AI platform acquired by Asana, uses Temporal Cloud to coordinate the complex process of turning files from Google Drive, SharePoint, and Dropbox into searchable knowledge bases for regulated customers. Its pipeline downloads files, extracts text and images, chunks content, and indexes it in a vector database, where failures from API limits, corrupt files, parser errors, or network issues had previously caused roughly 20% of files to fail. StackAI gradually replaced its Celery-based workflow with Temporal, enabling durable, observable workflows that retry only failed stages rather than repeating completed work and paid API calls. The team separated parsing into a dedicated service, routed files by type and size, used parser fallback chains and content-based caching, and handled long-running external parser jobs through durable waits and webhooks. Temporal’s visibility, task queues, priorities, and independent worker scaling improved debugging and performance while keeping the main product responsive during large ingests. Combined with parser and pipeline optimizations, these changes reduced indexing errors to below 0.1%, eliminated customer complaints about indexing, and lowered parsing costs to about one-tenth of their prior level.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.