How to Search, Scrape, and Crawl the Web from a Convex App with Firecrawl

Firecrawl’s official Convex component integrates web search, scraping, site mapping, and durable crawling into Convex applications, storing crawl state and completed pages in component-managed database tables. Because Convex queries and mutations cannot make network requests, Firecrawl operations must run in actions, while reactive queries can display live crawl progress and incoming pages without client-side polling. Crawls start asynchronously and return crawl and job IDs immediately, then Firecrawl processes pages on its servers and sends completed results through a signed webhook to a public Convex cloud deployment; local development can instead use polling mode. The setup requires installing and registering the component, configuring Firecrawl API and webhook-secret environment variables, and protecting the public webhook through HMAC and per-crawl token verification. The example documentation-search app uses search to find candidate sites, starts path-restricted crawls that produce cleaned markdown and optional screenshots, and renders results reactively, while accounting for Convex’s 1 MB document limit through truncation or omission of oversized page content.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.