Failure Mining in Robotics: Find Every Similar Failure Case

Failure mining in robotics uses a known failure as a starting point to identify similar events across large collections of robot recordings, helping teams measure recurring problems and create targeted training or evaluation data. Using the DROID dataset and FiftyOne, the workflow marks a dropped-cloth incident with a temporal tag, then uses Voxel51’s segment-level embedding search to retrieve visually similar five-second camera windows rather than entire episodes. Because embeddings recognize visual similarity but not intent, the search can confuse accidental drops with deliberate releases, so results are narrowed using metadata such as task outcomes and then manually reviewed. In the example, an initial search found 16 cloth-handling episodes, while filtering to unsuccessful episodes reduced the set to two candidate failures. Confirmed failures are temporally tagged, saved as a reusable dataset view, and exportable for downstream model training or evaluation; the tagging, filtering, and saving steps work in open-source FiftyOne, while segment-level similarity search requires Voxel51’s enterprise offering.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.