How to measure speaker diarization accuracy (cpWER) in Python

Concatenated minimum-permutation word error rate (cpWER) is presented as a speaker-diarization evaluation metric that measures transcription errors and incorrect speaker attribution by combining each speaker’s words, optimally matching system speaker labels to reference labels, and calculating word error rate under that mapping. The post provides a Python implementation using jiwer for word alignment, SciPy’s Hungarian assignment algorithm for speaker matching, and NumPy for the cost matrix, including support for unequal numbers of speakers. It contrasts cpWER with conventional WER, which ignores speaker assignments, and diarization error rate (DER), which evaluates time-based segmentation and may be affected by pauses, boundaries, and annotation choices. It also explains how to transcribe audio with AssemblyAI speaker labels, aggregate utterances by speaker, and compare the resulting transcript with a labeled reference set. The author recommends consistent text normalization and testing difficult conditions such as short turns and overlapping speech, while citing provider benchmark figures that place AssemblyAI Universal-3.5 Pro below several named competitors on average cpWER.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.