Whale Song Embeddings: Visualize Audio Models in FiftyOne

A FiftyOne demo notebook explores research suggesting that general-purpose audio models such as CLAP and AST can encode evolutionary relationships among marine mammals and birds nearly as effectively as a specialist bioacoustics model, despite not being trained on animal sounds or phylogeny. Using about 1,700 recordings from 32 species in the historic Watkins Marine Mammal Sound Database, the project compares embeddings from CLAP, AST, BEATs-bio, and an MFCC acoustic-feature baseline. FiftyOne projects these high-dimensional representations into interactive two-dimensional visualizations, where taxonomic groups such as whales, seals, dolphins, baleen whales, and belugas can be inspected alongside synchronized spectrograms and audio playback. The comparison shows clearer clade and family separation for the learned embedding models than for MFCC features, while CLAP also enables text-based audio retrieval, allowing prompts such as “barking seal” to locate relevant clips without a reference recording.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.