Beyond WER: Interactive Failure Analysis for Multilingual Audio Models... - Rashmi Nagpal, Poseidon

Beyond WER: Interactive Failure Analysis for Multilingual Audio Models in Jupyter - Rashmi Nagpal, Poseidon Word Error Rate is often treated as the final answer when evaluating speech recognition models. But a single aggregate score can hide where a model actually fails. In multilingual and conversational speech, two models with similar WER can behave very differently across dialects, code-switching, acoustic conditions, speaking styles, and data sources. In this talk, I will show how Jupyter can turn model evaluation from a static benchmark into an interactive and reproducible investigation. Using multilingual automatic speech recognition as a case study, we will build an evaluation workflow that moves from WER and CER to semantic metrics, dataset slices, error categories, and inspection of individual failures. I will also share lessons from evaluating speech models across Bengali and Hindi datasets, where aggregate metrics alone often failed to explain important differences between models. The goal is not to introduce another leaderboard. It is to show how notebooks can make AI evaluation easier to inspect, reproduce, extend, and challenge. Attendees will leave with a practical framework they can adapt to evaluate their own models and datasets.