Dubbing quality control often requires judging whether a candidate text line matches a speaker's visible articulation in both content and timing, using only silent video and text because the dubbed audio may not yet exist. Existing visual speech recognizers and video-language models struggle in this setting: even when fine-tuned to recover spoken content from lip motion, they remain largely insensitive to temporal errors.
To address this, researchers propose Align Then Reason (ATR), a multilingual lip-sync judge. ATR first establishes a monotonic alignment between frame-level lip representations and the phonetic units of the candidate line, then reasons over this alignment to make a final judgment. An alignment scorer supplies the LLM with local evidence for each phonetic unit and a calibrated global alignment score, enabling joint reasoning about content and timing.
On a seven-language benchmark, ATR improves mean AUC over corresponding Qwen3.5 SFT baselines by 59.4%, 50.2%, and 50.8% with 2B, 4B, and 9B reasoners, respectively. The gains generalize across LLM families, reaching mean AUC improvements of 45.9% and 46.6% over the best baseline for LLaMA-3.1-8B and Mistral-7B, respectively. They also transfer across datasets to three unseen MuAViC languages.
Beyond the benchmark, the team evaluates two downstream tasks built from real dubbing lines. On dub-line reranking, ATR-9B outperforms the best lip-reading baseline by 52.0%, while on script-to-clip assignment, ATR-9B improves over the best lip-reading baseline by 17.7%.
The work is presented as a research paper on Hugging Face, with the authors positioning ATR as a reference-free judge suited to dubbing workflows where audio is not yet available.