Researchers have introduced X-AuT, a progressive framework for compressing the audio encoders used in speech large language models. The work, posted on Hugging Face, targets a practical problem: reducing encoder depth lowers inference cost, but removing whole blocks disturbs the embeddings the decoder consumes, which can trigger deletion errors and premature end-of-sequence outputs.
Rather than dropping blocks outright, X-AuT selects layer combinations using short behavioral probes, then restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone stays frozen during this process, while attention LoRA adapters and the tied output embedding are updated during distillation.
Training draws on the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese-English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduced macro-average error from 5.61% to 5.27%. The 14-layer variant reached 5.75% error with 20.7% fewer audio-tower parameters.
Under the matched recipe, a 1.7B teacher produced 5.55% mean error versus 8.45% for self-distillation, and progressive 18-to-14 pruning outperformed direct pruning (5.75% vs. 6.73%). The authors note these are single-run results that establish two practical operating points, and that accuracy effects vary across benchmarks.
The project website, paper (arXiv:2609.11412), source code, and model weights are all linked from the release.