Speech emotion recognition (SER) is essential for real-time applications on edge devices, but the most accurate models are large self-supervised systems that are too computationally expensive for such environments. Multi-teacher knowledge distillation offers a way to compress these models into a lightweight student, yet two key challenges persist: teacher reliability varies across data batches, and standard logit-level distillation ignores the relational structure between samples.
To address these issues, researchers introduce Adaptive Multi-teacher Relational Distillation (AMRD). The method uses a one-class SVM on each teacher's logit similarity matrix to assign per-batch weights, favoring teachers that produce more coherent outputs. Additionally, a relational distillation loss aligns the similarity matrices of teachers and student, capturing inter-sample relationships that logit matching misses.
Experiments on the IEMOCAP and CREMA-D datasets across four student architectures show that AMRD outperforms single-teacher distillation baselines in most settings. Ablation studies confirm that both components—adaptive weighting and relational distillation—provide complementary gains, highlighting the effectiveness of the approach for lightweight SER.