Researchers from Hugging Face have released a paper examining how instruction tuning affects language models' confidence and the diversity of their reasoning rationales. The study, titled "Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity," addresses concerns about verbalized overconfidence in instruction-tuned models.
The authors evaluated three matched pairs of base and instruction-tuned models across question-answering benchmarks. They found that instruction tuning consistently alters answer confidence, even though predictive accuracy changes are limited and likelihood-based calibration actually decreases. This suggests that instruction tuning can make models more confident without making them more accurate or better calibrated.
Notably, the effect on rationale diversity is non-uniform: cross-rationale diversity (the variety of reasoning paths across different samples) consistently decreases, while surface-level lexical diversity (word choice variety) varies in both direction and magnitude depending on the model and benchmark. These differences persist even after controlling for answer selection and rationale length, indicating that confidence and rationale diversity capture distinct effects of instruction tuning.
The findings have implications for understanding how instruction tuning shapes model behavior, particularly in high-stakes applications where overconfidence could be problematic. The paper is available on Hugging Face and has been recommended alongside related work on calibration and consistency.