A new paper from Hugging Face researchers investigates whether small language models can serve as cognitive proxies for human behavior. The team trained 14 models ranging from 135M to 14B parameters across four architecture families on Psych-101, a dataset of 10.7 million trial-level choices from 160 experiments.
Surprisingly, for in-distribution simulations, scale barely matters. Models from 0.6B to 1B parameters match a 70B baseline on held-out participants, falling within a narrow performance band. However, out-of-distribution, the band widens into a steeper scaling gradient, with larger models clearly advantaged in generalizing to novel task structures.
To understand what information the models use, the researchers ran diagnostics by progressively stripping four prompt channels—task instructions, experimental stimuli, outcome feedback, and choice history—across 27 experiments, and permuting trial order. Masking stimuli and feedback destroyed 75.7% of learned information and pushed models below chance, demonstrating that choice history alone does not account for performance.
Permutation revealed invariance on tasks with independent trials but sensitivity where trial order is determined by prior responses. The authors conclude that small cognitively fine-tuned models show promise as noise ceiling estimators for psychological experiments, though their scope remains bounded by training paradigms.