Direct-decision models convert text into structured labels and scores with low latency, making them appealing for classification and automatic evaluation. But a new paper argues that reliability demands more than raw accuracy: the model must also honor the ordinal decision scale the user provides.
The study examines JEV 1.13 and three open KEV models. On ANLI, JEV reaches 74.95% accuracy yet assigns 38.8% of all predictions and 51.3% of its errors to the Neutral category, even though gold labels are nearly balanced and candidate positions are balanced.
Across 36 ordinal datasets, the authors report that final decisions use only 67–76% of the effective gold support, compared with 87–102% on four nominal tasks. Randomizing candidate order weakens the compression but does not eliminate it.
To isolate the effect, the researchers held items and source scores fixed while balancing gold support and positions, then refined scales from K=2 to K=14. Utilization dropped for every model, reaching 26–75% at K=14, although candidate probabilities remained broad for most models.
Targeted BA-LoRA post-training raised gold-relative utilization from roughly 47% to 86% on eight supervised scales at both KEV sizes. The authors conclude that the compression is learned and modifiable, not an immutable architectural limit, and name the phenomenon ordinal scale-utilization bias — a decision-stage candidate-space compression distinct from accuracy, gold imbalance, fixed position, and candidate count alone.
Code and data are available on GitHub.