Test-time scaling is widely used to improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based approaches, the inference budget is usually summarized by the number of generated candidates, N. A new paper argues that this single number is an incomplete description of the systems cost, because it says how many candidates are produced but not how they are executed.
The authors note that the same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. To test the effect, they first measured how increasing N affects reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. Raising N from 1 to 8 improved accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B.
Accuracy, however, does not capture the systems cost of a larger candidate budget. The team therefore fixed N = 8 and compared four generation schedules — 1x8, 2x4, 4x2 and 8x1, where axb denotes a generation calls with b candidates per call — while measuring latency, throughput, GPU-hours and gross GPU-device energy.
On A100 GPUs, eight serial calls consumed 4.64–4.86x as much gross GPU-device energy and showed 5.77–6.12x the P95 latency of a single batched call with eight candidates. The pattern held across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments.
The authors conclude that candidate count alone is insufficient to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory permits, fewer generation calls with larger batch sizes are more efficient. They recommend that evaluations report not only candidate count and accuracy, but also the generation schedule and GPU-level systems metrics.