Modern Transformer design and compression are largely exercises in allocating capacity under a budget. The conventional yardsticks for those decisions — parameter count and FLOPs — capture size and compute but say nothing about architectural structure. Two architectures with identical budgets but different depth-width, attention-head, or feed-forward allocations receive the same score, yet behave very differently.
To address that gap, researchers propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko–Pastur law makes NSC computable from the architectural specification alone — with no model instantiation, no data, and no gradients. Because the score is additive across layers, maximizing it under a resource budget becomes an exact dynamic program (NSC-DP) rather than a black-box search.
Empirically, the authors report that NSC outperforms parameter count, FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families. On FlexiBERT, NSC reaches Kendall's τ = 0.695 versus 0.485 for parameter count and 0.635 for W-PCA; on pairs whose parameter counts differ by less than 10%, NSC holds τ = 0.505 where parameter count collapses to 0.082.
In search, NSC-DP discovered a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline (23.087 vs. 23.279 perplexity), found in two seconds on a single CPU core. In pruning, it reduced LLaMA-7B to a 5.7B model that was best across eight commonsense reasoning tasks without any calibration data, roughly 5,900× faster than the strongest training-free proxy baseline.
The work has been accepted at NeurIPS 2026, with code released at github.com/Optima-CityU/neural-spectral-capacity.