Researchers from Hugging Face have introduced dRAE (Representation Autoencoder with Hyper-Spherical Codes), a novel visual tokenizer that addresses the long-standing problem of codebook collapse in vector quantization. The work, published as a research paper, identifies a fundamental metric mismatch between standard Euclidean codebook objectives and the anisotropic geometry of representation space as the root cause of codebook collapse.
To solve this, the team proposes Hyper-Spherical Quantization (HSQ), which decouples semantic content from feature magnitude by assigning codes based on cosine similarity rather than Euclidean distance. This angular routing prevents code assignment from being dominated by scale, aligning the quantization objective with the intrinsic geometry of pre-trained vision encoder features such as SigLIP2 and DINOv2.
The resulting dRAE model achieves 100% codebook utilization and consistent performance gains as the vocabulary size scales up to 131,072 tokens. The method also simplifies the training pipeline while maintaining high-fidelity reconstruction and strong semantic alignment across both understanding and generation tasks.
Experimental results demonstrate that HSQ-based quantization outperforms existing methods in preserving semantic coherence and scalability, marking a significant step toward bridging the gap between visual representations and language models.