Vector quantization (VQ) underpins the discrete visual tokenizers used by modern autoregressive and masked image generation models. While shared-projection codebook methods have pushed codebook utilization forward, the authors of a new paper argue that training stability has remained a critical and underexplored problem.
The paper, titled StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Training, attributes the root cause to the entanglement of encoder-decoder and codebook training. Because neither module can reliably fulfill its own responsibility in isolation, the system only works when the two subsystems happen to cooperate — a fragile condition that breaks down precisely when training is most stressed.
StableVQ revisits the proper learning objective of each module and addresses the problems that arise when each is trained to fulfill its own role independently. The approach combines three components:
- Dynamic STE, which corrects instability in the encoder's learning objective, letting it robustly optimize the reconstruction space under discrete regularization even when codebook utilization is low.
- Region VQ Loss, which reconceives the codebook's learning objective so it can independently guarantee full tracking of the encoder output distribution, without relying on encoder oscillations to drive activation.
- Decoupled Schedule, which assigns the encoder-decoder and the codebook independent learning rate schedules, reflecting their distinct responsibilities and optimization dynamics.
Built on top of shared-projection codebooks, StableVQ is described as lightweight and introduces no learnable parameters. Experiments on ImageNet report consistent improvements in training stability, codebook utilization, and reconstruction quality across diverse codebook sizes and initialization settings.