Hugging Face has published a technical report describing NCP-ArchPreview, a latent-space language model that extends standard autoregressive pretraining beyond next-token prediction (NTP). The model is trained jointly on NTP and a new objective called Next Concept Prediction (NCP), in which it learns to predict discrete concepts that span multiple tokens.
According to the report, the architecture builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, then learns to predict future concepts through a dedicated Concept Module. Those predicted concepts are fed back to the token level to guide subsequent generation, with both objectives trained end-to-end.
The authors scaled the design to 8.9B parameters and trained it on 5.73T tokens from the Dolma-3 dataset, which they describe as the largest demonstration of a latent-space language model to date. By consuming only 51.3% of the total training tokens, NCP-ArchPreview reportedly reaches the final pretraining loss of OLMo-3-7B.
After full pretraining, the model is said to outperform OLMo-3-7B by 2.45 points on the downstream macro-average, including a 5.99-point gain on GSM8K. Controlled experiments are reported to isolate performance gains from both the latent architecture and the NCP objective, and using only 85% of standard computation the model approaches the training loss of a strictly parameter-aligned 8.9B baseline.
The report also notes post-pretraining uses for the learned latent space: updating only the 17M-parameter VQ module provides a lightweight interface for domain adaptation, while injecting concept representations into a DFlash2 drafter improves mean accepted length by 4.17% with negligible overhead.