Pretraining large language models has long been associated with prohibitive costs, often placing it out of reach for academic and open-source communities. Even small-scale efforts like Llama-3.2-3B can exceed $1.5 million, while reproducing SmolLM3-3B requires over $700K. A new report from Hugging Face introduces Puro-2B, a collection of 2B-parameter models trained from scratch on consumer-grade RTX 5090 GPUs, with a fully open recipe.
The recipe leverages FP8 precision and a combination of techniques including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and a carefully designed data recipe. The best model, trained on up to 1.4 trillion tokens, achieves performance close to Qwen2.5-1.5B at a compute cost of less than $6.9K. The authors also derive a Puro Cost Scaling Law, suggesting that about $4.4K (under $5,090) is sufficient to match Qwen2-1.5B.
Beyond the recipe, the study provides an end-to-end case study on how pretraining data curricula affect downstream performance after post-training. This is enabled by full access to the pretraining pipeline, not just model weights. The entire recipe—including data, code, and model weights—is released under Apache 2.0.
The report also addresses community questions, such as the use of FP8 precision and the number of GPUs, though specific hardware counts and training duration are not detailed in the source. The authors acknowledge the FP32-accumulate throughput on consumer Blackwell GPUs and note that FP16-accumulate paths were not explored in depth.