Researchers at Hugging Face have unveiled UltraBERT, a bidirectional encoder that they say is the first of its kind to achieve state-of-the-art performance on both classification and retrieval tasks. According to the paper, the model reaches 91.0 on GLUE and 47.7 on BEIR.
The work revisits core pretraining design choices that, the authors argue, have remained inherited from small-scale conventions even as encoders have adopted innovations from causal large language models. Three findings stand out: encoders saturate on downstream tasks beyond a few trillion tokens, meaning extra compute past that point does not help; lower masking ratios favor classification while higher ratios favor retrieval, contradicting prior work that suggested larger ratios for larger models; and distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones.
Guided by these findings, UltraBERT assigns separate language-model heads to low and high masking ratios within a batch and restricts the distillation objective to the low ratio. The authors describe this as a principled basis for compute allocation, masking ratio design, and knowledge transfer in bidirectional models.
Alongside the paper, the team released several checkpoints: UltraBERT-150M and UltraBERT-380M, both pretrained encoders with 32K context, and UltraBERT-Embed-150M, a ready-to-use embedding model.