Byte-level hierarchical language models (LMs) have emerged as a robust alternative to subword-based models, but their inference speed has been hampered by the need to generate one byte at a time. A new paper from Hugging Face introduces multi-byte prediction (MBP), a technique that generates multiple bytes in parallel, significantly accelerating inference while maintaining performance.
The method builds on the established multi-token prediction (MTP) paradigm but introduces two key innovations. First, it uses a variable-length prediction window that aligns with the latent segments of the hierarchical model. Second, it implements a novel attention-masking scheme that allows parallel byte generation without violating causality.
According to the paper, MBP achieves a Pareto-optimal trade-off across several generative tasks, including instruction following, question answering, summarization, and machine translation. The approach delivers the best balance between performance and inference throughput, with no additional parameters required.
The paper was published on Hugging Face and has attracted recommendations for related research, including adaptive training paradigms for multi-token prediction and speculative decoding methods.