Hugging Face has released a new research paper and live dataset called the Last Translation Benchmark (LTB), designed to push the limits of state-of-the-art machine translation models. The benchmark collects human-authored, peer-reviewed examples spanning text, images, audio, and video that are specifically crafted to break leading translation systems.
The motivation behind LTB is the growing saturation of standard translation benchmarks and the unreliability of current automatic metrics, which are vulnerable to reward hacking and often provide unactionable assessments. Even human evaluation is criticized for lacking reproducibility, objectivity, and scalability, hindering objective progress tracking in the field.
To address these issues, each example in the benchmark comes with handcrafted verification rules that describe concrete failure cases, enabling reliable and actionable future evaluation. The dataset is live and accepts ongoing contributions; the latest version, LTBv1, includes contributions accepted before September 1st, 2026.
In a massive crowdsourcing effort, the team collected 3,456 unique hard-to-translate examples that break state-of-the-art models. Interestingly, the failures are not limited to figurative language; the authors suggest that next-generation models may need significant investment in multilingual and cultural reasoning.
The project is presented as a live paper and dataset, inviting further community participation. The paper is available on Hugging Face, and related research recommendations are provided via the Semantic Scholar API.