Text-to-image and personalized editing models have made significant strides in generating high-fidelity images of single subjects. However, when it comes to placing multiple named individuals in shared contact actions—such as embracing, carrying, or grappling—these models often fail dramatically, producing fused limbs, invented extremities, and interpenetrating bodies. Existing evaluation methods largely overlook these anatomical and geometric issues, and VLM-as-a-judge checklists often saturate on interaction quality while the errors remain obvious to human observers.
To address this gap, researchers from Hugging Face have introduced MPIE-Bench, a benchmark comprising 2,500 video-mined editing triplets across 405 scenes, 14 interaction categories, and four contact densities (C0-C3). Alongside the benchmark, they propose MPIE-Eval, a new evaluation framework with two axes that score contact-time geometry using a frozen public multi-person mesh reconstruction model.
The first axis, Anatomy, checks whether every human-like mass in the image is explained by a complete set of reconstructed bodies. The second axis, Interaction, measures whether the penetration and surface distance between those bodies match the contact described in the instruction. This dual approach provides a more nuanced assessment than existing VLM-based checklists.
In their experiments across ten different editors, the mesh-based Anatomy score topped out at 0.65 and the Interaction score at 0.72 on two different models, indicating that no single editor excels at both. In contrast, VLM checklists rated the same images above 0.95, highlighting the saturation problem. A five-rater study confirmed that both axes track human judgment more closely than a zero-shot VLM judge, and the rankings remained consistent under ablation of every weight and threshold.
This work underscores the need for more robust evaluation metrics in multi-person image editing and provides a valuable resource for future research in the field.