Researchers have introduced Puppeteer, a diffusion-based model for generating co-speech gestures that are temporally coherent, semantically aligned with speech, and physically grounded in the surrounding environment. The work addresses a gap in prior speech-driven gesture models, which the authors say emphasize audio-gesture alignment but do not explicitly account for posture constraints or nearby objects.
Puppeteer operates in a causal latent space. The method decomposes long gestures into structured primitives and learns a causal variational autoencoder that encodes them into temporally ordered latent tokens, where each token depends only on the past. Conditional diffusion is then performed directly in this causal latent space, conditioned on speech signals, motion history, an initial posture reference, and object geometry.
According to the authors, the temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis, the team also introduces new evaluation metrics tailored to the task.
The paper also presents SceneGes, described as the first curated synthetic 3D dataset of embodied co-speech gestures paired with corresponding 3D objects, enabling object-grounded gesture generation. Experiments reported by the authors indicate that Puppeteer produces more diverse and temporally synchronized gestures than prior methods while enabling object-grounded synthesis.