Researchers from Hugging Face have unveiled UniSwap, described as the first framework for streaming joint audio-visual identity replacement in talking videos. The system addresses a key challenge in character replacement: coordinating the transfer of both appearance and voice while preserving the original motion, scene, linguistic content, and audio-video timing.
Unlike existing methods that rely on separately optimized models for visual and audio modalities, UniSwap performs joint transfer within a single audio-visual diffusion transformer. This unified approach ensures multi-modal consistency, which is difficult to achieve with separate models. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre while maintaining the source content and dynamics.
To overcome the scarcity of aligned cross-identity training pairs, the team introduced a swap-and-reconstruct pipeline. This method removes visual and vocal identity from real clips and uses the original clips as reconstruction targets, effectively generating training data without needing paired identities.
The model is built on a bidirectional backbone and progressively adapted through several innovations: In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and an Efficient Self-forcing DMD mechanism that reduces sampling from 30 to just 3 denoising steps per block. Additionally, Efficient Multi-LoRA Switching allows the three DMD roles to share a single frozen backbone, while Feature-RoPE Decomposition keeps cached positions within the training range, enabling stable long-form inference.
Experiments reported in the paper demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation, positioning UniSwap as a significant step forward in real-time talking-video editing.