Fine-tuning large vision-language-action (VLA) models for new robotic tasks remains a major challenge. Traditional behavior cloning requires hundreds of hours of expensive teleoperation data, while reinforcement learning (RL) is often sample-inefficient, especially for long-horizon tasks. Moreover, RL with VLA models poses additional difficulties due to their size and architecture.
To address these issues, researchers introduce EXIMO, an efficient algorithm that fine-tunes VLA policies in three stages: explore, imitate, and optimize. In the explore phase, a vision-language model (VLM) acts as a planner, breaking down complex long-horizon problems into shorter, more manageable sub-tasks for the VLA. Together, the VLM and VLA collect an orchestrated dataset on new tasks.
During the imitate phase, the VLA is fine-tuned using this orchestrated data. Finally, in the optimize stage, residual off-policy reinforcement learning further refines the policy. The researchers ablated all three stages and found that EXIMO significantly outperforms existing approaches in both sample efficiency and final performance.