Researchers have proposed ViSkill, a visual-native skill learning framework that aims to make skill-augmented agents less dependent on text. According to the paper, most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language and thereby losing critical geometric structure.
ViSkill instead encodes successful interactions as composite visual skill cards that VLM agents can access directly. Retrieved skills are used to guide both inference and reward shaping, while successful trajectories are distilled back into the skill library — forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. The paper also describes an optional cold-start mechanism intended to accelerate early-stage learning.
The framework couples geometry-aware retrieval, skill-guided reinforcement learning, and online skill distillation. The authors note that prior work incorporating visual evidence typically constructs and updates skills separately from policy optimization, leaving their mutual improvement underexplored — a gap ViSkill is designed to address.
Evaluation covered Sokoban, FrozenLake, and PrimitiveSkill. ViSkill reports an overall success rate of 0.89, rising to 0.91 with cold-start initialization. The authors state it outperforms all evaluated proprietary and open-source baselines and converges faster than standard PPO.
Code is available at github.com/ZJU-REAL/ViSkill.