Researchers from Microsoft and Hugging Face have unveiled RESOURCE2SKILL, a novel framework that converts human-created multimodal resources—including tutorial videos, code repositories, articles, and reference artifacts—into executable skills for software agents. The work addresses a key gap: while the internet is rich with procedural knowledge, existing skill libraries are mostly hand-written or text-centric, leaving valuable multimodal content underutilized.
RESOURCE2SKILL organizes extracted skills into a hierarchical Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals: videos capture temporal operations and visual effects, code provides executable tool patterns, and articles offer conceptual grounding. At inference time, agents retrieve and compose relevant skills; when coverage is insufficient, the system can acquire new skills online.
The framework was evaluated across seven practical authoring domains: PowerPoint, Excel, Web Generation, Blender, CAD, UI/UX, and Music Production. Results show that RESOURCE2SKILL improves the average overall score by +11.9 percentage points over no-skill agents. It outperformed strong harness baselines in 26 of 28 main-aggregate model-domain cells and achieved consistent gains across all 28 model-domain settings.
Ablation studies confirmed the value of the multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition. The researchers note that this is the first framework to demonstrate strong and consistent skill-driven gains across such a broad range of real software environments, suggesting that many valuable agent skills may already exist implicitly across the internet.
The project is open-source, with code available on GitHub and the paper on arXiv. The team believes that skills can serve as a general interface for adapting foundation models to new frameworks, domains, and software environments.