Computer-use agents are handling increasingly complex graphical-interface tasks, but unfamiliar software and workflows that depend on human domain knowledge remain difficult. A new paper posted to Hugging Face, Work2Skill, proposes capturing human GUI experience so that it can be turned into reusable agent skills.
The framework rests on three components. WorkRecorder provides the infrastructure for collecting demonstrations, aligning screen recordings with mouse and keyboard events and available accessibility-tree information. Using it, the authors built WorkDemo-350, described as the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation; its 350 demonstrations were screened for completeness and task success.
On top of this data, the authors propose an agentic skill-induction method that inspects evidence on demand to produce skills capturing workflow rules, step dependencies, and completion checks.
In experiments, the same four execution models were equipped with skills induced by each compared method under shared execution settings. Work2Skill reported the highest scores and success rates, improving mean scores over each model's corresponding No Skill baseline by 4.67–10.54 points on MyPCBench Bounded Action and 3.11–7.50 points on a 286-task subset of OSWorld-Verified.
Code and data are available at the project's GitHub repository.