Computer-use agents often waste expensive frontier inference re-deriving routines the user has already performed, because their memory records what was said, not what was done. A new paper from Hugging Face introduces Activity Frames, a deterministic, zero-model compiler that turns locally captured screen activity into structured agent memory.
The pipeline segments a local capture stream into typed activity frames—bounded episodes carrying application, site, timing, input volume, and evidence pointers back to raw rows. With no model in the loop, the output is byte-identical, cacheable, and mechanically auditable, a stark contrast to typical vector-store-based memory systems.
In a single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a context block 86x smaller in 68 ms. An agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%), versus 66-80% for an LLM summary of the same capture. Notably, a mid-tier model reading the block matches a frontier model.
The compiler also serves as a demand-side cost instrument, measuring two parameters previously assumed but unmeasured in agent-cost models: the Routine Overhead Ratio R (60-343x, a modeled upper bound) and delegable recurrence (9.0% in-sample, 7.7% out-of-sample), implying an all-fleet token ceiling near 8%. A compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit.
The authors acknowledge trade-offs: replay is for "same task, new values," and intent is deliberately excluded from the measured tier. Drift across UI states is handled via per-step guards that hand back to the model on mismatch. The schema, compiler, and evaluation harness are open-sourced under MIT.