Researchers have introduced VisionHOPE, a visual backbone formulated as a self-modifying learning system, according to a paper posted on Hugging Face. The work positions itself as an alternative to convolutional neural networks, vision transformers, and Mamba-based architectures.
The paper traces the evolution of visual backbones from CNNs with local aggregation, through ViTs with global interactions and state-space models with input-dependent state transitions, to test-time training layers that adapt an inner learner while processing an image. It argues that while visual computation has grown increasingly adaptive to each input, the rules governing that adaptation have largely remained prescribed by the trained backbone.
VisionHOPE builds on the self-referential construction of Nested Learning. The authors describe five coupled memories that store content, generate key and value representations, and govern learning rate and retention, evolving jointly as visual context accumulates along each scan.
Because applying the unconstrained self-referential update directly to a visual backbone leads to instability, the authors derive a stability-matched step-size control scheme. It combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and the paper proves the resulting memory dynamics are non-expansive along each scan.
For two-dimensional feature maps, the method adapts Nested Learning's chunk formulation by aligning chunks with image rows and columns across four directional scans. The authors report competitive results on ImageNet-1K, COCO, and ADE20K, and frame self-modifying learning systems as a practical foundation for general-purpose visual backbones. Code is available on GitHub.