A newly posted Hugging Face paper, One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts, proposes reViT, an architecture that applies a single Transformer block recurrently to match the accuracy of a full-depth vision encoder at comparable inference FLOPs — and, notably, without intermediate feature distillation.
The core idea is to restore depth-specific transformations inside the reused block. reViT represents the feed-forward network (FFN) at each recurrent depth as a convex combination of a small shared expert bank. A continuous normalized-depth coordinate programs this mixture, defining a resampleable trajectory through FFN parameter space.
The authors evaluate the design in two regimes: supervised ImageNet-1k training and distillation from a DINOv2 teacher. Across both, controlled adaptations identify weight-space merging as the strongest tested mixture-of-experts family at a matching one-FFN budget, ahead of token-dispatch and output-mixture alternatives.
Reported results include reViT-B/16 trained from scratch attaining DeiT III accuracy with roughly 70% fewer stored parameters. An 8-expert model distilled using only the teacher's output features reportedly retains nearly all of its DINOv2 teacher's linear-probe accuracy and transfers across classification, segmentation, and depth prediction.
The paper also describes elastic-depth training, which lets one checkpoint operate at multiple tested depths by resampling the same normalized coordinate interval. For fixed-depth deployment, the recurrent block can be materialized as a conventional dense graph, removing online routing and merging without changing the one-FFN-per-depth compute — though this expands deployment storage.