A newly posted paper on Hugging Face, Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks, explores a general-purpose layer for deep learning that departs from the usual practice of compressing training data into fixed-size weight matrices. Instead, the approach stores a new key-value representation for every data point during training and retrieves and recombines those representations through an attention mechanism at inference time, yielding what the authors call a growing neural network.
The work builds on a known duality: any linear layer in a deep network trained by gradient descent can be expressed as linear attention over training data points, a perspective advanced by Irie et al. (arXiv:2202.05798). The authors note, however, that replacing linear attention with more powerful attention functions is not straightforward — naively transferring learning rules from the linear-attention case to advanced kernels does not produce principled optimization.
To address this gap, the paper develops functional gradient-based learning rules for kernelized attention layers based on radial basis function (RBF) and softmax-like kernels, establishing what the authors term the "retrieval-centric deep learning" (RCDL) paradigm.
Empirically, the authors report promising performance and learning efficiency for RCDL on image classification and synthetic teacher-student learning tasks. They also show that substituting linear attention in the dual form of neural networks with advanced variants such as MesaNet and DeltaNet yields a formal connection to recently proposed optimizers for conventional fixed-size networks, offering a new perspective on deep learning optimization.
A short TLDR accompanying the paper describes the work as demonstrating strong performance and sample efficiency on simple regression and classic CIFAR image classification tasks.