Retrieval has become a critical component as LLM agents take on complex tasks with varied information needs. Because retrieval depends on an index that represents each document through index keys, the quality of results hinges on how well those keys expose the knowledge inside each document.
According to the paper, effective index representations differ from one retrieval environment to another, which makes any single fixed optimization strategy unreliable. Today, adapting an index to its environment is still largely a manual process: people must diagnose retrieval failures, refine the optimization strategy, and reprocess the index.
To address this, the authors propose SELF-INDEX, a framework designed to let an index self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the index keys responsible, and validates each revision before updating the index.
Beyond reacting to observed retrieval demands, SELF-INDEX uses a Query Simulator to proactively explore additional demands, allowing the index to evolve past the queries already available for optimization.
The paper reports that SELF-INDEX consistently improves retrieval performance across diverse corpora and retrievers, outperforming existing index optimization methods. The authors also state that the benefits carry over to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.