Researchers from Hugging Face have introduced a new framework for evaluating and optimizing document sets used by large language models (LLMs) and AI agents. The paper, titled "Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking," argues that traditional evaluation metrics like nDCG score documents independently, ignoring critical inter-document interactions such as redundancy, conflict, and complementarity.
The proposed framework, called SetwiseEvalKit, is a three-level, nine-dimension benchmark covering both short-form and long-form scenarios. It comprises approximately 28,000 high-quality evaluation rubrics, each tailored to a specific query. The researchers systematically evaluated 12 rerankers and found that even the best method achieves no more than 45% coverage, with cross-document coordination dimensions universally weak. No single method maintains top performance across both short-form and long-form settings.
Building on these insights, the team developed Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals. According to the paper, Rubric4Setwise achieves the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both short-form and long-form scenarios, validating the effectiveness of closing the loop from evaluation to optimization.
The authors note that each query's rubric is tailor-made, generated using the query and answer page, ensuring relevance. The work addresses a common pain point in RAG pipelines: individual retrieval scores may look fine, but the document set is often full of redundancy and contradiction. By scoring the set as a set, the framework provides a more accurate measure of quality.