Researchers at Hugging Face have introduced the Conformal Relevance framework, a novel approach that automates the design of score functions for conformal prediction in natural language processing (NLP). The framework leverages in-context learning (ICL) and ensembling to enhance conciseness while maintaining coverage across a range of retrieval-based tasks.
Many NLP tasks, such as summarization and extractive question answering, involve retrieving relevant content from documents under two competing constraints: coverage (retaining enough pertinent information to achieve a goal) and conciseness (removing as much irrelevant information as possible). Conformal prediction methods have been used to guarantee coverage, but optimizing for conciseness requires careful design of a score function.
Traditional state-of-the-art scoring functions rely on hand-engineered LLM prompts that ask the model to rate content importance—a labor-intensive and task-specific process. The Conformal Relevance framework instead uses curated ICL examples and ensembling to create a score function that maintains coverage while improving conciseness with minimal manual input.
The researchers demonstrated the framework on seven NLP tasks and also provided a theoretical analysis of ensembled conformal scores. They introduce a complementarity condition that characterizes when ensembling improves worst-case sentence scores, along with a saturation bound on the potential improvement from ensembling.
This work suggests that many NLP tasks can be conformalized using a single prompt format based on ICL, rather than requiring task-specific manual crafting, potentially simplifying the deployment of conformal prediction in language applications.