Hugging Face researchers have introduced Enoki, an open information extraction framework designed to tackle hallucination detection in large language models (LLMs) at multiple levels. The framework aims to bridge the gap between claim-level verification, which provides interpretable factual units, and span-level localization, which identifies unsupported text segments.
Traditional approaches often require separate pipelines for each level, leading to high computational costs due to multiple decomposition and verification calls, as well as additional alignment steps. Enoki addresses this by extracting text-anchored relational facts that serve as a shared representation for both claim-level and span-level tasks. This design eliminates the need for separate claim-to-span alignment, streamlining the detection process.
Enoki supports three extraction regimes—LLM-based, encoder-based, and rule-based—allowing users to balance accuracy and inference cost through a common interface. According to the paper, Enoki remains competitive with strong claim-level systems while using fewer resources, and it achieves superior performance on fine-grained span- and entity-level localization.
Alongside the framework, the team released EnokiQA, a dual-granularity dataset with aligned annotations for both claim-level verification and span-level localization. The dataset is intended to support further research in hallucination detection and factuality checking.