Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations, turning activation analysis into a chat interface. They are often used to audit models that hide information, such as backdoors or concealed goals, by training the oracle on the subject model's activations to become a specialist reader.
However, a new paper from Hugging Face shows that this approach can backfire. In a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure, fine-tuned AOs became concept-specific anti-readers: they selectively failed to recover the concept that was persistently present during their own training.
The failure is not due to the concept being absent from the subject or oracle representations. The target remains decodable inside the oracle, and LogitLens and layer-ablation analyses indicate the issue lies in the AO readout pathway. Ablating the oracle's mid-to-late layers restores recovery, and the failing oracle is usually confident rather than uncertain.
Across all five concepts tested, an AO fine-tuned on a subject that hides concept c became selectively worse at recovering c, while still reading other concepts fine. For cooperative subjects, exact recovery of the hidden concept dropped from 94% for a base AO to 35% for the oracle trained on that subject, even though an oracle trained on a different hidden concept still reached 71%.
The results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.