Large language models are increasingly used as agents, but reliable decision-making at inference time—such as whether to defer to a stronger model, call a tool, or abstain—remains challenging. Existing methods rely on prompt-level routing or external orchestration, which are costly and hard to maintain as models evolve.
In a new paper, Hugging Face researchers propose Multi-Head Latent Control (MHLC), a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to generate control signals. Two specialized heads are trained on latent traces from the same backbone: a Capability Head predicts whether the model can solve the instance or should defer, and a Resolution Head decides among Clarification, Tool Use, Abstention, or Direct Answering.
The method enables post hoc adaptation without modifying the model. In routed execution (small + large model), MHLC reduces large-model usage by up to 90.7% on AndroidWorld and 27–53% on average across benchmarks, while retaining most of the large model's performance. It also improves tool-use decision quality, achieving up to +158% relative score gain and 65.5% fewer missed required tool calls.
This approach shifts control from input-side signals to latent generation dynamics, offering a cost-effective way to deploy LLM agents with better quality-cost tradeoffs.