Routing inference work across multiple large language models has become a common way to push the cost-quality trade-off of LLM serving. Most production systems route at the session or query level, but recent algorithmic research suggests that routing decisions made at the level of individual tokens can deliver meaningful gains in both efficiency and output quality.
Serving such fine-grained routing, however, is difficult for today's infrastructure. Existing systems are built around the assumption of a single LLM, and the paper reports that under token-level routing they suffer from severe step desynchronization and frequent batch admission delays, while also placing a heavy implementation burden on developers.
To address this, the authors propose TokenRouter, described as an efficient and developer-friendly serving system for token-level routed inference. Its design follows a principle the researchers call request-centric programming, model-centric execution: developers express routing logic from the viewpoint of a single request, while the runtime spins up a dedicated subserver for each LLM and dispatches requests asynchronously.
Each subserver uses a delayed-batching scheduler whose optimal hyperparameters are derived from a mathematical throughput model of the system. According to the paper, TokenRouter achieves 2.01–64.15x higher decoding throughput than existing systems across a range of routing algorithms, workloads, and model pairs.
The authors frame the work as advancing the serving efficiency of token-level LLM routing, and note that the code is available at github.com/thu-nics/TokenRouter.