Large language models are increasingly deployed as interactive agents and behavioral simulators, but a new study questions whether they genuinely recover latent sequential structure or merely reproduce surface-level action frequencies. The distinction matters for simulation work, where actions are typically shaped by prior context rather than marginal frequencies alone.
The research, posted on Hugging Face, uses two controlled settings: two-player Rock-Paper-Scissors interactions and a one-player stochastic n-gram continuation task. Across these experiments, the authors test whether LLMs can identify latent strategies, follow simple Markov rules, and sustain higher-order conditional dependencies.
The framework deliberately separates distribution matching from conditional rule following — a methodological choice that lets the researchers distinguish surface-level statistical alignment from genuine mechanistic understanding.
The results are sobering on three fronts: longer context does not improve strategy identification, correct recognition of a strategy does not guarantee faithful simulation of it, and higher-order dependencies substantially degrade rule recovery. In other words, apparent behavioral fidelity can mask incorrect generative mechanisms.
The authors argue these findings challenge how LLMs are evaluated as behavioral simulators and interactive agents, suggesting that current benchmarks may reward surface-level pattern matching while missing deeper failures in sequential reasoning.