Large language models (LLMs) excel at mathematical reasoning benchmarks, but the underlying skills that drive their success remain poorly understood. A new framework called SHAPE aims to change that by dissecting chain-of-thought (CoT) reasoning through two lenses borrowed from mathematics education: semantic spaces and heuristics.
Semantic spaces refer to the model's evolving mathematical interpretations of a problem—such as algebraic or geometric perspectives—while heuristics are the specific actions taken within those spaces, like simplifying the problem or working backward. By applying SHAPE to various models, the researchers found that the heuristics a model employs are better predictors of final answer correctness than traditional CoT features.
Interestingly, models that concentrate their reasoning within a few semantic spaces—rather than exploring many disparate ones—are more likely to arrive at correct solutions, a pattern that mirrors human behavior. The framework also reveals that reinforcement learning during post-training induces mode-seeking in heuristic usage, meaning models tend to favor certain heuristics over others.
To improve post-training, the team experimented with promoting diverse heuristics, which demonstrated effectiveness in boosting accuracy. SHAPE offers a theoretically grounded diagnostic tool for decoding LLM reasoning and opens new avenues for enhancing mathematical capabilities in LLMs. The code is available on GitHub.