Sparse autoencoders (SAEs) are widely used to probe the internal representations of large language models, yet it remains unclear what kind of linguistic structure their latents actually expose. A new paper accepted at EMNLP 2026, titled Parts-of-Speech as Emergent Categories in SAE Latent Space, uses part-of-speech (PoS) categories as a controlled test case to examine whether morpho-syntactic information is encoded by individual latents or by structured groups of features.
Working with SAE activations from LLaMA-3-8B on the UD English GUM treebank, the authors report that PoS distinctions are highly recoverable from SAE activations. Crucially, this recoverability is not reducible to lexical memorisation, meaning the signal reflects grammatical structure rather than simply remembering which words tend to appear together.
The study finds no one-latent-per-tag mapping. Instead, each PoS category relies on a compact group of latents, and just 498 latents out of roughly 130,000 are enough for multi-class PoS classification. The authors also observe a clear split between category types: closed classes such as DET, PRON, and CCONJ use small, focused groups of latents, while open classes such as NOUN, ADJ, and ADV spread across broader ones.
These latent groups remain stable on held-out data, but they also overlap between related categories, suggesting they additionally capture surface form and contextual information. The authors conclude that SAEs localise morpho-syntactic information in a distributed and category-dependent form rather than through atomic grammatical features.
The paper's code and dataset are publicly released. The authors invite discussion on whether the findings would hold under causal interventions or with other SAE architectures.