Tokenizer gains come from readout and granularity rather than just roots

Researchers found that the performance gains of morphology-aware tokenizers over byte-pair encoding (BPE) often stem from sub-token readout and granularity rather than linguistic structure.

The study analyzed Amharic, a Semitic language with a complex Ge'ez abugida script, and Hausa, a language written in the Latin alphabet. The team separated three factors in their comparison: granularity, root assignment, and word-level readout.

They discovered that while correct root assignment improved named-entity recognition in Amharic, the frequency of those roots explained very little of the improvement. In small models built from scratch, the choice of how the model reads out tokens changed the F1 score by as much as 13.4 points. This shift was large enough to reverse the results of every comparison between root-and-pattern tokenizers and BPE.

In larger pretrained encoders, the readout choice had a much smaller impact, moving the F1 score by less than one point. The study also found that the effects of granularity changed depending on the specific language and implementation used. The authors suggest that researchers must report the readout method and decompose tokenizer effects before claiming that improvements are due to linguistic units.