tokenization

Reframing tokenization as a core modeling decision in LLMs rather than a preprocessing step, arguing for context-aware tokenizer and model co-design.

A new metric for multilingual tokenization evaluation that goes beyond fertility-based measures.