Beyond Fertility: STRR as a Metric for Multilingual Tokenization Evaluation

Abstract

We propose STRR as a metric for multilingual tokenization evaluation that goes beyond fertility-based measures, providing a more comprehensive assessment of tokenizer quality across languages.

Publication
NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling