The same Amharic sentence costs 1.94x English on one model and 7.69x on another. Picking on benchmark scores alone can quadruple your bill for identical work. Here is the measured data for eight languages, and the trap that makes the best tokenizer the wrong choice.
If you are building for speakers of an African language, the model-selection advice written for English does not transfer. Benchmark scores will not tell you what you need, and neither will the rate card. This is what to look at instead.
Why the rate card is not the price
A token is not a word. Tokenizers are trained on a corpus, and that corpus decides which sequences of characters earn a short encoding. English earned most of them. A Swahili sentence carrying exactly the meaning of an English one gets cut into more pieces, and you are billed for every piece.
So two models advertising the same dollars-per-million rate can charge you very different amounts for the same job. The rate is flat. Your cost is not.
The measured multipliers
We ran FLORES-200, 1,012 sentences translated by professionals, through each model’s own tokenizer and took the per-sentence ratio against English. Eight languages, 44 of the 46 models we serve. The two that are missing say “not measured” rather than carrying a guess.
| Model | Swahili | Hausa | Yoruba | Amharic |
|---|---|---|---|---|
gemma-3-27b | 1.65x | 1.76x | 2.45x | 1.94x |
gpt-oss-20b | 1.47x | 1.58x | 2.20x | 5.85x |
qwen3-30b | 1.96x | 1.98x | 2.76x | 4.11x |
llama-3.1-8b | 1.95x | 2.01x | 2.77x | 7.69x |
Read the Amharic column twice. Same language, same sentences, and a four-fold spread between models. That is not a rounding difference in a rate card, it is the difference between one bill and four for identical work, decided entirely by a choice most people make on other grounds.
Two things worth knowing before you generalise from this:
- The multiplier is a property of the tokenizer, not of the language. So it does not follow language families, and it changes between models in ways you cannot predict from the script. Hausa and Wolof sit close together around 1.9x on most models while Yoruba sits near 2.8x, and the ordering shifts from tokenizer to tokenizer.
- A per-language cost claim is meaningless without naming the model. “Swahili costs about twice as much” is true on average and useless for a decision.
The trap: a better tokenizer can still cost more
Here is where the obvious conclusion goes wrong.
On the table above gpt-oss-20b tokenizes Swahili 11% better than gemma-3-27b, 1.47x against 1.65x.
So it should be the less expensive choice for a Swahili product.
We measured it on an identical one-line prompt. It cost 6.4 times more per request.
The reason is that gpt-oss-20b reasons before answering, and the reasoning is billed. On a short
reply the thinking dwarfs any saving the tokenizer produced. There is a second, smaller effect too:
every request carries chat-template overhead, about 44 tokens on that model against about 10 on
Gemma, so on short prompts the tokenizer advantage never arrives at all.
Compare per request on your own workload, not per token. If your replies are long the tokenizer advantage starts to matter; if they are one-liners it never will.
A method that works
- Shortlist on the multiplier for your language, not on an English benchmark. Two or three candidates.
- Check whether each one reasons. If your replies are short, prefer a model that does not. If
your task genuinely needs multi-step reasoning, budget for it and raise
max_tokens, because a low ceiling on a reasoning model spends the whole budget thinking and returns an empty string. - Run your ten most typical prompts through each candidate and compare the credits charged, not the tokens. Our playground shows tokens in, tokens out and credits on every turn, taken from the ledger rather than estimated, which is the number you actually pay.
- Check quality in the language, with a speaker. A low-cost model that is wrong in Yoruba has not saved you anything. Nothing in the cost data speaks to output quality and we would not claim otherwise.
Where the data lives
Every multiplier is on the models page, per model and per language, alongside context window, price and measured capabilities. Every model we serve today is open-weight, so you can download the tokenizer and reproduce the measurement against the same public corpus. A number nobody can check is a claim rather than a measurement.
If you have a parallel corpus for a language that is not in the table, tell us and we would rather publish your number than our estimate.