Official BitNet-b1.58-2B-4T GGUF has no tokenizer.ggml.pre: llama.cpp falls back to the default pre-tokenizer and tokenizes differently from the official tokenizer #684

Summary

The official GGUF microsoft/bitnet-b1.58-2B-4T-gguf (ggml-model-i2_s.gguf, HF revision a1f2f1c, still the current one) has no tokenizer.ggml.pre key. When it loads, llama.cpp falls back to the default pre-tokenizer and logs:

missing pre-tokenizer type, using: 'default'
GENERATION QUALITY WILL BE DEGRADED!

The model uses the LLaMA 3 tokenizer, which needs the llama-bpe pre-tokenizer. With default, numbers, $ and contractions are split differently from the official tokenizer.json.

Reproduction

Setup: bitnet.cpp 404980e, built with setup_env.py -q i2_s, on an Apple M1 (macOS). The official GGUF above was used unchanged.

Sentence: In 1999, 1234567 people paid $3.50 each; it's 2,560 x 6912.

On the whole WikiText-2 test set (wiki.test.raw):

• The HF tokenizer gives 289,077 tokens.
• The GGUF as distributed gives 325,588 tokens through llama-cli, 12.6% more. The llama-cli count is one token higher than llama-tokenize for every model we checked. BitNet-b1.58-2B-4T: bitnet.cpp perplexity ~6x higher than the reference implementation on the same tokens (x86 and aarch64) #639 reports 325,587 for the same file with llama-tokenize, which suggests the token ids measured there came from the default pre-tokenizer.
• With --override-kv tokenizer.ggml.pre=str:llama-bpe, the count matches the HF tokenizer (289,078 through llama-cli).

Effect on perplexity

Protocol: WikiText-2, first 40 chunks of 512 tokens, llama-perplexity protocol, bitnet.cpp 404980e.

The first number scores a different token sequence, so it is not directly comparable per token. Still, with the right pre-tokenizer bitnet.cpp matches the reference within 0.04%.

Suggested fix

Add tokenizer.ggml.pre = "llama-bpe" to the published GGUF, and to whatever pipeline produces it. Until then, passing --override-kv tokenizer.ggml.pre=str:llama-bpe restores the official tokenization.