Repository navigation
bitnet-b1.58-2B-4T runs FFN with SiLU instead of relu2 - wrong logits on every backend #602
Description
bitnet-b1.58-2B-4T runs the FFN with SiLU instead of relu² — wrong logits on every backend
Summary
The official microsoft/bitnet-b1.58-2B-4T model declares "hidden_act": "relu2" in its config.json, but the BitNet.cpp runtime never reads it: the FFN graph is built with a hardcoded LLM_FFN_SILU. The model therefore runs with the wrong activation function on every backend (CPU and GPU), producing wrong-but-finite output — the same failure class as #586.
Root cause
src/models/bitnet.cpp:133—build_ffn(...)is called with a hardcodedLLM_FFN_SILU.src/llama-model.cpp— theLLM_FFN_OP_TYPES_FROM_STRINGmap has"relu"but no"relu2", andllm_ffn_op_type_from_string()is never wired into the bitnet arch.utils/convert-hf-to-gguf-bitnet.py/utils/convert-ms-to-gguf-bitnet.py— neither converter writes<arch>.hidden_activation, so the existing plumbing (llama_hparams::llm_ffn_op,LLM_KV_HIDDEN_ACT,LLM_FFN_RELU_SQRinsrc/llama-graph.cpp) is never exercised for bitnet models.
The graph builder already implements LLM_FFN_RELU_SQR (relu → sqr, src/llama-graph.cpp:1690); with LLM_FFN_PAR gating it produces exactly relu²(gate) * up, matching the HF BitnetMLP (act_fn(gate) * up).
Fix (attached patches)
src/llama-model.cpp: add{ "relu2", LLM_FFN_RELU_SQR }toLLM_FFN_OP_TYPES_FROM_STRING.src/models/bitnet.cpp: read<arch>.hidden_activationinload_arch_hparams(mirroringllama_model_modern_bert), defaulting toLLM_FFN_SILUfor backward compatibility with existing GGUF files; usehparams.llm_ffn_opin the graph.- Converters: write
add_hidden_act("relu2").
Verification
- Convert
microsoft/bitnet-b1.58-2B-4Twith the patched converter;gguf-dumpshould showbitnet_b158.hidden_activation = "relu2". - Compare logits before/after: with the fix the model produces coherent text; without it, output is wrong-but-finite.
- Old GGUF files (no key) keep the previous SiLU behavior — no regression.
Related
- Fix i2_s ARM CPU kernel: correct 128/32 weight-group layout (fixes #585) #586 — same wrong-but-finite class (ARM i2_s layout).
Activity
Fix broken inference (ReLU^2 FFN), one-button build & data pipeline
shifulegend commentedon Aug 23, 2026
Great find. We hit a very similar architecture drift issue when implementing the 1.58-bit FFN block for Project Zero. Because so many 1.58-bit models diverge from the standard LLaMA topologies, hardcoding
LLM_FFN_SILUis an easy trap to fall into, so we actually solved this by just parsing thehidden_actkey directly from the metadata into the C99 graph builder and failing fast on unknown activations. It forces the converter to stay honest, which saves you from these silent wrong-but-finite errors. Glad you caught this, it explains the degraded perplexity we were seeing.paulanthonyhebbes-sudo commentedon Oct 12, 2026
Confirming this on Linux as well, with the published GGUF and the same pinned submodule.
Environment
390c30775(build b9918)microsoft/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.ggufBefore (SiLU), with the corrected template (
User: …<|eot_id|>Assistant:via--jinja --chat-template-file) andtokenizer.ggml.pre=str:llama-bpealready applied: answers started correctly, then repeated until the token limit (finish_reason: "length"). Token-level output from/completionwithn_probsshowed the model never emitting<|eot_id|>(128009). After "Paris." it continued with791("The").After applying the same narrow workaround as homeofe (@homeofe) (
LLM_FFN_RELU_SQRforLLM_ARCH_BITNET_B158/LLM_TYPE_2B) and rebuilding: the model emits128009straight after "Paris." and stops (finish_reason: "stop"). It also now answers the bat-and-ball question correctly ($0.05) with correct working.A note for anyone else reproducing this: in this llama.cpp version, passing
--override-kvmore than once keeps only the last value (there's a DEPRECATED warning at startup). Use one flag with comma-separated values, e.g.--override-kv tokenizer.ggml.eos_token_id=int:128009,tokenizer.ggml.pre=str:llama-bpe.bitnet-b1.58: squared-ReLU FFN (hidden_act = relu2) instead of SiLU