[eejd]

Draft candidate for #3; stacked on fix/macos-arm64-build (the first 5 commits are that PR). Only the last five commits are new.

• ggml-cpu: I2_S NEON kernel for the canonical layout – patches/llama.cpp/0002, the quants.c hunk of ggml-cpu: fix ARM i2_s dot product producing constant output isHuangXin/llama.cpp#6 verbatim. The pinned submodule used a scalar fallback on ARM that (a) decodes weights sequentially instead of the 128/32 interleaved layout, (b) maps codes to −1/0/+1 although callers subtract the activation sum, (c) implements only the by==0 calling convention so prompt-processing GEMM leaves most outputs unwritten.
• bitnet-b1.58: squared-ReLU FFN – patches/llama.cpp/0003; hidden_act of 2B4T is relu2, the graph hard-codes SiLU ([Bug]: BitNet FFN uses SILU instead of ReLU² — perplexity 99.8 vs 17.1 on every CPU backend microsoft/BitNet#588/bitnet-b1.58-2B-4T runs FFN with SiLU instead of relu2 - wrong logits on every backend microsoft/BitNet#602). Selecting on arch is not a complete design (1bitLLM checkpoints share the arch but use SiLU); the config-driven key of feat: support config-based FFN activation for BitNet isHuangXin/llama.cpp#4 is the proper fix.
• ggml: llama-quantize I2_S emits the canonical layout – patches/llama.cpp/0004. llama-quantize … I2_S (what setup_env.py calls) was invalid ftype 'I2_S', and quantize_i2_s wrote the sequential layout. Now: I2_S ftype registered; canonical 128/32 layout; one call per tensor (the scale is per tensor, stored after the packed data, so no row chunking/threads); the 28 unused trailer bytes are zeroed; ggml_validate_row_data accepts I2_S.
• llama-vocab: assume llama-bpe for the official bitnet-b1.58 GGUF – patches/llama.cpp/0005. The official GGUF has the Llama-3 BPE vocabulary but no tokenizer.ggml.pre, so llama.cpp used the generic default regexes and split e.g. (n):\n differently from Llama-3 (def fibonacci(n):\n → ids 7,77,1680,198 instead of 1471,997). The fallback applies only when arch is bitnet-b1.58, the vocab has 128256 tokens and 280147 merges (merge-table sha identical to Llama-3.2-1B's) and the key is missing; any other file keeps the old warning (checked with a Llama-3.2 GGUF whose key was blanked). The converter already writes llama-bpe for this tokenizer (its check-hash equals Meta-Llama-3's; verified on the HF tokenizer), so only existing files needed this.
• docs.

Results (Apple M6, -ngl 0, official ggml-model-i2_s.gguf vs mlx_lm float32 mlx-community/bitnet-b1.58-2B-4T, gate from the test/i2s-correctness-gate PR)

build	gate	teacher-forced (prompts)	perplexity gate text (ref 7.1532)
fix/macos-arm64-build (unpatched kernels)	FAIL	NaN/Inf in 10.2 M logits	NaN
+ 0002 only (llama.cpp#6 or #2)	FAIL	top-1 0.600, mean KL 1.081	43.9906
+ 0002 + 0003	PASS (needs --override-pre llama-bpe)	top-1 1.000, mean KL 0.0026, max KL 0.038, min cos 0.996	7.1785 (+0.35 %)
+ 0002..0005 (this branch), unmodified official GGUF, no override	PASS, tokenizer parity 5/5	same	same

The capital of France is → Paris. Paris is a city that is known for its rich history, culture, (identical to the reference); 3 of 5 greedy continuations are identical over 16 tokens, the other two diverge at reference near-ties (margins 0.028 and 0.174, threshold 0.464). Same result with 1, 4, 8 threads, and with a build without dotprod/i8mm (mean KL 0.0021, ppl 7.1780).

Full run at this head (clean configure + build with the default recipe, patches 0001–0005): kernel tests (linked libggml-cpu) 58 / 58; gate unit tests 24 / 24; synthetic gate PASS; real-model gate PASS with and without the pre-tokenizer override; llama-quantize … I2_S on the synthetic f32 model: all 14 tensors byte-identical to the canonical packing at default/1/4 threads, gate PASS on the quantized file.

Known gaps – not fixed here

• llama-quantize … I2_S was verified on the synthetic model only; no HF master weights of the real model were on disk, so a locally converted 2B4T GGUF has not been run through the gate. quantize_i2_s keeps the BitNet convention (scale = first non-zero |w|), so inputs must already be ternary × scale. Rows must be divisible by 128 (asserted).
• 0003 selects squared-ReLU by arch, which is wrong for 1bitLLM checkpoints (SiLU).
• The --mad variants of the kernel test (src/ggml-bitnet-mad.cpp, not compiled by the CMake build) still fail 0 / 29: dead code with the old kernels.
• One model, one host, one gate text; the real-model tolerances were not calibrated before the run (see tolerances.json history).

Files

• CMakeLists.txt, cmake/ApplyLlamaPatches.cmake, docs/macos-arm64.md
• patches/llama.cpp/0001 (src1_cont), 0002 (I2_S NEON kernel), 0003 (relu²), 0004 (llama-quantize I2_S), 0005 (llama-bpe fallback)