Repository navigation
Conversation
…el needed)
utils/test_gemm_kernel.sh is a timing benchmark on random data and never
checks a result (it also needs Linux-only -fopenmp/clock_gettime/libggml.so).
Add utils/i2s_kernel_test/: a standalone test that compares
L0 dequantize_row_i2_s / quantize_i2_s against the documented 128-weight /
32-byte I2_S layout,
L1 ggml_gemv_i2_i8_s / ggml_gemm_i2_i8_s against sum(code*y),
L2 ggml_mul_mat(I2_S, F32) end to end (int8 activation quantization,
act-sum correction, weight scale) at 1 and 4 threads,
for K/M/N including the BitNet b1.58 2B4T shapes (K=2560, 6912).
run.sh builds it against a static ggml build tree; --mad additionally
compiles src/ggml-bitnet-mad.cpp (not part of the CMake build) so the
kernels patched by microsoft#586/microsoft#633 can be tested.
Refs #3
The first draft used rel-L2 < 0.03 against float W@x. With an activation outlier (amax/rms ~ 20) the int8 absmax quantization alone gives ~4-8 %, so the fixed bound flagged a bit-exact kernel (L2 emulation error ~1e-7). Use 4 * (amax/rms) / (127*sqrt(12)), derived from the quantization step, and keep the primary check (exact emulation, 1e-4) unchanged. Refs #3
…el-level run tolerances.json fixes, with rationale, how close the engine must be to the mlx_lm bitnet reference: teacher-forced top-1 agreement / KL / cosine, free-running greedy prefix, and a perplexity ratio on a fixed text. compare.py implements them; test_compare.py checks on synthetic data that documented int8-activation noise passes and that scrambled weights, constant output, NaN/Inf/1e27 garbage, flattened distributions and perplexity blow-ups fail. No GGUF or reference weights existed on the host when this was written. Refs #3
…estrator, synthetic model, README run_gate.py exits 3 (BLOCKED, with the list of missing files) when the official GGUF or the reference weights are not present locally; it never downloads. engine_logits dumps per-position logits through libllama on the CPU and records llama_print_system_info(). make_synthetic_model.py writes a tiny random BitNet in mlx and GGUF(I2_S) form from the same ternary weights so the plumbing can be tested without the real checkpoint. Refs #3
…capped Documented post-hoc change. v1 used a fixed 0.25-logit near-tie margin. The first synthetic end-to-end run (llama.cpp#6 kernel + relu2 vs mlx_lm) passed every teacher-forced and perplexity check (top-1 0.972, mean KL 0.0028, min cosine 0.9993, |ln ppl ratio| 0.0197 vs bound 0.02) but failed one of five free-running greedy checks: a flip at reference margin 0.446 while the measured margin-error sigma is ~0.15. v2 uses max(0.25, 4*sigma) with sigma measured in the same run, and fails outright if sigma > 10 % of the reference logit std, so an engine that is wrong by O(1) can never excuse itself (unit-tested). When the cap is exceeded the band falls back to the 0.25 floor so top-1 agreement stays meaningful. Everything else (KL, cosine, perplexity bounds) is unchanged. v1 results are kept in the evidence directory. Refs #3
The official ggml-model-i2_s.gguf has no tokenizer.ggml.pre key, so llama.cpp falls back to the 'default' pre-tokenizer and splits e.g. "(n):\n" differently from the Llama-3 tokenizer the model was trained with (engine ids 7,77,1680,198 vs reference 1471,997 for 'def fibonacci(n):\n'). With tokenizer.ggml.pre=llama-bpe the engine reproduces the reference ids exactly (also checked against a Llama-3.2 GGUF that carries the key). The tokenizer-parity check therefore fails on the unmodified official GGUF; the gate can pass --override-pre llama-bpe to the engine to judge the numerics separately. Refs #3
…tize tests synthetic-f32.gguf carries the same ternary*scale values as float so that 'llama-quantize --token-embedding-type f16 f32.gguf out.gguf I2_S 1' (what setup_env.py does) can be compared byte-for-byte with the canonical I2_S packing in synthetic-i2s.gguf.
Evaluation branch only. Diff taken verbatim from 'gh pr diff 586 --repo microsoft/BitNet'.
3 tasks
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evaluation only – do not merge.
gh pr diff 586 --repo microsoft/BitNetapplied unmodified on top oftest/i2s-correctness-gate(kernel tests + gate).Result on Apple M6 (
utils/i2s_kernel_test/run.sh --build-dir <static build> --mad):src/ggml-bitnet-mad.cppcompiled into the test: baseline 0 / 29 pass → with Fix i2_s ARM CPU kernel: correct 128/32 weight-group layout (fixes #585) microsoft/BitNet#586 26 / 29 (dotprod and NEON-only variants); the 3 failures arequantize_i2_slayout (the PR's own "follow-up" note).src/ggml-bitnet-mad.cppat all (src/CMakeLists.txtonly sets variables; the submodule'sggml-cpu/CMakeLists.txtlists onlyggml-bitnet-lut.cpp), so applied to the real build tree the linked-library tests are unchanged (3 / 55) and model output is unchanged. Fix i2_s ARM CPU kernel: correct 128/32 weight-group layout (fixes #585) microsoft/BitNet#586 only helps oncemad.cppis actually the kernel thatggml_vec_dot_i2_i8_sresolves to (the submodule currently defines its own, wrong, ARM fallback inggml-cpu/quants.c).