arm64 build broken on main: i2_s sources never wired into CMake; after source fixes llama-cli hangs/SIGSEGV with official i2_s gguf

Environment

macOS 15 (arm64, Apple M-series), Apple clang 21.0.0, CMake 4.4.5, Python 3.12.15
Repo state: current main (cloned 2026-10-31, --recursive)
Command: .venv/bin/python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

Repro

Step 1 — build fails at link (arm64):

src/CMakeLists.txt is currently a 10-line stub that sets variables and calls
include_directories but defines no CMake target, so src/ggml-bitnet-mad.cpp
(which defines quantize_i2_s) and src/ggml-bitnet-lut.cpp are never compiled
into any target. Link fails on the ggml.c type-traits references:

"_dequantize_row_i2_s", referenced from: _type_traits in ggml.c.o
"_quantize_i2_s", ... symbols not found for architecture arm64

Note 3rdparty/llama.cpp/ggml/src/ggml-cpu/CMakeLists.txt only wires in
../../../../src/ggml-bitnet-lut.cpp, so the i2_s quantization unit in
src/ggml-bitnet-mad.cpp has no home on ARM.

Step 2 — after compiling src/ggml-bitnet-mad.cpp into ggml-base (local
target_sources patch), build fails again:

3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:1495:40: error: use of undeclared identifier 'src1_cont'

src1_cont is declared inside another guarded region (~line 1210) and the I2_S
GEMV/GEMM fast path at line ~1490 uses it unguarded.

Step 3 — after all local patches, build succeeds but inference is broken

Local patches that made it compile:

3rdparty/llama.cpp/ggml/src/CMakeLists.txt:

target_sources(ggml-base PRIVATE ../../../../src/ggml-bitnet-mad.cpp)
target_include_directories(ggml-base PRIVATE ../../../../include ../../../../src ../../../../3rdparty/llama.cpp/ggml/src/ggml-cpu)

3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c (~line 1495):

const bool src1_cont = ggml_is_contiguous(src1);  /* LOCAL PATCH */

dequantize_row_i2_s duplicated into src/ggml-bitnet-mad.cpp (it was
defined only in ggml-cpu/quants.c, unreachable from ggml-base).

Then with the official microsoft/BitNet-b1.58-2B-4T-gguf ggml-model-i2_s.gguf:

build/bin/llama-cli -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -n 64 -t 8 -p "..." -ngl 0 -c 2048 -- temp 0.8 → SIGSEGV(11) during prompt processing (exit from run_inference.py)
build/bin/llama-cli -m ... -n 16 -t 4 -p "Hi" -ngl 0 → no crash but
spins >120s producing zero tokens on a 6-core-class Apple Silicon machine

Expected

setup_env.py -q i2_s on arm64 should produce a working llama-cli for the
official i2_s gguf without source-level patching.

Update: root cause of the SIGSEGV found

After further debugging, the SIGSEGV on ARM64 is caused by an activation-pointer
addressing bug in the I2_S fast path in ggml-cpu.c:

// before (wrong): activations are i8 (1 byte), not 4-per-byte packed
const char * src1_col_de = (const char *)wdata + (i11 * nb11 / 4);

// after (correct)
const char * src1_col_de = (const char *)wdata + (i11 * nb11);

The /4 was carried over from the i2 weight addressing (which IS 4-per-byte
packed) but activations are i8_s (1 byte each), so the pointer landed 4x too
early and read out of bounds.

Additional fixes applied (all in the PR linked below):

src1_cont undeclared in the I2_S fast path (declared in another guarded scope).
dequantize_row_i2_s reads past the packed data on the final partial block
(out-of-bounds read).
src/ggml-bitnet-mad.cpp had no build target (CMakeLists.txt stub) →
quantize_i2_s/dequantize_row_i2_s undefined on arm64.
run_inference.py hung in the interactive loop on EOF stdin → added
--single-turn.

Remaining issue: degenerate output

Even with the crash fixed, the model produces degenerate output (@@@@...) for
all test prompts. The matmul kernel ggml_vec_dot_i2_i8_s (scalar fallback on
ARM) may be using a different bit-packing convention than Microsoft's published
quantized model, or the activation scale convention differs. I could not
determine the correct convention without a reference implementation or the
original f32 model to re-quantize and compare.

PR: #683 (build + runtime fixes; output
quality still degenerate, documented in the PR).

Happy to iterate with guidance on the expected i2_s layout on ARM.

Update: degenerate output is a known upstream issue

The remaining degenerate output (@@@@...) matches the existing reported issue
#348 "Incorrect Output or Generating Random Characters" (macOS, Apple Silicon,
i2_s quantization). So the numerical kernel correctness problem is pre-existing
upstream, not introduced by my build fixes.

My PR #683 addresses the build + crash/hang fixes (CMake wiring, src1_cont,
src1_col_de /4 addressing bug, dequantize bounds, --single-turn) which are
independently correct and unblock compilation on ARM64. The numerical correctness
of the i2_s matmul on ARM remains the open question tracked by #348.

Happy to help test any kernel fix the maintainers land for #348 on Apple Silicon.

Navigation Menu

Repository navigation

arm64 build broken on main: i2_s sources never wired into CMake; after source fixes llama-cli hangs/SIGSEGV with official i2_s gguf #682

Environment

Repro

Expected

Update: root cause of the SIGSEGV found

Remaining issue: degenerate output

Update: degenerate output is a known upstream issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

arm64 build broken on main: i2_s sources never wired into CMake; after source fixes llama-cli hangs/SIGSEGV with official i2_s gguf #682

Description

Environment

Repro

Expected

Activity

hari2353 commented on Nov 1, 2026

Update: root cause of the SIGSEGV found

Remaining issue: degenerate output

hari2353 commented on Nov 1, 2026

Update: degenerate output is a known upstream issue

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions