Repository navigation
fix: macOS arm64 build - static libs, src1_cont, CPU-only backends, docs (#2) - #5
Merged
Merged
Conversation
ggml.c (libggml-base) references quantize_i2_s and dequantize_row_i2_s,
which live in ggml-cpu/quants.c (libggml-cpu). ld64 rejects undefined
symbols in a shared library, so the default shared build fails at
libggml-base with:
Undefined symbols for architecture arm64:
"_dequantize_row_i2_s", referenced from: _type_traits in ggml.c.o
"_quantize_i2_s", referenced from: _ggml_quantize_chunk in ggml.c.o
Default BUILD_SHARED_LIBS to OFF on Apple when the user did not set it.
Refs microsoft#682, microsoft#641, microsoft#635 (reproduced on Apple M6 /
Apple clang 21.0.0, see #2).
…d_mul_mat ggml-cpu.c #undefs GGML_USE_LLAMAFILE when __ARM_FEATURE_MATMUL_INT8 or __ARM_FEATURE_SVE is defined, which compiles out the only declaration of src1_cont. The I2_S GEMV/GEMM fast path uses it unconditionally, so every arm64 CPU with i8mm (M2 and later, Graviton3, ...) fails with: ggml-cpu.c:1495:40: error: use of undeclared identifier 'src1_cont' The submodule (isHuangXin/llama.cpp) is not ours to modify, so carry the fix as patches/llama.cpp/0001-*.patch and apply it at configure time with cmake/ApplyLlamaPatches.cmake (idempotent; BITNET_APPLY_LLAMA_PATCHES=OFF to disable). The same one-line change is in isHuangXin/llama.cpp#1 and #6. Refs microsoft#618, microsoft#682, microsoft#641 (reproduced on Apple M6 / Apple clang 21.0.0, see #2).
…S backend)
bitnet.cpp's I2_S/TL1 kernels exist only for the CPU backend, but on macOS
ggml also builds the Metal and Accelerate (BLAS) backends, and both
interfere with I2_S models even when the user passes -ngl 0:
* Metal built, BLAS not: creating the llama_context aborts in
ggml_backend_sched_split_graph:
ggml.c:1789: GGML_ASSERT(view_src == NULL || data_size == 0 ||
data_size + view_offs <= ggml_nbytes(view_src)) failed
* BLAS built: ggml_backend_blas_device_supports_op accepts MUL_MAT whose
src0 has a to_float() (I2_S has one) once ne0, ne1 and ne10 are >= 32, so
every prompt batch of >= 32 tokens is routed to ggml_backend_blas_graph_compute
and the process dies with SIGSEGV (backtrace in the evidence log).
Default GGML_METAL and GGML_BLAS to OFF on Apple unless the user sets them.
Measured on an Apple M6 with a synthetic I2_S model: Metal+BLAS (old default)
crashes for >= 31-token batches; Metal OFF + BLAS OFF runs 1..300 tokens at
1 and 4 threads.
Refs microsoft#682 (SIGSEGV during prompt processing), microsoft#641 (Metal
crash on I2_S), #2
Metal+BLAS: SIGSEGV in ggml_backend_blas_graph_compute (>= 31 tokens); Metal only: GGML_ASSERT at context creation; BLAS only: no crash, but I2_S batches >= 32 tokens return garbage (perplexity 4.4e27 vs 3.7e5 on the synthetic model with the corrected kernels). The previous docs text implied a SIGSEGV for every BLAS build. Refs #2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Part of #1, fixes the build half of #2. Does not make I2_S output correct on ARM (that is #3, see the stacked/related PRs).
What changes (one commit per fix)
build: default to static ggml libraries on macOS–libggml-basereferencesquantize_i2_s/dequantize_row_i2_sthat live inlibggml-cpu; ld64 rejects the shared build (microsoft/BitNet arm64 build broken on main: i2_s sources never wired into CMake; after source fixes llama-cli hangs/SIGSEGV with official i2_s gguf microsoft/BitNet#682, macOS / Apple Silicon: setup_env.py build fails (ggml-base can't find i2_s symbols); after working around it, Metal crashes and CPU output is garbage microsoft/BitNet#641, fix(build): add CPU backend check for tensor transforms and resolve macOS/Windows I2_S linker symbols microsoft/BitNet#635).build: declare src1_cont in the I2_S fast path– one-line patch for the submodule, carried aspatches/llama.cpp/0001-*.patchand applied idempotently at configure time bycmake/ApplyLlamaPatches.cmake(microsoft/BitNet ggml-cpu.c:src1_contundeclared → build fails on ARM CPUs with i8mm (Apple M2/M3/M4, Graviton3, ...) microsoft/BitNet#618, arm64 build broken on main: i2_s sources never wired into CMake; after source fixes llama-cli hangs/SIGSEGV with official i2_s gguf microsoft/BitNet#682, macOS / Apple Silicon: setup_env.py build fails (ggml-base can't find i2_s symbols); after working around it, Metal crashes and CPU output is garbage microsoft/BitNet#641).3rdparty/llama.cppisisHuangXin/llama.cpp, which this repo cannot push to.build: CPU-only ggml on macOS by default– Metal built (BLAS off) →GGML_ASSERTinggml_backend_sched_split_graphwhen the context is created; Metal + Accelerate/BLAS built →ggml_backend_blasclaims I2_SMUL_MATfor batches ≥ 32 tokens and the process dies with SIGSEGV (what arm64 build broken on main: i2_s sources never wired into CMake; after source fixes llama-cli hangs/SIGSEGV with official i2_s gguf microsoft/BitNet#682 reports as "SIGSEGV during prompt processing"); BLAS built, Metal off → no crash, but those batches return garbage (synthetic gate with corrected kernels: ppl 4e27 vs 3.7e5). The commit message of this commit says "SIGSEGV" for BLAS in general; the precise matrix is indocs/macos-arm64.md(follow-up commit).docs: macOS arm64 build recipe–docs/macos-arm64.md(conda-free, out-of-tree build, reproduction notes).Evidence (Apple M6, macOS 27.0.1, Apple clang 21.0.0, CMake 3.31)
_quantize_i2_s,_dequantize_row_i2_s)setup_env.pyflags, shared libs)src1_contundeclaredggml-cpu.c:1495:40(flags-mcpu=native+dotprod+i8mm+nosve+sme)-ngl 0; backtrace ends inggml_backend_blas_graph_compute. The ">120 s spin on-p Hi" was not reproducedsrc1_contGGML_ASSERTat context creation; with BLAS it is the SIGSEGV above@@@@outputThe capital of France is@@@@@@@@@@@@@@@@(cause: #3)tensor->backendguard does not apply (this ggml has no such field); the submodule bump was not used (static libs instead)gguf0.19.0 shadows the submodule's →AttributeError ... 'MODEL_ARCH' has no attribute 'BITNET_B158';PYTHONPATH=3rdparty/llama.cpp/gguf-pyfixes it, no conda neededClean clone of this branch, out-of-tree build with the commands in
docs/macos-arm64.md(default UI on): configure + build rc 0;llama-cli --version→version: 9918 (390c30775) built with AppleClang 21.0.0.21000334 for Darwin arm64; officialggml-model-i2_s.ggufloads and generates tokens (garbage, see above); synthetic I2_S model:llama-benchand 1..300-token batches at 1 and 4 threads run without crash.Not done: Windows/Linux, TL1 (
-DBITNET_ARM_TL1=ON), theBUILD_SHARED_LIBS=ON+ submodule-fix route of microsoft#635.