Activity

1. matheus-c-feitosa commented on Oct 16, 2026

   

   Matheus C. Feitosa (matheus-c-feitosa)

   on Oct 16, 2026

   Last edited by matheus-c-feitosa

   More actions

   Short answer: the GGUF header can rule a file out, but it can't prove the file has the same format, because the bit order inside an I2_S tensor is not recorded anywhere in the file.

   What the header can check (no need to load the model):

   • The weight matrices have ggml type id 36, which is I2_S in bitnet.cpp's llama.cpp fork. ggml-model-i2_s.gguf has 210 of them, plus 121 F32 tensors and 1 F16.
   • Each I2_S tensor takes exactly n/4 + 32 bytes: 2 bits per weight, then 32 bytes that hold the scale. For example, blk.0.ffn_down.weight has 17,694,720 weights and takes 4,423,712 bytes.
   • The type id alone isn't proof, and neither is general.file_type: in mainline llama.cpp, id 36 was IQ4_NL_4_4 (since removed), and the official file has file_type 40 while the gguf-py in bitnet.cpp's submodule defines MOSTLY_I2_S = 41.
   • general.architecture tells you which model graph runs the file, not the format: bitnet-b1.58 here (bitnet.cpp's converter also uses it for 1bitLLM's models), llama in the Falcon3 1.58-bit i2_s files.

   What the header can't check: in src/ggml-bitnet-mad.cpp (main, 0b341e5), quantize_i2_s packs groups of 128 weights in 32 bytes on x86 and groups of 64 weights in 16 bytes on ARM (and a third way on x86 if ACT_PARALLEL is turned off in include/gemm-config.h). setup_env.py makes i2_s files with that llama-quantize, so the same model quantized on an x86 machine and on an ARM machine should give files with identical headers and sizes but different bytes. I read this in the code; I haven't quantized on ARM to compare. The official ggml-model-i2_s.gguf uses the 128/32 layout. #585 describes both layouts, and #412 asks for the format to be documented.

   So the reliable test is to run the file in the runtime you plan to use. A layout mismatch gives garbage: !!!! in #585, random tokens in #470, and ???? with current main in my ARM runs. At 112f853 I measured a perplexity of about 40 million.

   Script for the header checks (Python standard library, about 0.3 s)

   # Does a GGUF file use the same layout as ggml-model-i2_s.gguf?
   # Reads only the header and tensor table. Python standard library only.
   import os, struct, sys
   from collections import Counter
   
   f = open(sys.argv[1], 'rb')
   rd = lambda fmt: struct.unpack('<' + fmt, f.read(struct.calcsize('<' + fmt)))[0]
   rs = lambda: f.read(rd('Q')).decode('utf-8', 'replace')
   SCALAR = {0: 'B', 1: 'b', 2: 'H', 3: 'h', 4: 'I', 5: 'i', 6: 'f', 7: '?',
             10: 'Q', 11: 'q', 12: 'd'}
   
   def value(t):
       if t == 8: return rs()
       if t == 9:  # array: skipped
           et, n = rd('I'), rd('Q')
           if et in SCALAR: f.seek(n * struct.calcsize(SCALAR[et]), 1)
           else: [value(et) for _ in range(n)]
           return None
       return rd(SCALAR[t])
   
   assert f.read(4) == b'GGUF', 'not a GGUF file'
   version, n_tensors, n_kv = rd('I'), rd('Q'), rd('Q')
   meta = {}
   for _ in range(n_kv):
       key = rs(); meta[key] = value(rd('I'))
   tensors = []
   for _ in range(n_tensors):
       name = rs(); dims = [rd('Q') for _ in range(rd('I'))]
       ggml_type, offset = rd('I'), rd('Q')
       n = 1
       for d in dims: n *= d
       tensors.append((offset, name, ggml_type, n))
   align = meta.get('general.alignment', 32)
   data_start = -(-f.tell() // align) * align
   end = os.path.getsize(sys.argv[1]) - data_start
   pad = lambda x: -(-x // align) * align
   
   tensors.sort()
   print('architecture:', meta.get('general.architecture'))
   print('tensor types (ggml type id: count):',
         dict(Counter(t[2] for t in tensors).most_common()))
   ok = bad = 0
   for (off, name, ggml_type, n), nxt in zip(tensors, tensors[1:] + [(end,)]):
       if ggml_type != 36: continue
       if nxt[0] - off == pad(n // 4 + 32): ok += 1
       else: bad += 1; print('  size mismatch:', name)
   print(f'type 36 tensors with the I2_S size (n/4 + 32 bytes): {ok} of {ok + bad}')

   Output for ggml-model-i2_s.gguf (sha256 4221b252…):

   architecture: bitnet-b1.58
   tensor types (ggml type id: count): {36: 210, 0: 121, 1: 1}
   type 36 tensors with the I2_S size (n/4 + 32 bytes): 210 of 210
   

   The Falcon3-1B and 3B Instruct 1.58-bit i2_s files also pass (126 of 126 and 154 of 154). Other formats show their own type ids and no type 36: the same BitNet model converted to ik_llama.cpp's iq2_bn_r4 (335), Bonsai-1.7B Q1_0 (41), Ternary-Bonsai-1.7B legacy Q2_0 (42) and PQ2_0 (142). A synthetic file with one type 36 tensor at 4 bits per weight is reported as size mismatch.

   (Analysis assisted by Claude Code; the script was run on my machine and the numbers checked by me.)
2. kukuyuer commented on Oct 20, 2026

   

   kukuyuer

   on Oct 20, 2026

   Author

   More actions

   除了文件头外，还要读取内部数据排列才可能准确判定了。
   怪不得其他模型推理输出一直错误。
3. matheus-c-feitosa commented on Oct 21, 2026

   

   Matheus C. Feitosa (matheus-c-feitosa)

   on Oct 21, 2026

   More actions

   Right, the header can only rule a file out. But wrong output isn't necessarily a format problem. On current main (0b341e5), bitnet.cpp gives wrong output even for BitNet-b1.58-2B-4T with the official ggml-model-i2_s.gguf:

   • x86-64: the text looks fluent, but perplexity is about 6× the reference, because the FFN uses SiLU where this model needs ReLU² ([Bug]: BitNet FFN uses SILU instead of ReLU² — perplexity 99.8 vs 17.1 on every CPU backend #588).
   • ARM (aarch64): the output is ???? since 112f853 (ARM I2_S inference produces gibberish/garbage output after commit 112f853 (CPU Optimization update) #470).

   Neither fix is merged yet. What gave correct output in my tests: #605 plus --override-kv bitnet-b1.58.hidden_activation=str:relu2 (on ARM, also isHuangXin/llama.cpp#2), or the one-line change described in #588.

   A quick check that needs no reference model:

   llama-completion -no-cnv -m ggml-model-i2_s.gguf -p "The capital of France is" -n 12 --temp 0
   

   A correct build continues with "Paris. Paris is a city that is known for its rich". Current main on x86 gives "a small city, and the capital of France is a small". The full comparison table is here: https://github.com/matheus-c-feitosa/lowbit-atlas#bitnet-b158-2b-4t-which-fix-actually-works

   Which models give you wrong output, and on which CPU? If you paste the command and the first lines of output, I can tell you which of these cases it is.

   (Assisted by Claude Code; the measurements are mine.)
4. matheus-c-feitosa commented on Oct 21, 2026

   

   Matheus C. Feitosa (matheus-c-feitosa)

   on Oct 21, 2026

   More actions

   简要：输出错误不一定是格式问题。在当前 main 上，官方的 BitNet-b1.58-2B-4T 文件本身在 CPU 上就会输出错误（x86 见 #588，ARM 见 #470）。请告诉我是哪个模型、哪种 CPU，并贴出命令和前几行输出。（中文由 AI 翻译）
5. kukuyuer commented on Oct 26, 2026

   

   kukuyuer

   on Oct 26, 2026

   Author

   More actions

   我是使用fpga处理，BitNet-b1.58-2B-4T可以正常推理（也测试了其他几个Falcon3也正常），但是BitNet-0.6B输出是！！！！。是否BitNet-0.6B不同？
6. matheus-c-feitosa commented on Nov 1, 2026

   

   Matheus C. Feitosa (matheus-c-feitosa)

   on Nov 1, 2026

   More actions

   Sorry for the slow reply, I was away for a few days with university work.

   If by BitNet-0.6B you mean BitNet-embedding-0.6B (the 0.6B model in this repo's README), then yes, it's different in two ways.

   1. It's an embedding model, not a text generator. The I2_S guide says it "does not generate any human-readable texts" (docs/bitnet-embeddings-i2s-guide.md, section 7), and the converter skips output.weight because there is no token generation. Token id 0 in its vocab is !, so a run of ! usually means the argmax keeps returning id 0, which is what you get when the logits are all equal or NaN. What this model should produce is a 1024-dim vector:

   llama-embedding -m bitnet-embeddings-0.6b-bf16-i2_s.gguf -p "query: What is BitNet?" --embd-normalize 2 --embd-output-format array --pooling last
   

   --pooling last is there because the published GGUF declares mean pooling, while the model card says last-token pooling (#640, fix proposed in #659).

   2. The graph is different from BitNet-b1.58-2B-4T (from the GGUF headers):

   	BitNet-b1.58-2B-4T	BitNet-embedding-0.6B
   general.architecture	bitnet-b1.58	qwen3
   Extra norms	attn_sub_norm, ffn_sub_norm	an RMSNorm before each of the 7 ternary matrices (*_norm_in), plus QK-norm (attn_q_norm, attn_k_norm)
   FFN activation	ReLU²	SiLU
   Heads (KV), head_dim	20 (5), 128	16 (8), 128, so Q is 1024→2048 and attn_output is 2048→1024
   Non-ternary tensors	norms F32, token_embd F16	all F16 (norms and token_embd)

   The I2_S packing itself is the same (blocks of 128 weights in 32 bytes, scale at the end), so if your FPGA path runs 2B-4T and Falcon3 correctly, the ternary matmul is probably fine. What's missing is the qwen3 graph and the embedding output. To check it, compare your vector with the CPU one from the command above for the same input: the cosine similarity should be close to 1.

   Is that the model you used? If not, which file was it?

   (Analysis assisted by Claude Code; I read the GGUF headers on my machine.)
7. matheus-c-feitosa commented on Nov 1, 2026

   

   Matheus C. Feitosa (matheus-c-feitosa)

   on Nov 1, 2026

   More actions

   抱歉回复晚了，这几天忙于大学的课业。简要：如果你说的是 BitNet-embedding-0.6B，它是嵌入（embedding）模型，官方指南说明它不生成文本。词表中 id 0 就是 !，所以一串 ! 通常表示 logits 全部相同或为 NaN。应使用 llama-embedding（加 --pooling last，见 #640），输出 1024 维向量。另外它的计算图是 qwen3（每个三值矩阵前有 RMSNorm、QK-norm、SiLU、Q 为 1024→2048），与 2B-4T 不同；I2_S 打包方式相同。请确认你用的是哪个文件。（中文由 AI 翻译）
8. kukuyuer commented on Nov 3, 2026

   

   kukuyuer

   on Nov 3, 2026

   Author

   More actions

   非常感谢，已经正常了：
   [MODEL-CONFIG] Dynamic Parameters:

   • task_type : Text Embedding (Representation Encoder)
   • n_layers : 28
   • n_embd : 1024
   • head_dim : 128
   • n_heads : 16 (Q)
   • n_kv_heads : 8 (KV, GQA ratio: 2)
   • n_ffn : 3072
   • vocab_size : 151936
   • max_seq_len: 4096
   • rope_base : 1000000.0
   • rms_eps : 1.00e-06
   • sub_norm : NO
   • activation : SwiGLU (SiLU)
   • ffn_mode : 2-Stage Fused Gate+Up
     [V37PACK] Loaded 423 pre-packed tensors from bitnet-0.6b-v5-mode6.v37p (425.7 MB)
     [V37] Embedding model detected. Skipping LM_HEAD cache initialization.

   --- EMBEDDING RESULT ---
   Prompt 1: "The hardware accelerator is designed for deep learning"
   Tokens : 9 (Latency: 2150.18 ms, 4.19 TPS)
   Dim : 1024
   L2 Norm : 1.000000
   Vector : [-0.050889, -0.047030, -0.000000, 0.009574, 0.001552, ... -0.030491, 0.031368, 0.010969, 0.034504, 0.085237]

   Prompt 2: "FPGA processor speeds up neural network inference"
   Tokens : 9 (Latency: 2156.19 ms, 4.17 TPS)
   Vector : [-0.001581, -0.131533, 0.000000, -0.006891, 0.030495, ... -0.002127, -0.031699, -0.007790, -0.027089, -0.019928]

   ==== Semantic Similarity Metric ====
   Cosine Similarity(Prompt 1, Prompt 2): 0.4651
   Cosine Similarity(Prompt 1, Prompt 1): 1.0000 (自对比一致性校验)