Skip to content

[Project share] llama.cpp on the LLM-8850: Qwen3-0.6B straight from GGUF at 7.9 tok/s — NPU engine weight format reverse-engineered #31

Open

Description

@woolcoxm

Project share — not a bug report. Posting here because this community runs the same AX8850 hardware; hoping it's useful. Happy to move/delete if there's a better venue.

What this is

A llama.cpp backend (ggml-axcl) for the Axera AX8850 — tested on an M5Stack LLM-8850 card hosted on a Raspberry Pi 5. Full offline generation with the open llama.cpp toolchain: no cloud services, no vendor runtime required.

The headline: the GGUF is the only model artifact. Weights are patched into pre-compiled whole-layer NPU engines at load time — no model conversion, no per-model compile step. Drop in any Qwen3-0.6B-family GGUF (Q8_0, Q4_K_M, …) and it runs.

Results

Qwen3-0.6B, 2048-token context, measured on-card:

decode prefill host CPU card CMM
this work 7.9 tok/s 7.9 tok/s ~8% of one core 2.4 GB
vendor runtime (closed, baked weights) 13.5 tok/s — — —

All model compute runs on the NPU as 29 engine calls per token: 28 whole-layer graphs (RMS norms, Q/K/V with q/k-norm, RoPE, attention over the on-card KV cache, SwiGLU FFN) plus one post graph (final norm + the 151 936-wide lm_head). The hidden state is bf16 and never leaves the card — the CPU only builds the llama.cpp graph and samples.

How the dynamic weights work

axclrtEngineLoadFromMem exists but hangs in driver V3.6.5_P1, and the axmodel container embeds weights in an undocumented layout — so normally you're stuck recompiling per model with Pulsar2. I reverse-engineered the layout instead:

  1. Differential marker builds — feed Pulsar2 checkpoints whose weights encode coordinate bits (V = 16·code(r,k) + dither, with anchors pinning the row max) and diff the resulting blobs. Codes reveal where every element lands; dither values cross-validate each assignment, killing false positives.
  2. The bf16-engine layout turned out to be clean — weights are stored as raw bf16 at deterministic file offsets: no norm folding, no quantization transform. A value-anchored row search (each matrix row's first few bf16 values at a fixed stride) pins all 15.73 M weight positions plus the 2 304 norm-vector entries.
  3. Byte-exact validation — patch layer-0's template with layer-1's weights and you reproduce the baked layer-1 engine file to 3 bytes (layer-index microcode), and the patched engine computes identically to the baked one on-card (max output delta 0.0 across random inputs).

The loader then just: reads a template engine → dequantizes each GGUF tensor row → bf16-rounds it → scatters it into a copy of the file via a position sidecar → loads it. Patched engines are cached keyed by a hash of the weights, so warm starts skip the scatter entirely.

Gotchas documented along the way

In case anyone else walks this road:

  • -w s8 engine builds secretly store int4 weights whose quantization drift (~3 %/layer, compounding) garbles generation — -w bf16 is the safe path.
  • The runtime mishandles sub-buffer bindings — binding an offset into a device table is silently ignored; refresh dedicated buffers per call instead.
  • Uninitialized device caches can be NaN-shaped, and NaN scores poison the softmax through the −inf mask (NaN + −inf = NaN). Zero-fill at allocation.
  • An engine must never read its input from the buffer it writes — double-buffer the hidden state.
  • Loading/unloading many engines in quick succession (process churn) transiently fails — retry with backoff.

Repo & verification

  • Backend: woolcoxm/llama.cpp, branch Axera-8850-GGUF-support-PoC-qwen3-0.9b-Q4KM-Q8 — ggml/src/ggml-axcl/ggml-axcl.cpp is the whole thing
  • Research lab (layout tooling, marker generators, validation harnesses, evals): woolcoxm/LLMTest

Verified: 12/12 E2E suite (1→3000-token prompts, unicode/emoji/metacharacters, empty prompts, SIGINT handling, leak checks) and 94 % greedy-decode token agreement vs the CPU reference on the same GGUF (the residual divergence is bf16-engine vs q8 weight numerics on near-tie tokens).

Also cross-posted to Axera's runtime repo: AXERA-TECH/axcl-runtime#10.

Next targets

  • Engine weight dtype — currently DRAM-bound on bf16 reads; a correct 8-bit format should close most of the gap to 13.5 tok/s
  • The prefill shape-group ladder (m=128 engines exist but are unused)
  • Generalizing the layout table across architectures

Happy to go deeper on any part of the format findings or the patching pipeline — the tooling in LLMTest reproduces all of it.

Activity

  1. woolcoxm commented on Aug 27, 2026

    @woolcoxm
    Author

    hopefully someone finds this useful.

    i believe i can get MoE running with CPU offloading of experts(since we only have 8gb to play with).

    it took me 4 days straight just to get the gguf to load into the axera.

  2. woolcoxm commented on Aug 27, 2026

    @woolcoxm
    Author

    LLM-8850 on Raspberry Pi 5: llama.cpp now runs GGUF models at 19.6 tok/s — faster than the bundled runtime

    Update to my LLM-8850 project thread. Open source: github.com/woolcoxm/llama.cpp (branch Axera-8850-GGUF-support-PoC-qwen3-0.9b-Q4KM-Q8) + the research repo: github.com/woolcoxm/Axera-AX8850-GGUF-Support.

    Quick summary for fellow LLM-8850 owners: I've been building a custom llama.cpp backend (ggml-axcl) that drives the card's AX8850 NPU directly — Qwen3-0.6B runs straight from a GGUF file (q8_0 and Q4_K_M both work, no model conversion), with every transformer layer compiled as a single NPU graph and the KV cache living on the card.

    The numbers (Pi 5 host)

    decode prefill Pi CPU load card memory
    llama.cpp + int8 engines 19.6 tok/s 18.5 tok/s 2% of one core 1.3 GB
    llama.cpp, dynamic GGUF weights 10.2 tok/s 5.0 tok/s ~5% 2.4 GB
    bundled closed runtime (same card) 13.5–14.5 tok/s — — —

    So the open stack now beats the card's own bundled runtime by ~40%, while leaving the Pi's CPU essentially free (it only tokenizes and samples). That headroom means you could run the Pi's CPU cores for other work while decoding. Numbers are for Qwen3-0.6B at 2048 context; identical between q8_0 and Q4_K_M GGUFs in the fastest mode (the NPU engines carry the weights there; the GGUF drives tokenizer/sampling).

    What made the difference

    1. Killing per-token host overhead. The card's runtime charges ~1 ms for any small unpinned memory transfer. The backend now pins everything, refreshes per-layer state once per token instead of 28 times, binds engine IO buffers once, writes K/V rows directly into the on-card cache, and defers syncing the KV cache back to host memory until something actually needs it. Host cost per token: 35 ms → ~2 ms.
    2. The int8 engines. The AXERA-TECH w8a16 engine set for Qwen3-0.6B (same file layout the vendor runtime uses) runs the NPU's native int8 path — half the time and a third of the memory of the bf16 engines I compiled myself. The backend speaks the same IO protocol to both.
    3. Pipelining the 28 per-layer engine calls on one async stream.

    There's also a fully "GGUF-native" mode where the GGUF's own weights are patched into the engine files at load (that's the 10.2 tok/s row) — that one needed reverse-engineering the engine weight layout, which is documented in the repo.

    Try it

    Build llama.cpp with -DGGML_AXCL=ON, install the engine set, and:

    GGML_AXCL_LAYER=1 GGML_AXCL_FA=1 GGML_AXCL_STREAM=1 \
    GGML_AXCL_LAYER_DIR=$HOME/Qwen3-0.6B \
    GGML_AXCL_POST_MODEL=$HOME/Qwen3-0.6B/qwen3_post.axmodel \
    ./llama-simple -m qwen3-q8.gguf -n 128 "your prompt"

    Full instructions, the benchmark matrix (both quants), E2E test results (12/12), and the whole research log are in the LLMTest README.

    One heads-up for M5Stack/Axera

    The card's engines contain a 128-token prefill "ladder" that would make long prompts process many times faster — but it doesn't function through the PCIe host runtime (the engine ignores the input binding for those shape groups; the bundled runtime never uses them either, which we verified by tracing it). Reported to Axera with full details. And a practical tip: if a run gets killed mid-flight, the card driver can wedge (next process hangs loading engines) — a Pi reboot fixes it.

    Happy to answer setup questions — and thanks M5Stack for making this little card available; it's been a great NPU sandbox.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions