Repository navigation
Description
Project share — not a bug report. Posting here because this community runs the same AX8850 hardware; hoping it's useful. Happy to move/delete if there's a better venue.
What this is
A llama.cpp backend (ggml-axcl) for the Axera AX8850 — tested on an M5Stack LLM-8850 card hosted on a Raspberry Pi 5. Full offline generation with the open llama.cpp toolchain: no cloud services, no vendor runtime required.
The headline: the GGUF is the only model artifact. Weights are patched into pre-compiled whole-layer NPU engines at load time — no model conversion, no per-model compile step. Drop in any Qwen3-0.6B-family GGUF (Q8_0, Q4_K_M, …) and it runs.
Results
Qwen3-0.6B, 2048-token context, measured on-card:
| decode | prefill | host CPU | card CMM | |
|---|---|---|---|---|
| this work | 7.9 tok/s | 7.9 tok/s | ~8% of one core | 2.4 GB |
| vendor runtime (closed, baked weights) | 13.5 tok/s | — | — | — |
All model compute runs on the NPU as 29 engine calls per token: 28 whole-layer graphs (RMS norms, Q/K/V with q/k-norm, RoPE, attention over the on-card KV cache, SwiGLU FFN) plus one post graph (final norm + the 151 936-wide lm_head). The hidden state is bf16 and never leaves the card — the CPU only builds the llama.cpp graph and samples.
How the dynamic weights work
axclrtEngineLoadFromMem exists but hangs in driver V3.6.5_P1, and the axmodel container embeds weights in an undocumented layout — so normally you're stuck recompiling per model with Pulsar2. I reverse-engineered the layout instead:
- Differential marker builds — feed Pulsar2 checkpoints whose weights encode coordinate bits (
V = 16·code(r,k) + dither, with anchors pinning the row max) and diff the resulting blobs. Codes reveal where every element lands; dither values cross-validate each assignment, killing false positives. - The bf16-engine layout turned out to be clean — weights are stored as raw bf16 at deterministic file offsets: no norm folding, no quantization transform. A value-anchored row search (each matrix row's first few bf16 values at a fixed stride) pins all 15.73 M weight positions plus the 2 304 norm-vector entries.
- Byte-exact validation — patch layer-0's template with layer-1's weights and you reproduce the baked layer-1 engine file to 3 bytes (layer-index microcode), and the patched engine computes identically to the baked one on-card (max output delta 0.0 across random inputs).
The loader then just: reads a template engine → dequantizes each GGUF tensor row → bf16-rounds it → scatters it into a copy of the file via a position sidecar → loads it. Patched engines are cached keyed by a hash of the weights, so warm starts skip the scatter entirely.
Gotchas documented along the way
In case anyone else walks this road:
-w s8engine builds secretly store int4 weights whose quantization drift (~3 %/layer, compounding) garbles generation —-w bf16is the safe path.- The runtime mishandles sub-buffer bindings — binding an offset into a device table is silently ignored; refresh dedicated buffers per call instead.
- Uninitialized device caches can be NaN-shaped, and NaN scores poison the softmax through the −inf mask (
NaN + −inf = NaN). Zero-fill at allocation. - An engine must never read its input from the buffer it writes — double-buffer the hidden state.
- Loading/unloading many engines in quick succession (process churn) transiently fails — retry with backoff.
Repo & verification
- Backend: woolcoxm/llama.cpp, branch
Axera-8850-GGUF-support-PoC-qwen3-0.9b-Q4KM-Q8—ggml/src/ggml-axcl/ggml-axcl.cppis the whole thing - Research lab (layout tooling, marker generators, validation harnesses, evals): woolcoxm/LLMTest
Verified: 12/12 E2E suite (1→3000-token prompts, unicode/emoji/metacharacters, empty prompts, SIGINT handling, leak checks) and 94 % greedy-decode token agreement vs the CPU reference on the same GGUF (the residual divergence is bf16-engine vs q8 weight numerics on near-tie tokens).
Also cross-posted to Axera's runtime repo: AXERA-TECH/axcl-runtime#10.
Next targets
- Engine weight dtype — currently DRAM-bound on bf16 reads; a correct 8-bit format should close most of the gap to 13.5 tok/s
- The prefill shape-group ladder (m=128 engines exist but are unused)
- Generalizing the layout table across architectures
Happy to go deeper on any part of the format findings or the patching pipeline — the tooling in LLMTest reproduces all of it.
Activity
woolcoxm commentedon Aug 27, 2026
hopefully someone finds this useful.
i believe i can get MoE running with CPU offloading of experts(since we only have 8gb to play with).
it took me 4 days straight just to get the gguf to load into the axera.
woolcoxm commentedon Aug 27, 2026
LLM-8850 on Raspberry Pi 5: llama.cpp now runs GGUF models at 19.6 tok/s — faster than the bundled runtime
Update to my LLM-8850 project thread. Open source: github.com/woolcoxm/llama.cpp (branch
Axera-8850-GGUF-support-PoC-qwen3-0.9b-Q4KM-Q8) + the research repo: github.com/woolcoxm/Axera-AX8850-GGUF-Support.Quick summary for fellow LLM-8850 owners: I've been building a custom llama.cpp backend (
ggml-axcl) that drives the card's AX8850 NPU directly — Qwen3-0.6B runs straight from a GGUF file (q8_0 and Q4_K_M both work, no model conversion), with every transformer layer compiled as a single NPU graph and the KV cache living on the card.The numbers (Pi 5 host)
So the open stack now beats the card's own bundled runtime by ~40%, while leaving the Pi's CPU essentially free (it only tokenizes and samples). That headroom means you could run the Pi's CPU cores for other work while decoding. Numbers are for Qwen3-0.6B at 2048 context; identical between q8_0 and Q4_K_M GGUFs in the fastest mode (the NPU engines carry the weights there; the GGUF drives tokenizer/sampling).
What made the difference
There's also a fully "GGUF-native" mode where the GGUF's own weights are patched into the engine files at load (that's the 10.2 tok/s row) — that one needed reverse-engineering the engine weight layout, which is documented in the repo.
Try it
Build llama.cpp with
-DGGML_AXCL=ON, install the engine set, and:Full instructions, the benchmark matrix (both quants), E2E test results (12/12), and the whole research log are in the LLMTest README.
One heads-up for M5Stack/Axera
The card's engines contain a 128-token prefill "ladder" that would make long prompts process many times faster — but it doesn't function through the PCIe host runtime (the engine ignores the input binding for those shape groups; the bundled runtime never uses them either, which we verified by tracing it). Reported to Axera with full details. And a practical tip: if a run gets killed mid-flight, the card driver can wedge (next process hangs loading engines) — a Pi reboot fixes it.
Happy to answer setup questions — and thanks M5Stack for making this little card available; it's been a great NPU sandbox.