AI Notes - Revision history

PeterHarding: Replaced content with "= A Collection of AI Related Notes = * Using Bonsai QWEN Models Category:AI Category:QWEN Category:Llama.cpp"

2026-08-26T23:19:05Z

Replaced content with "= A Collection of AI Related Notes = * Using Bonsai QWEN Models Category:AI Category:QWEN Category:Llama.cpp"

@@ Line 3: / Line 3: @@
 * [[Using Bonsai QWEN Models]]
@@ Line 40: / Line 10: @@
 [[Category:AI]]
 [[Category:QWEN]]

PeterHarding at 23:17, 26 August 2026

2026-08-26T23:17:22Z

← Older revision		Revision as of 09:17, 27 August 2026
Line 1:		Line 1:
	= A Collection of AI Related Notes =		= A Collection of AI Related Notes =

			* [[Using Bonsai QWEN Models]]

	See - https://www.youtube.com/watch?v=V6LmF7TuBmY (Bonsai 27B Runs Qwen 3.6 27B at 10x less memory)		See - https://www.youtube.com/watch?v=V6LmF7TuBmY (Bonsai 27B Runs Qwen 3.6 27B at 10x less memory)

PeterHarding: Created page with "= A Collection of AI Related Notes = See - https://www.youtube.com/watch?v=V6LmF7TuBmY (Bonsai 27B Runs Qwen 3.6 27B at 10x less memory) Not a corrupt download — a format mismatch. Q2_0 in that filename isn't stock llama.cpp's quant; it's Q2_0_g128, PrismML's custom ternary packing (2-bit slots, one FP16 scale per 128 weights) that only their fork's kernels understand. Your stock build computes a different byte size for the ternary tensors, so its running offset drif..."

2026-08-26T23:16:44Z

Created page with "= A Collection of AI Related Notes = See - https://www.youtube.com/watch?v=V6LmF7TuBmY (Bonsai 27B Runs Qwen 3.6 27B at 10x less memory) Not a corrupt download — a format mismatch. Q2_0 in that filename isn't stock llama.cpp's quant; it's Q2_0_g128, PrismML's custom ternary packing (2-bit slots, one FP16 scale per 128 weights) that only their fork's kernels understand. Your stock build computes a different byte size for the ternary tensors, so its running offset drif..."

New page

= A Collection of AI Related Notes =

See - https://www.youtube.com/watch?v=V6LmF7TuBmY (Bonsai 27B Runs Qwen 3.6 27B at 10x less memory)

Not a corrupt download — a format mismatch. Q2_0 in that filename isn't stock llama.cpp's quant; it's Q2_0_g128, PrismML's custom ternary packing (2-bit slots, one FP16 scale per 128 weights) that only their fork's kernels understand. Your stock build computes a different byte size for the ternary tensors, so its running offset drifts from what's written in the header, and gguf_init_from_reader bails at the first tensor where the two disagree — here output_norm.weight.

Corroboration: their g64 file is 7.59 GB vs 7.17 GB for the g128 one. That 0.42 GB delta is exactly one extra FP16 scale per 64 weights across ~27B weights — the scale density is the thing your build is guessing wrong.

Two fixes, easiest first:

1. Use the g64 file with your existing build. The model card says Ternary-Bonsai-27B-Q2_g64.gguf matches the 64-value-group Q2_0 packing already in llama.cpp:

<pre>
hf download prism-ml/Ternary-Bonsai-27B-gguf Ternary-Bonsai-27B-Q2_g64.gguf --local-dir ./models
./llama-server -m ./models/Ternary-Bonsai-27B-Q2_g64.gguf --ctx-size 16384
</pre>

2. Build their fork if you want the g128 file and the tuned Metal kernels:

<pre>
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build -j # Metal is on by default
</pre>

Two other things:

--ctx-size 65536 will hurt on an M1. Weights are ~7.2 GB and the README's 100K-context figure is 14.7 GB peak without KV compression. On a 16 GB machine you'll swap; start at 8–16K and climb. Their throughput numbers are M4/M5 Pro — expect meaningfully less on M1.
Ignore the "Use this model" snippets on the HF page (llama serve -hf ...:F16) — those are auto-generated by Hugging Face, not the author's instructions, and F16 is the 53.8 GB file.

If the g64 file throws the same offset error, run python gguf-py/gguf/scripts/gguf_dump.py --no-tensors <file> and check general.file_type — that'll confirm whether it's a type-ID collision rather than the group size, and it's worth a post in their discussions.

Sources: Model card · File listing · PrismML llama.cpp fork

[[Category:AI]]
[[Category:QWEN]]