Difference between revisions of "AI Notes"

From PeformIQ Wiki
Jump to navigation Jump to search
(Replaced content with "= A Collection of AI Related Notes = * Using Bonsai QWEN Models Category:AI Category:QWEN Category:Llama.cpp")
Tag: Replaced
 
Line 3: Line 3:
* [[Using Bonsai QWEN Models]]
* [[Using Bonsai QWEN Models]]


See - https://www.youtube.com/watch?v=V6LmF7TuBmY (Bonsai 27B Runs Qwen 3.6 27B at 10x less memory)
Not a corrupt download — a format mismatch. Q2_0 in that filename isn't stock llama.cpp's quant; it's Q2_0_g128, PrismML's custom ternary packing (2-bit slots, one FP16 scale per 128 weights) that only their fork's kernels understand. Your stock build computes a different byte size for the ternary tensors, so its running offset drifts from what's written in the header, and gguf_init_from_reader bails at the first tensor where the two disagree — here output_norm.weight.
Corroboration: their g64 file is 7.59 GB vs 7.17 GB for the g128 one. That 0.42 GB delta is exactly one extra FP16 scale per 64 weights across ~27B weights — the scale density is the thing your build is guessing wrong.
Two fixes, easiest first:
1. Use the g64 file with your existing build. The model card says Ternary-Bonsai-27B-Q2_g64.gguf matches the 64-value-group Q2_0 packing already in llama.cpp:
<pre>
hf download prism-ml/Ternary-Bonsai-27B-gguf Ternary-Bonsai-27B-Q2_g64.gguf --local-dir ./models
./llama-server -m ./models/Ternary-Bonsai-27B-Q2_g64.gguf --ctx-size 16384
</pre>
2. Build their fork if you want the g128 file and the tuned Metal kernels:
<pre>
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build -j  # Metal is on by default
</pre>
Two other things:
--ctx-size 65536 will hurt on an M1. Weights are ~7.2 GB and the README's 100K-context figure is 14.7 GB peak without KV compression. On a 16 GB machine you'll swap; start at 8–16K and climb. Their throughput numbers are M4/M5 Pro — expect meaningfully less on M1.
Ignore the "Use this model" snippets on the HF page (llama serve -hf ...:F16) — those are auto-generated by Hugging Face, not the author's instructions, and F16 is the 53.8 GB file.
If the g64 file throws the same offset error, run python gguf-py/gguf/scripts/gguf_dump.py --no-tensors <file> and check general.file_type — that'll confirm whether it's a type-ID collision rather than the group size, and it's worth a post in their discussions.
Sources: Model card · File listing · PrismML llama.cpp fork




Line 40: Line 10:
[[Category:AI]]
[[Category:AI]]
[[Category:QWEN]]
[[Category:QWEN]]
[[Category:Llama.cpp]]

Latest revision as of 09:19, 27 August 2026

A Collection of AI Related Notes

Retrieved from "https://performiq.com/kb/index.php?title=AI_Notes&oldid=5546"