AI Notes

A Collection of AI Related Notes

• Using Bonsai QWEN Models

See - https://www.youtube.com/watch?v=V6LmF7TuBmY (Bonsai 27B Runs Qwen 3.6 27B at 10x less memory)

Not a corrupt download — a format mismatch. Q2_0 in that filename isn't stock llama.cpp's quant; it's Q2_0_g128, PrismML's custom ternary packing (2-bit slots, one FP16 scale per 128 weights) that only their fork's kernels understand. Your stock build computes a different byte size for the ternary tensors, so its running offset drifts from what's written in the header, and gguf_init_from_reader bails at the first tensor where the two disagree — here output_norm.weight.

Corroboration: their g64 file is 7.59 GB vs 7.17 GB for the g128 one. That 0.42 GB delta is exactly one extra FP16 scale per 64 weights across ~27B weights — the scale density is the thing your build is guessing wrong.

Two fixes, easiest first:

1. Use the g64 file with your existing build. The model card says Ternary-Bonsai-27B-Q2_g64.gguf matches the 64-value-group Q2_0 packing already in llama.cpp:

hf download prism-ml/Ternary-Bonsai-27B-gguf Ternary-Bonsai-27B-Q2_g64.gguf --local-dir ./models
./llama-server -m ./models/Ternary-Bonsai-27B-Q2_g64.gguf --ctx-size 16384

2. Build their fork if you want the g128 file and the tuned Metal kernels:

git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build -j   # Metal is on by default

Two other things:

--ctx-size 65536 will hurt on an M1. Weights are ~7.2 GB and the README's 100K-context figure is 14.7 GB peak without KV compression. On a 16 GB machine you'll swap; start at 8–16K and climb. Their throughput numbers are M4/M5 Pro — expect meaningfully less on M1. Ignore the "Use this model" snippets on the HF page (llama serve -hf ...:F16) — those are auto-generated by Hugging Face, not the author's instructions, and F16 is the 53.8 GB file.

If the g64 file throws the same offset error, run python gguf-py/gguf/scripts/gguf_dump.py --no-tensors <file> and check general.file_type — that'll confirm whether it's a type-ID collision rather than the group size, and it's worth a post in their discussions.

Sources: Model card · File listing · PrismML llama.cpp fork