Post

got bonsai and 27b running decently on my Rx 6700 (10GB VRAM), packages as a nix flake so anyone can run it with minimal work 9b -- 813 tokens/s prefill, 84 tokens//s 27B -- 267 tokens/s prefill, 26 tokens/s github.com/alyraffauf/b...

github.comGitHub - alyraffauf/bonsai-buddy: Run Prism Bonsai 2 9B and 27B locally with Nix, tuned AMD GPU profiles, and OpenCode.Run Prism Bonsai 2 9B and 27B locally with Nix, tuned AMD GPU profiles, and OpenCode. - alyraffauf/bonsai-buddy

aly

@aly.codes

*bonsai 9b and 27b, oops also got decent context windows given size

24/10/26, 6:10 pm0 reposts15 likes

aly

@aly.codes

16K for 27b is still pretty tight but 128k with 9b is actually usable!

Frank Blank

@frank8465.bsky.social

I wonder if you can eke out a little more context from bonsai by doing a 1:9 pipeline parallelism between your cpu and gpu, and I also wonder when the accuracy falloff begins from the quantization

aly

@aly.codes

tbh i specifically wanted gpu only except some image stuff

Frank Blank

@frank8465.bsky.social

(this was a bench I was contemplating writing after noticing that quantized model inaccuracy is less a constant "these tokens are less worth it" and more a "error accumulation builds up, and your first few thousand/ten thousand tokens are actually very usable)

Frank Blank

@frank8465.bsky.social

basically what I'm trying to say in this comment is, maybe on contexts that small, going down to q5 or q4 kv might not actually be very damaging to model intelligence whatsoever