Post

got bonsai and 27b running decently on my Rx 6700 (10GB VRAM), packages as a nix flake so anyone can run it with minimal work 9b -- 813 tokens/s prefill, 84 tokens//s 27B -- 267 tokens/s prefill, 26 tokens/s github.com/alyraffauf/b...

*bonsai 9b and 27b, oops also got decent context windows given size

16K for 27b is still pretty tight but 128k with 9b is actually usable!

I wonder if you can eke out a little more context from bonsai by doing a 1:9 pipeline parallelism between your cpu and gpu, and I also wonder when the accuracy falloff begins from the quantization