Post

@barredspirals.comint.su have you tried Strata? it's not my computer, and I can't test it thoroughly rn, but strata bench reports ~2k t/s prefill ~80 t/s decode on 12GB VRAM + 64GB RAM for qwen3.8-flash-next iq3_s, and testing with some leetcode prompts via curl seems to confirm it

I haven't tried it yet since there's no Vulkan support and i haven't been willing to go fight rocm again. if it's *that* fast it might actually be worth it tho (seems insane; those are the same specs I have and I get like 6tps decode, tops)

like I only get speeds like that from 3–4b models normally

like, I couldn't get these speeds from qwen3.6-35b-a3b on the same machine with llama.

hm i don't see support for my Radeon 6700 XT mentioned anywhere

it's still RDNA2? different llvm target than the ones they have support for (gfx1031 for 6700 vs. gfx1030 for the others) though