Post

@barredspirals.comint.su have you tried Strata? it's not my computer, and I can't test it thoroughly rn, but strata bench reports ~2k t/s prefill ~80 t/s decode on 12GB VRAM + 64GB RAM for qwen3.8-flash-next iq3_s, and testing with some leetcode prompts via curl seems to confirm it

I haven't tried it yet since there's no Vulkan support and i haven't been willing to go fight rocm again. if it's *that* fast it might actually be worth it tho (seems insane; those are the same specs I have and I get like 6tps decode, tops)

like I only get speeds like that from 3–4b models normally

yeah I have absolutely no slightest idea how it managed that. I have only tried the bench and curling stuff like "Implement an AVL tree in C..." though

hm i don't see support for my Radeon 6700 XT mentioned anywhere

it's still RDNA2? different llvm target than the ones they have support for (gfx1031 for 6700 vs. gfx1030 for the others) though

github.com/Niko1221/Str... mentions 6700: > Has a package, not tested on a real card