Post

tbh i specifically wanted gpu only except some image stuff

(this was a bench I was contemplating writing after noticing that quantized model inaccuracy is less a constant "these tokens are less worth it" and more a "error accumulation builds up, and your first few thousand/ten thousand tokens are actually very usable)

goat. have you messed with Strata or any of its forks?

How much of the fever did you get? Are you doing vulkan vs rocm benchmarks and testing disabling reasoning for more loops? There's an ornith 9b to play with...😈

sory

fun fact I pretend to have your GPU so rocm runs bc its not supported lol

you can do a decent bit with that card...

Nice!

thx for sharing!