Post

im going to be put on a list called "datacenter defenders" or something but like. if you think the nearest datacenter is ruining the environment then your local power sources are almost definitely worse lol

even just running an ollama service, I probably waste a ton of power and time waiting for the model to load into VRAM before running inference with something like karakeep. a cloud where the model is always ready is def better.

i think that depends a lot on the hardware though i.e. a gaming laptop that's built on components that optimize performance per watt feels like it should still be more efficient even taking all the overhead into account, just because DC hardware is optimized for different variables

ehh this seems like an unfair comparision. local LLM inference would likely use less total power because consumer grade hardware is limits the size of model that people can run. it wouldn’t be worse for the environment if more people used smaller models even if the per-watt efficiency went down.

Absolutely agree! I think there's little way around it (like basically all the efficiencies and benefits we can secure for local AI could be replicated, possibly even more effectively, in centralized systems). I think people are conflating local with small, where we could just say what we mean...

as a local ai endorser... economies of scale [shocked pikachu]