Post

for all talks of local-first AI, given controversies over datacenters, I don't think people are willing to hear that local LLM inference is *probably* less environmentally friendly than cloud inference the majority of the time.

im going to be put on a list called "datacenter defenders" or something but like. if you think the nearest datacenter is ruining the environment then your local power sources are almost definitely worse lol

even just running an ollama service, I probably waste a ton of power and time waiting for the model to load into VRAM before running inference with something like karakeep. a cloud where the model is always ready is def better.

i think that depends a lot on the hardware though i.e. a gaming laptop that's built on components that optimize performance per watt feels like it should still be more efficient even taking all the overhead into account, just because DC hardware is optimized for different variables

(e.g. runs slower, generates less pleasing results). they're not even much more efficient than desktops, and they run on lithium ion batteries, which introduce inefficiencies of their own.