Post

for all talks of local-first AI, given controversies over datacenters, I don't think people are willing to hear that local LLM inference is *probably* less environmentally friendly than cloud inference the majority of the time.

even just running an ollama service, I probably waste a ton of power and time waiting for the model to load into VRAM before running inference with something like karakeep. a cloud where the model is always ready is def better.

i think that depends a lot on the hardware though i.e. a gaming laptop that's built on components that optimize performance per watt feels like it should still be more efficient even taking all the overhead into account, just because DC hardware is optimized for different variables

datacenters are definitely concerned with performance per watt! it's pretty key to profitability, they're not *trying* to maximize power usage. the way that gaming laptops achieve portability/any battery life at all is by sacrificing a great deal of performance and fidelity