Post

datacenter inference is and will probably always be less environmentally destructive than local inference btw. the environmentalist option is not 'local AI'.

tbh i can't wait until gpus have more specialized hardware for inference (we have compute cores yes but i'm thinking about something like a tpu in your gpu) so that you can run small models for translation, accessibility features (better ocr screen readers), transcription, etc

i know that a lot of modern cpus have these but afaict these have basically zero software support