Post

John Harrold
@abumirchi.com

Local inference (running LLMs on your local machine) is getting pretty cheap and it works really well. We will eventually be able to run open weights models (models that have been trained on the frontier models and have comparable abilities) in our house. They may not be the best..

Josh Marshall@joshtpm.bsky.social

These sound like non-trivial problems.

John Harrold
@abumirchi.com

But they are pretty damn good. Hardware is expensive now, but in a couple years when you can run basically Opus or Sol in your house, it will be difficult for Anthropic or Open AI to compete with that. It's like the frontier labs didn't plan for or even consider commodification of their products.

1 repost2 likes

Creep-uscular Canard
@turnitup5000db.bsky.social

You don’t even need great hardware. My inferrence machine I built out of an old gaming pc and a bunch of obsolete parts and gpus I did the repair work on myself, but the price tag for all the hardware is around $500. It’s not fast and it is jank but it does work.

John Harrold
@abumirchi.com

I agree you don't need great hardware. I'm thinking about a not so distant future where people with limited technical skill can do this at home. I can see someone selling a box that connects to someones wifi network and "just works".

Creep-uscular Canard
@turnitup5000db.bsky.social

Oh sure that probably happens inside a decade and technically can be a thing now, for the right (and hefty) price, just pointing out that these things can be done on a budget if you’re willing to learn the technical skills.

Barbara Rybolt 🐸
@barbararybolt.bsky.social

Seems to me no one was planning ahead by more than a week or month. I know they were planning on world conquest, but seem unable to remember there are several levels of learning & growth b4 the robots could take over the world. Such a shame. The scientists/computer nerds can’t science! #ProudBlue