Post

very cool! I don't have as much RAM as you or I might try it too

The Qwen 32B models appear to have GPT-4 like capabilities as well, and I've heard from people who got those working in 24GB of RAM

looks like llama3.2 passes the Hillel "gives hilarious answers to dumb riddles" test

I'm the wrong person to ask, I like using my own! (terminal CLI based) llm.datasette.io

Yeah, I think the environmental impact of inference (as opposed to training) is often over-stated These models have got massively more efficient over the past two years

Yep. This is economic incentives at work, too. Which is why I feel the cryptocurrency analogy that a lot of naysayers have doesn't fit. Everyone involved wants models to be cheaper and faster, and so that's what's happening.

Wikipedia can be stored on a $5 usb drive for some time now.

But I would still be cautious about the existance of a capability cap. It could well be that model performance scales much more with data quality and quantity and there is a already close sealing on high quality data usage

Hey Simon! It's been ages! Great article. I actually downloaded a Chrome beta — you can now even run some experimental language translation and summarization locally bundled inside Chrome, too developer.chrome.com/docs/extensi...

Made me smile that with 64 gigs of ram firefox + vscode made a dent!

This was exactly my reaction when running this over the weekend. After trying and failing dismally to get 3.2:405B to run on my laptop (got it to load but that was about it), seeing 3.3:70B run easily & with good results is fantastic. Opens up a whole set of new use cases!

For one thing, it's a great insurance policy against investors losing faith and stopping injecting billions of dollars a year into keeping the existing big vendors afloat and running at a loss!

I have tried it, it's hit and miss even with the large models. I hopeful it's going to get better. many instances of it not honouring parameters and or not using the response from the tool when it does work.

Thank you ! I have a few agents and a tag implementation but have I’d love to make totally free and open but the tool calling / Agentic stuff hasn’t quite gotten there yet

Same here! Haven't gotten LLama to return tool calls. Claude and ChatGPT have been passable. but I wonder if it's just my vllm configurations or something; gorilla.cs.berkeley.edu/leaderboard.... shows Llama3.1:70b doing decently at this.