Post

I can now run a GPT-4 class model on my laptop (The exact same laptop that could just about run a GPT-3 class model 20 months ago) The new Llama 3.3 70B is a striking example of the huge efficiency gains we've seen in the last two years simonwillison.net/2024/Dec/9/l...

This finally inspired me to try downloading and running an llm locally. Jesus does this run slowly. Is the only way to get it faster to get a better GPU?

Yeah, to get good performance you need enough GPU memory to fit the whole model - which is about 40GB for this one, so best off on a high-end Mac laptop or with multiple expensive NVIDIA graphics cards

looks like llama3.2 passes the Hillel "gives hilarious answers to dumb riddles" test