Post

Working on a new blog post about local LLMs today! While running one of the tasks against llama 3.1 on my blazing fast M1 Max, I took a mactop snapshot. It's interesting to see the energy and GPU usage.

29/3/25, 9:27 am5 reposts16 likes

Chrissy LeMaire

@funbucket.dev

29 Mar 2025

Loaded Ollama and RAM jumped from 13.8 GB to 17.8 GB. GPU hit 97%, power spiked 27×. This is the before:

Chrissy LeMaire

@funbucket.dev

29 Mar 2025

Interesting difference. The table below shows idle vs. maxed out doing work/inference, not just a loaded model. Before inference, only memory spiked as the model was loaded into memory. Not GPU or CPU. As an infrastructure person and toolmaker newish to AI, I find this all fascinating!

Russell Garner

@rgarner.bsky.social

29 Mar 2025

Lmk which you're using, I have a 32GB M1Max but all the models I wanted weren't compatible with the GPU/architecture

Chrissy LeMaire

@funbucket.dev

29 Mar 2025

Llama 3.1 on ollama -- was re-following the instructions on this post with a different computer. Great model with highly accurate (but not perfect) results on larger datasets.

blog.netnerds.netAsking Tiny Questions: Local LLMs for PowerShell DevsRecently, I was trying to figure out if the content of two headlines overlapped. I needed a script that would answer if Pacific Palisades is part of Los …

Russell Garner

@rgarner.bsky.social

29 Mar 2025

Ah, I was trying to get a local specialised PDF extraction kit going. Bit more niche and clearly done by Windows people!

Chrissy LeMaire

@funbucket.dev

29 Mar 2025

Wonderful use-case! Requires a lot of python for the text extraction when OCR is required eh? I did something similar in this Azure Function, but ...without the OCR. That shit's tough with dotnet. OpenAI does it with Python.

github.comsoaps/azure-function at main · potatoqualitee/soapsStructured Objects and PowerShell. Contribute to potatoqualitee/soaps development by creating an account on GitHub.

gallaugher

@gallaugher.bsky.social

30 Mar 2025

Aside. Did you write the descriptive text or was it generated? In either case, well done. And if you wrote it, you are nominated for a Booker Prize.

Chrissy LeMaire

@funbucket.dev

30 Mar 2025

Generated! I have a custom GPT for it. My original prompt sucked then @usrbinkat.io shared her prompt and I replaced mine.

gist.github.comAlt-Text ChatGPT BotAlt-Text ChatGPT Bot. GitHub Gist: instantly share code, notes, and snippets.