Post

Working on a new blog post about local LLMs today! While running one of the tasks against llama 3.1 on my blazing fast M1 Max, I took a mactop snapshot. It's interesting to see the energy and GPU usage.

Lmk which you're using, I have a 32GB M1Max but all the models I wanted weren't compatible with the GPU/architecture

Llama 3.1 on ollama -- was re-following the instructions on this post with a different computer. Great model with highly accurate (but not perfect) results on larger datasets.

Ah, I was trying to get a local specialised PDF extraction kit going. Bit more niche and clearly done by Windows people!

Wonderful use-case! Requires a lot of python for the text extraction when OCR is required eh? I did something similar in this Azure Function, but ...without the OCR. That shit's tough with dotnet. OpenAI does it with Python.

I stumbled across that part because of the token limitation. I had to use PyMuPDF to pre-chew its food, but 3.7 Sonnet did a very creditable job of making semantic HTML out of things. Sadly I was then reduced to writing coordinate maths for footnotes

IN RUBY, which was a relief. Python has the best tools but the language is ... let's say not entirely to my taste.