Post

I run a portion of all of my automated work through open models via an inference provider. When the observed difference in output becomes negligible, I demote that task to an open model full-time. Partly cost optimization, but mostly to ensure my tooling is still valuable when token subsidies end.

Do you have an example of a task that recently got demoted?

It's kind of hard to describe without providing a ton of background, but basically all of the code generation steps in our automated issue workflow are now done by cheaper models. The issue shaping, implementation validation, etc are still done by high-end models.

I have a fairly sizable local lab for running open weights. I use it to replace casual chat LLMs, I only use cloud tokens for coding, but would like to transition usage to entirely local inference. Focus of my work is to have a fully local autonomous stack with in-weight learning.

Are you creating your own model for in-weight learning or are you using some existing base model and building on top of that? How often do you update the weights? Do you have problems with catastrophic forgetting?

It's a lot of *infrastructure*. Less exciting work has happened (so far) at the model side than making sure weights are check-pointed and distributed via Git / Huggingface, server side sandboxed tool use, OIDC / Oauth scopes and policy, etc. Model-wise, working with Qwen3 dense models.

I’ve been using Kimi 2.5 via the OpenCode free preview, it’s not the smartest thing at solving problems, but it does a good job for planning & drafting docs that other models can then pick up on. Briefly tried Qwen 3 coder via ollama locally, but it’s slow and often fails at tool calls.

What’s your motivation for trying these models? Cost, curiosity, something else?

Yeah. The original experience was a collection of Pydantic AI agents and a loose collection of torch and various bits. Today I've switched over to goose for the easy to use recipes that can be used as sub agents and let you specify a model. Unfortunately the logging leaves a lot to be desired.

I have mostly abandoned thinking models for loops and supervisors. I use a fair amount of obsidian (basic features, mostly just markdown, webclipper, and quickadd). Having a couple of goose recipes/Pydantic AI agents to go parse through todo and reminders to take care of my pets is the most basic.

For actual work, I use GLM at the moment to go through and build grpc connectors, write documentation, and create diagrams with the mermaid MCP. Mostly not necessarily to write the code but to improve code quality. And embeddings. Lots of local vector search/hybrid search.

Used GLM-5 through OpenCode as the implementer of a plan that Opus created. Price was the main factor. It felt fast, and got the job done.

1. Extracting summaries/info from paragraph to page-length texts across medium scale (100-1000 books); also framing and answering questions about them. 2. Local. 3. I’ve gotten better at it and have realized that reasoning models can be overkill. 4. I don’t try open for coding and big concept tasks.

Motivation is slightly cost, but more that I can share an open workflow with students and collaborators without making *them* pay or introducing complex accounting.

Have you developed a good sense for categories of tasks that don’t require reasoning? What’re some examples?