Post

I am happy to share a little thing we have been cooking up: github.com/agent-substr... Problem statement: How can we effectively host a bazillion mostly-idle agents in a regular-sized Kubernetes cluster? This is *demo-grade* software, but it shows some ideas.

I watched the video and read the readme, but I'm having a hard time visualizing what's required to run this (especially for "teleporting") Can you draw me a picture of the infrastructure components?

Actors: things that can be suspended/resumed Workers: "stem cell" pods that can resume any actor Proxy: Assign inbound requests for a particular actor (think "chat session" = actor) to available workers When an actor goes idle, suspend to storage & unassign. Most agents are idle most of the time.

The effective result is: 1) an agent can run on one worker, be suspended, and be resumed on another worker very quickly 2) we can multiplex many agents onto few workers This is not PhD level insight, of course. It's trying to bring together a set of fairly established ideas into a working model.

Agent's are idle most of the time but waiting for replies. I'm assuming when the reply returns from an LLM it would be routed to the correct actor (with memory and filesystem context) Where is the snapshot state stored? CRDs? Is there other requirements like RWX storage? It doesn't look like it

For this phase, "idle" means it is done with a "turn" and waiting for a new input. Over time we can get more aggressive with that, but it's tricky when moving around means IP changes. Baby steps :) Snapshots are stored in GCS/S3 for now. We know we need local caching and locality in assignment. :)

An actor is a single writer, so we can side-step RWX storage with snapshots (consistent across process memory and filesystem). RWX may be an optimization but we haven't explored that yet. Some agents (will) want real RWX (simultaneous multi-writer) and that's a tomorrow problem :)