Post

what if part of sync state was a small per-repo LRU cache of block CIDs? since most updates (at least with tid rkeys) stay on a few hot edges of the mst, maybe you could avoid sending most CIDs with most commits, just referencing a cache index instead

we debated a similar optimization at some point, I think why came up with it based on filesystem implementation tricks: have a small array of records be referenced directly (by path and CID) from the commit object. they would still go out on firehose, but wouldn't churn the MST structure

the firehose event would be just the new (signed) commit object, and the (new?) records. no MST nodes. the motivation was this was a high rate of small interaction records such as "likes".

I personally didn't like this because it adds implementation complexity in a bunch of places (PDS, consumers, anybody requesting a proof chain). you need to be very careful doing existence lookups, handling invalid corner-cases, etc. I just think correctness+reliably would suffer in practice

so you could even have local helper programs to proxy your connection through which handle the format change with STAR that’s definitely true, in bandwidth reduction in multiples. this might not be that easy (some extra state on both sides) but that’s not obvious to me yet