Post

yes, people should treat data here like it’s the public web. we had a long discussion today about whether/how to discourage scraping/archiving — @pfrazee.com?

yeah I'm gonna go Big Technical on this one

every user in atproto has a signed data repository, which is quite similar to a git repository. changes get written in commits and each repository maintains a history (commit log) we do that because the network is largely pull-based, and maintaining the history makes catchup sync efficient

designing the data repository is a bit of an art, because it's a merkle tree and you're balancing the size of proofs for a given record or subset of the data against the flexibility of the data it contains. @dholms.xyz is your guy on how all that works

as it turned out, the way image blobs get stored in the repository make it easy to just drop them and leave dangling references. that's why we're able to ship that quickly records, however, are harder to expunge without losing the ability to verify the proofs

daniel tells me it might be possible to change the data structure so that it's possible, but he had that kind of quiver in his voice when he said it that tells me I shouldnt expect it any time soon that means if we want to expunge the actual records, we have to do a "rebase"

a "rebase" basically means rewriting the history to remove any record of the past writes. generally speaking you want to do this every so often as a form of compaction, because otherwise every repository is going to grow unbounded. so, that's something the team designed for from the start

the problem with a rebase is that it interferes with replication. daniel can expand on this, but basically there are some conditions (which he calls the "happy path") where a rebase will cause no problems, and some where it'll mean having to refetch the whole repo

Rabble

@rabble.nz

So, in some ways at_protocol, although much better, still suffers some of the weird problems with delete that scuttlebutt has? I assumed the whole point of the Merkel DAG was to fix that. It’s a hard problem we never did solve with scuttlebutt.