Post

yes, people should treat data here like it’s the public web. we had a long discussion today about whether/how to discourage scraping/archiving — @pfrazee.com?

yeah I'm gonna go Big Technical on this one

every user in atproto has a signed data repository, which is quite similar to a git repository. changes get written in commits and each repository maintains a history (commit log) we do that because the network is largely pull-based, and maintaining the history makes catchup sync efficient

designing the data repository is a bit of an art, because it's a merkle tree and you're balancing the size of proofs for a given record or subset of the data against the flexibility of the data it contains. @dholms.xyz is your guy on how all that works

as it turned out, the way image blobs get stored in the repository make it easy to just drop them and leave dangling references. that's why we're able to ship that quickly records, however, are harder to expunge without losing the ability to verify the proofs

daniel tells me it might be possible to change the data structure so that it's possible, but he had that kind of quiver in his voice when he said it that tells me I shouldnt expect it any time soon that means if we want to expunge the actual records, we have to do a "rebase"

a "rebase" basically means rewriting the history to remove any record of the past writes. generally speaking you want to do this every so often as a form of compaction, because otherwise every repository is going to grow unbounded. so, that's something the team designed for from the start

the problem with a rebase is that it interferes with replication. daniel can expand on this, but basically there are some conditions (which he calls the "happy path") where a rebase will cause no problems, and some where it'll mean having to refetch the whole repo

it also takes half a second to execute the rebase. so, for obvious reasons, we were trying to avoid doing it very often. reality is now blasting us in the face, and so more frequent rebases are on the table and the team will have to adapt. that's how this stuff goes

okay now let's talk about privacy models the data repositories are designed to be public. they're designed to be easy to cache and easy to aggregate. large scale was a design requirement from the get-go -- we want this network to be able to do what traditional services can

4/5/23, 3:09 am7 reposts80 likes

we did not, however, fall on our heads and forget about privacy models we also have designs for how to do user-to-user and user-to-group communication. they just have to happen *off the repo* and let me explain why

we often refer to atproto as a hybrid of peer-to-peer and federation. the data repository is a concept very similar to the work many members of the team did with IPFS and dat/hyper the main difference is we don't use a dynamic hosting mesh via a DHT. instead you appoint a home server

however, the same premise that's at play with eg bittorrent is at play here. you're gossiping around these signed data structures and, because theyre signed, they carry the authentication with them. (we sometimes refer to that as self-certifying data)