Post

Post deletes being complete happening soon! Thanks @jay.bsky.team! NOTE regardless of network, scraping happens and can't really be stopped. For years a bot reposted my every tweet. Please be careful. BlueSky is a playground - not a different universe - even if we rejoice in momentary reprieve.

yes, people should treat data here like it’s the public web. we had a long discussion today about whether/how to discourage scraping/archiving — @pfrazee.com?

yeah I'm gonna go Big Technical on this one

every user in atproto has a signed data repository, which is quite similar to a git repository. changes get written in commits and each repository maintains a history (commit log) we do that because the network is largely pull-based, and maintaining the history makes catchup sync efficient

designing the data repository is a bit of an art, because it's a merkle tree and you're balancing the size of proofs for a given record or subset of the data against the flexibility of the data it contains. @dholms.xyz is your guy on how all that works

as it turned out, the way image blobs get stored in the repository make it easy to just drop them and leave dangling references. that's why we're able to ship that quickly records, however, are harder to expunge without losing the ability to verify the proofs

daniel tells me it might be possible to change the data structure so that it's possible, but he had that kind of quiver in his voice when he said it that tells me I shouldnt expect it any time soon that means if we want to expunge the actual records, we have to do a "rebase"

This smells like the kind of thing that will lead to something like “The Great Skeet Massacre” or “Rebaseageddon” around 2027

Rebasing sounds a little bit like garbage collection

yeah pretty much

the problem with a rebase is that it interferes with replication. daniel can expand on this, but basically there are some conditions (which he calls the "happy path") where a rebase will cause no problems, and some where it'll mean having to refetch the whole repo

it also takes half a second to execute the rebase. so, for obvious reasons, we were trying to avoid doing it very often. reality is now blasting us in the face, and so more frequent rebases are on the table and the team will have to adapt. that's how this stuff goes

okay now let's talk about privacy models the data repositories are designed to be public. they're designed to be easy to cache and easy to aggregate. large scale was a design requirement from the get-go -- we want this network to be able to do what traditional services can

One of the very cool things about this design (IMO, as someone who has used them a lot, for other things), is that the merkle sync does very naturally and elegantly recover from a rebase. It just ends up having to fetch a lot more data.

I’m not technically minded, so I might have this completely wrong, but would it be possible to give users the ability to rebase their own accounts only when they need/want to via a settings menu or something like that?

So, in some ways at_protocol, although much better, still suffers some of the weird problems with delete that scuttlebutt has? I assumed the whole point of the Merkel DAG was to fix that. It’s a hard problem we never did solve with scuttlebutt.

This will lead to great pain and will end with deeply confused users. It brings on all the problems that CRDTs haven’t solved regarding automatic conflict resolution. The answer is to represent the root as a set. It’ll get you deletes. @dholms.xyz, we should chat.

I'm interested 🤔 "set" as in "bloom filter" or?

I was also initially scared of rebases, but we actually have a pretty happy path for them I don't think repos don't need to work as CRDTs because we have an actor with the authority to linearize & declare the current state

grandmaster flash voice: aaaaaAAAAAAH REBASE! do it now and get it over with imo

Two thoughts: 1. Could use use tombstones and compact later? It’s a soft delete so data is technically still there, but should keep meekly intact. 2. Could you use crypto key shredding similar to how PII is dealt with when users want to delete their data for GDPR?

To expand on (1): in my mind the idea would be to *add a piece of data* that marks something as deleted. It's annoying to have dangling CIDs in IPFS data structures, but the whole ecosystem will need to deal with that at some point IMO. This is a problem that appears in lots of places.

Bad hash lists are perhaps one way of doing this, but I'd say for posts it should probably be something more specific (e.g. signing the record to be deleted)

1. nah the problem is the proof authentication relies on the record being available (as I understand it) 2. you'd need a key for each post and I'm not sure how you reliably shred since you need to publish the key to anybody who can read in the first place

Can we get a \deltree*.* button on our own account to torpedo our own personal repository without deleting our account?

yeah that's a rebase, not a problem at all