Post

Post deletes being complete happening soon! Thanks @jay.bsky.team! NOTE regardless of network, scraping happens and can't really be stopped. For years a bot reposted my every tweet. Please be careful. BlueSky is a playground - not a different universe - even if we rejoice in momentary reprieve.

yes, people should treat data here like it’s the public web. we had a long discussion today about whether/how to discourage scraping/archiving — @pfrazee.com?

yeah I'm gonna go Big Technical on this one

every user in atproto has a signed data repository, which is quite similar to a git repository. changes get written in commits and each repository maintains a history (commit log) we do that because the network is largely pull-based, and maintaining the history makes catchup sync efficient

designing the data repository is a bit of an art, because it's a merkle tree and you're balancing the size of proofs for a given record or subset of the data against the flexibility of the data it contains. @dholms.xyz is your guy on how all that works

as it turned out, the way image blobs get stored in the repository make it easy to just drop them and leave dangling references. that's why we're able to ship that quickly records, however, are harder to expunge without losing the ability to verify the proofs

daniel tells me it might be possible to change the data structure so that it's possible, but he had that kind of quiver in his voice when he said it that tells me I shouldnt expect it any time soon that means if we want to expunge the actual records, we have to do a "rebase"

a "rebase" basically means rewriting the history to remove any record of the past writes. generally speaking you want to do this every so often as a form of compaction, because otherwise every repository is going to grow unbounded. so, that's something the team designed for from the start

Two thoughts: 1. Could use use tombstones and compact later? It’s a soft delete so data is technically still there, but should keep meekly intact. 2. Could you use crypto key shredding similar to how PII is dealt with when users want to delete their data for GDPR?