Post

I've heard this once or twice but haven't seen a full proposal. here are my half-baked thoughts, having looked at blake3 a few years ago for distributed general-purpose data preservation.

first, my understanding is that streaming or mid-file blake3 verification requires the intermediate tree nodes on hand. aka, "tree mode", or "bao". sorta like having MST nodes, or IPFS chunking. my fuzzy memory is that scales with size of file, and is around 0.1% to 1% (?). so 1 MByte+ for 1 GByte

(maybe this is more efficient than I remember, and maybe there is a way to transfer subsets of this metadata? lmk if i'm wrong) that is a big chunk of metadata! unlike a final hash, it doesn't fit simply in an atproto record '$blob' for example, or in a database row.

but it is still really nifty. how could folks experiment with this today? you could store big files outside of atproto blobs (eg, S3 w/ range reads), store the bao hashes as an atproto blob, and reference them both from records. clients fetch the bao meta, then do range reads + streaming verify

I kind of suspect that "big file PDS instances" would be a niche thing though, and have a bunch of other needs, like media processing/transcoding, etc. and hard to migrate. if already niche, why not a sidecar service (like git/knot for tangled)? fixes migration issue, more provider flexibility.

basically I think that BDASL/blake3 in atproto is simple on the surface, but actual use-cases open up a can of worms that BDASL alone doesn't solve. I'd love to see a fully worked use-case and how BDASL is a big unlock for it, and why it can't be done better at application layer or side-car.

I don't want to be a gatekeeper or blocker. I think this is a good topic to discuss openly in any venue (eg, IETF WG, DASL, etc). but I also don't think the specs really are a big blocker to building or moving forward in app layer or "userland".