Post

I've heard this once or twice but haven't seen a full proposal. here are my half-baked thoughts, having looked at blake3 a few years ago for distributed general-purpose data preservation.

first, my understanding is that streaming or mid-file blake3 verification requires the intermediate tree nodes on hand. aka, "tree mode", or "bao". sorta like having MST nodes, or IPFS chunking. my fuzzy memory is that scales with size of file, and is around 0.1% to 1% (?). so 1 MByte+ for 1 GByte

(maybe this is more efficient than I remember, and maybe there is a way to transfer subsets of this metadata? lmk if i'm wrong) that is a big chunk of metadata! unlike a final hash, it doesn't fit simply in an atproto record '$blob' for example, or in a database row.

but it is still really nifty. how could folks experiment with this today? you could store big files outside of atproto blobs (eg, S3 w/ range reads), store the bao hashes as an atproto blob, and reference them both from records. clients fetch the bao meta, then do range reads + streaming verify

one step to make this more "on-protocol" would be to put the full files as blobs, and add support to range-requests on getBlobs endpoint. I'm interested in that generally, though need review of how HTTP redirects would work. also need chunked uploads for big blobs.

I kind of suspect that "big file PDS instances" would be a niche thing though, and have a bunch of other needs, like media processing/transcoding, etc. and hard to migrate. if already niche, why not a sidecar service (like git/knot for tangled)? fixes migration issue, more provider flexibility.

basically I think that BDASL/blake3 in atproto is simple on the surface, but actual use-cases open up a can of worms that BDASL alone doesn't solve. I'd love to see a fully worked use-case and how BDASL is a big unlock for it, and why it can't be done better at application layer or side-car.

I don't want to be a gatekeeper or blocker. I think this is a good topic to discuss openly in any venue (eg, IETF WG, DASL, etc). but I also don't think the specs really are a big blocker to building or moving forward in app layer or "userland".

This was really helpful, thank you. I'm an outsider to the hash side of things, so I went and poked at it to make sure I understood your points. Sharing what I found in case it's useful, and very open to being wrong:

You're right that bao trees are big. I ran the crate on a test file and the full outboard came out to 6.25%. But it's built purely from the file's bytes, so a proxy can build & cache it on first fetch. The PDS doesn't *have* to store it.

And there's a proxy that already wants to: Porxie (@blooym.dev) has to buffer the whole blob to verify the SHA-256 CID before serving it. Its Cargo.toml even has BLAKE3 commented out, "if it ever gets added to the spec." Maybe that's the worked use case? tangled.org/blooym.dev/p...