Post

I built a Bluesky post backlink index for @shelf.cafe and wrote up all the juicy technical details. Features kite+jetstream, @microcosm.blue's constellation, and amazingly, bloom filters! One of the most fun blog posts I've written. jola.dev/posts/buildi...

jola.devMaking your own atproto backlink indexI built an atproto backlink index using Latch for primitives and Kite for the jetstream, and I sped it up with some neat bloom filter and erlang atomics action.

13/10/26, 12:57 pm8 reposts55 likes

nate

@zzstoatzz.io

bloom filters rock! also, i learned some stuff from @calabro.io ‘s gloom lib github.com/jcalabro/gloom

Jim Calabro ♟️

@calabro.io

I've got a blog post in drafts on this, need to dust it off

👀

Calling it an unnecessary optimization and then doing it properly on :atomics is my kind of blog post. Bloom filters can't forget, so does a deleted submission trigger a rebuild, or just cost the extra database check?

johanna

@jola.dev

Costs unnecessary database check! But at the size of this bloom filter, even at a million dead items it would only impact false positive rate by 1 in 500K

awesome!!

Thanks great read! How ressource hungry is that? For low traffic apps it's pretty easy to make a very low cost appview, but I still use Constellation for backlinks on askimut.at because I didn't felt like listening to the whole network (or all of bsky posts). But maybe I'm wrong?

askimut.atAskimut

johanna

@jola.dev

No noticeable impact on CPU usage on my 8 CPU server when I added this. Even if I removed the collection filter it would keep up just fine at peak with one process, based on the rough estimates I did at the time, although I didn't write any numbers down

Etienne

@etienneburdet.com

Will try it then, 🙏 I was a bit afraid of parsing every bsky post, but why not after all.