Post

Relay operation learnings, storage edition - 48h content retention window: 582GB. Weekends are a little slower on atproto so say ~300GB/retention-day typical. This is less than I expected (i got 4TB) - sync state and everything else in postgres: 2.2GB. this thing is pretty efficient ✨

wait 582GB is the total disk usage according to btop, but `du` says 384GB for the persist folder. so even better than ^^, I guess mod whatever filesystem overhead etc.

got victoriametrics scraping node_exporter on the relay host so i can let btop take a rest finally. and get some better bandwidth numbers. whew. (insert tailscale-is-cool take about how it's literally the same for me to do this on the montreal relay vs my at-home pi from the at-home vicmetrics)

back from a lil crash and machine reboot: looks like the bluesky PDS sync is going to need a similar level of patience as the initial sync. instead of babysitting each bsky pds server, i’ve requestCrawled all of them at once. it’s now only pulling ~150mbps but hopefully will continue to progress

it might be some hours until it’s fully live again. should work if you connect now, just with records pretty delayed. once it’s caught up with bsky servers, i’ll reconnect all the indie PDSs that didn’t connect on reboot.

oh come onnnnn, i might be on a heavily oversold server their “dedicated cpu” offering is starting to look more attractive 😭

all ur CPU is getting yoinked :c

this noisy neighbour! (…i guess i am also their noisy neighbour, probably)

pain

wow k so that’s 4 hours to settle after connecting to 10 bsky pds servers (~1M users) although 2h was another CPU steal event (boo) there are 80 servers. i think something is not happening the way it’s supposed to when i connect.

My guess is you may be connecting with an old cursor or cursor=0 or something like that and end up replaying the whole event buffer or smth maybe?

i keep wondering that. there are an awful lot of “dropping commit with old rev” messages after connect

verifying by replaying all of puffball to my house to compare the event counts. (it’s very cool that the sync xrpc is same for relay/pds) if this is what’s happening then technically i can say i’ve done a full network backfill already

*coral (puffball is bigg) very very sketchy way of counting but it seems like less than full backfill. i *think* it’s just backfill since the last time it was up. it’s been down for quite a bit now, guess i should just hard-reset it

PDS firehose backfill windows are roughly 24 hours and roughly 41-45M events per day so for ~80 prod PDS instances the you'd expect somewhere around 500k events in the full backfill window on average but the earlier PDSs end up with a lot more activity than more recent ones so 100k-1M range is right

*41-45M events per day is the total firehose throughput, not for a single instance

ohhhhh huh I didn't realize pds backfill was also a limited window (this makes 1000% sense) i cleared out the db and persist files and now connecting to new PDSs is really snappy. i think it's mostly a matter of *any* full-network backfill just being a very big lift for this lil server.

i think it also makes sense that it was a struggle to get it sync'd the first time, i was restarting a bunch, so the final run-up had a lot of backfill by the time i was getting to the last servers. very short window for me to recover when it crashes 😬

i lowered host concurrency to 4, since it seems to get pretty cpu-bound when connecting to many mushrooms at the same time. will see if it helps. (going to be redeploying this on a bigger machine soon though. yay community)

I have a patch coming which may help with situations where hosts get added, never really consumed, and then process restarts. I think the "pull list of hosts" thing is probably always going to be bump for smaller servers; maybe a delay between re-connection would help in that case?