Post

fig (aka:[phil])
@bad-example.com

learned so far: - 4 cores seems _enough to run_ but on this particular host it’s kind of marginal. 16GB ram seems plenty. - it’s been dropping pds websockets, and doesn’t seem to recover on its own. not sure why. - bandwidth is less than i was expecting. - +jetstream on this server is too much

fig (aka:[phil])@bad-example.com

kinda still a relay noob but uh i think it’s doing the thing non-us jurisdiction (montreal) non-us-company-cloud-provider relay. chillin.

fig (aka:[phil])
@bad-example.com

- pds backfill is rough on this server, cpu pegs 100% - made worse by having jetstream on the same server trying to do a higher event rate - also interesting: jetstream just crashes when it’s relay websocket closes? fine if the supervisor auto-restarts it i guess.

fig (aka:[phil])
@bad-example.com

- 4 shared cores on a 2017-era 2ghz epyc is not enough for a relay. could probably tune some things to get it there—seeing significant overhead from userspace networking + logging—but i def want more performance margin than this. for $5.5CAD i can get two more shared cores, or double that for 8

0 reposts17 likes

Jaz
@jaz.sh

Run your container in host network mode, I bet 80% of that CPU is copying network packets around. Alternatively don't use a user space proxy for your container.

fig (aka:[phil])
@bad-example.com

will try!! on my list before ditching this server to just drop the whole container and run it in the host (it’s a dedicated vm for it already) then to remember enough iptables to restrict public interface traffic from reaching it directly, or hack it bind on localhost instead of 0.0.0.0

fig (aka:[phil])
@bad-example.com

jetstream is up!

fig (aka:[phil])@bad-example.com

indie jetstream from an indie atproto relay? yeah we got that too: wss://jetstream.fire.hose.cam/subscribe obligatory pdsls demo: pdsls.dev/jetstream?in... here, have an @tools: atp.tools/jetstream (open the little expand-y thing to add the wss URL)

fig (aka:[phil])
@bad-example.com

Relay operation learnings, storage edition - 48h content retention window: 582GB. Weekends are a little slower on atproto so say ~300GB/retention-day typical. This is less than I expected (i got 4TB) - sync state and everything else in postgres: 2.2GB. this thing is pretty efficient ✨

Eric
@edavis.dev

did you have to do anything special to get this working with pdsls? I have Caddy fronting one and it works with websocat but not with pdsls

fig (aka:[phil])
@bad-example.com

this provider has a very tempting “dedicated 4 cores” offering that might work but it can’t be configured with enough storage for much of a replay window. otherwise i’m looking at ~$120/mo and that gets me a whole big real dedicated *machine*. maybe if i push for some kofi/github sponsors i could.

fig (aka:[phil])
@bad-example.com

i *should* know more about what’s happening but: i think it’s doing a full sync for each pds? would be nice to say “just trust the latest root and validate going forward from there” with sync1.1, but maybe i don’t know what i’m talking about. manual requestCrawl per PDS and letting it finish is ~3m

fig (aka:[phil])
@bad-example.com

i have unlimited bandwidth throttled at 250mbit/s. just guessing from btop it appears to spend ~30s downloading from the PDS and then a while after processing. in between it looks pretty good! but there are many mushrooms to go.