Post

fig (aka:[phil])
@bad-example.com

learned so far: - 4 cores seems _enough to run_ but on this particular host it’s kind of marginal. 16GB ram seems plenty. - it’s been dropping pds websockets, and doesn’t seem to recover on its own. not sure why. - bandwidth is less than i was expecting. - +jetstream on this server is too much

fig (aka:[phil])@bad-example.com

kinda still a relay noob but uh i think it’s doing the thing non-us jurisdiction (montreal) non-us-company-cloud-provider relay. chillin.

fig (aka:[phil])
@bad-example.com

- pds backfill is rough on this server, cpu pegs 100% - made worse by having jetstream on the same server trying to do a higher event rate - also interesting: jetstream just crashes when it’s relay websocket closes? fine if the supervisor auto-restarts it i guess.

0 reposts24 likes

fig (aka:[phil])
@bad-example.com

- 4 shared cores on a 2017-era 2ghz epyc is not enough for a relay. could probably tune some things to get it there—seeing significant overhead from userspace networking + logging—but i def want more performance margin than this. for $5.5CAD i can get two more shared cores, or double that for 8

Jaz
@jaz.sh

Run your container in host network mode, I bet 80% of that CPU is copying network packets around. Alternatively don't use a user space proxy for your container.

fig (aka:[phil])
@bad-example.com

will try!! on my list before ditching this server to just drop the whole container and run it in the host (it’s a dedicated vm for it already) then to remember enough iptables to restrict public interface traffic from reaching it directly, or hack it bind on localhost instead of 0.0.0.0

fig (aka:[phil])
@bad-example.com

jetstream is up!

fig (aka:[phil])@bad-example.com

indie jetstream from an indie atproto relay? yeah we got that too: wss://jetstream.fire.hose.cam/subscribe obligatory pdsls demo: pdsls.dev/jetstream?in... here, have an @tools: atp.tools/jetstream (open the little expand-y thing to add the wss URL)

fig (aka:[phil])
@bad-example.com

this provider has a very tempting “dedicated 4 cores” offering that might work but it can’t be configured with enough storage for much of a replay window. otherwise i’m looking at ~$120/mo and that gets me a whole big real dedicated *machine*. maybe if i push for some kofi/github sponsors i could.

fig (aka:[phil])
@bad-example.com

i *should* know more about what’s happening but: i think it’s doing a full sync for each pds? would be nice to say “just trust the latest root and validate going forward from there” with sync1.1, but maybe i don’t know what i’m talking about. manual requestCrawl per PDS and letting it finish is ~3m

fig (aka:[phil])
@bad-example.com

i have unlimited bandwidth throttled at 250mbit/s. just guessing from btop it appears to spend ~30s downloading from the PDS and then a while after processing. in between it looks pretty good! but there are many mushrooms to go.

fig (aka:[phil])
@bad-example.com

see, i will come back to do something with docker after a while not needing to touch it and i just do not want to have to remember things like “quotes are interpreted as literal around values in .env files” (seems this is why relay_trusted_domains and log level weren’t working)

fig (aka:[phil])
@bad-example.com

because of the log thing, about 65% of one of the four cpu cores is just journalctl+syslog(?!) (thanks podman) i’m in to deep syncing bluesky pdss now to restart for it, it’s just slowing things down til next time indigo/relay hit list new item - [ ] set up runtime slog level changes

Jaz
@jaz.sh

We generally use svlogd in production to handle logs and log rotation like this in the docker compose: command: dumb-init sh -c 'relay serve 2>&1 | svlogd -b100000000 /log' volumes: - type: bind source: ${LOG_DIR_ON_HOST} target: /log And add a 'config' file: s1000000000 n2