Post

Steve Simkins
@stevedylan.dev

It's rough out there fam 😅 I think if you only store valid records it helps a little bit with DB size, but doesn't stop the mass firehose of data that comes inbound.

Thomas Karpiniec
@octet-stream.net

~30 days uptime on my indexer, including verification traffic. 👀 I'm only getting away with it because it's running at a VPS company that doesn't care about ingress

bryan newbold
@bnewbold.net

with tap bandwidth do you mean ingress or egress? the full firehose does add up to a lot of ingress every month, but that is no-cost with many hosting providers. in theory you could split out just the tap part to one of those, and keep the rest of your setup where it is

Steve Simkins
@stevedylan.dev

That’s kinda where I ended up. I’m currently running tap on railway and sending webhooks to Cloudflare to handle data processing, but since it’s getting so many records it’s pushing egress costs up.

Matt Kane
@mk.gg

Have you looked at Durable Objects on Cloudflare for listening? It shouldn't cost too much to listen to the Jetstream from there. Ingress and egress are free. You just pay for the DO minutes, which would be within the free tier.

Steve Simkins
@stevedylan.dev

For a while I moved the tap instance to my home server which did work, but eventually the ingress was so high my home internet was getting throttled (kids and wife were not happy lol).

Steve Simkins
@stevedylan.dev

I’m likely going to look at quickslice since it can use jetstream instead, so might be going back to a local setup if it’s not too intense. I think even then I might have to filter out bridgy records because there’s just so many.

Steve Simkins
@stevedylan.dev

I did some deeper digging around my billing for Railway which has been the primary cost, and its kinda split between egress and CPU. Currently switching to a durable object on cloudflare that listens to jetstream which should help a bunch.

bryan newbold
@bnewbold.net

another option would be switching over to jetstream, which can filter by collection (but provides different guarantees, doesn't do backfill, different interface, etc)

flo-bit
@flo-bit.dev

We really need a shared thing for that i think, there’s probably 10+ of those running when we really only need 2-3