Post

Jaz
@jaz.sh

Hi protocol friends! I've been cooking up a Very Large PDS (vlpds) implementation capable of scaling up to billions of accounts and down to just one. It uses Object Storage as its only state, supports HA operation, live scaling up and down, is feature-complete with the reference PDS, and is FOSS.

vlpds.jazco.devOverview · vlpds docsAn atproto PDS whose only durable storage is an object store. Any node serves any request, one node can run a personal server for free, and a few nodes can carry all of Bluesky's write load.
Tom Sherman
@tom.sherman.is

So sick, I assume you were inspired by the various s3-based distributed clusters/databases that are making the rounds recently? Celld comes to my mind specifically

Jaz
@jaz.sh

I truly believe that Object Storage is the only real datastore and everything else should be cache and compute on top of it. Databases are fake, Object Storage is the last database. All your systems should be built on top of it and use NVME and RAM as caches and local compute.

4 reposts32 likes

Jaz
@jaz.sh

If you need low latency, simply support shard replicas and accept writes as durable when a quorum of replicas have written the write into memory and let them slowly flush to object storage in efficient batches that amortize operation costs. That's durable enough for pretty much anything.

Jaz
@jaz.sh

Of course for ^ to be durable you want your replicas to be on different physical nodes and maybe different racks but yeah. Horizontally scalable DBs use RAM as a cache and NVME as durable storage with writes durable at R=2 nodes reporting in-memory writes. Now we have RAM -> NVME -> Object Storage.

56quarters.xyz
@56quarters.xyz

This is roughly how Grafana's telemetry databases (Mimir, Loki, Tempo, Pyroscope) work. Writes are replicated to 3 nodes and succeed when 2 accept the write. Data is stored in memory, written to a crash log, and periodically uploaded to object storage.

Jaz
@jaz.sh

This is nice because it allows single node operation and also allows recovery and seamless handoff by pulling from object storage instead of having to do heavy node-to-node reads to catch up or onboard a new node. Your durability is also a lot higher and you can run your local NVME hot.