Post

In the last few months, we've been working on making Neon storage faster. Here are 7 improvements we shipped to boost performance under the hood 🧵

1/ Sharded ingest We stopped broadcasting WAL to all shards and now route writes only where they’re needed. Less network traffic, lower CPU, and ~2× faster ingestion for large tenants.

2/ WAL compression WAL is now compressed with Zstd before it’s sent to Pageservers. This cut Safekeeper → Pageserver bandwidth by ~70%.

3/ Faster disk writes Safekeepers now use sparse files and fewer flushes. WAL write throughput improved by 230%.

4/ Scan prefetching Instead of fetching one page at a time, Pageservers now prefetch up to 100. In our benchmarks, schema finalization dropped from 10h to 2.5h.

5/ Smarter compaction We now prioritize L0 compaction to avoid read amplification during ingestion. This dropped p99 read amplification by >50%.

6/ Parallel S3 uploads Uploads no longer block layer writes. Upload throughput doubled, and ingest time in our benchmarks dropped.