Post

On the Xet team at @huggingface.bsky.social we're always looking for ways to move bytes to computer near you as fast as possible. To do this, we're redesigning the upload and download infrastructure on the Hub. This post describes how, check the thread for details 🧵 huggingface.co/blog/rearchi...

Our goal? Replace Git LFS with Xet-backed storage to make uploads smarter and downloads faster. The analysis? A 24 hour snapshot of upload requests in October - 88 countries, 8.2M requests, and 130.8TB of data - to design a scalable infrastructure.

The result? A control plane with a custom protocol routing requests to content addressed stores in 3 AWS regions: us-east-1, eu-west-3, and ap-southeast-1. 🌍