Post

jsulz
@jsulz.com

On the Xet team at @huggingface.bsky.social we're always looking for ways to move bytes to computer near you as fast as possible. To do this, we're redesigning the upload and download infrastructure on the Hub. This post describes how, check the thread for details 🧵 huggingface.co/blog/rearchi...

huggingface.coRearchitecting Hugging Face Uploads and DownloadsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
jsulz
@jsulz.com

Our goal? Replace Git LFS with Xet-backed storage to make uploads smarter and downloads faster. The analysis? A 24 hour snapshot of upload requests in October - 88 countries, 8.2M requests, and 130.8TB of data - to design a scalable infrastructure.

0 reposts6 likes

jsulz
@jsulz.com

The result? A control plane with a custom protocol routing requests to content addressed stores in 3 AWS regions: us-east-1, eu-west-3, and ap-southeast-1. 🌍