Post

DBOS
@dbos.dev

S3Mirror is an open source application for transferring large genomic datasets b/t S3 buckets efficiently, reliably, & with full observability. Built on DBOS Transact, it was created to meet the needs of a pharmaceutical organization handling clinical trial data. www.biorxiv.org/content/10.1...

biorxiv.orgS3Mirror: Making Genomic Data Transfers Fast, Reliable, and Observable with DBOSTo meet the needs of a large pharmaceutical organization, we set out to create S3Mirror - an application for transferring large genomic sequencing datasets between S3 buckets quickly, reliably, and ob...

1 repost4 likes

DBOS
@dbos.dev

2/ Transferring data at this scale presents three major challenges: -High data volume -Transfer errors -Limited observability

DBOS
@dbos.dev

3/ S3Mirror addresses all three using DBOS. It retries on intermittent errors, resumes transfers after crashes without redoing work, and logs file-level status to a Postgres database that can be queried at any time.

DBOS
@dbos.dev

4/ Performance benchmarks: -Peak throughput of 24.9 GiB/s -Transferred 11.88 TiB in 8.1 minutes -Incurred $0.10 in DBOS Cloud Pro usage costs vs. $183.03 using AWS DataSync

DBOS
@dbos.dev

DBOS Cloud Pro does require a $99/mo subscription. Even factoring that in, the total cost would still be lower than AWS DataSync. All transfer metadata is stored durably and accessible during and after execution. DBOS ensures consistency, reliability, and real-time visibility into every file moved.