Post

Amazon S3 just grew "append"! It's only available for the more expensive, lower latency S3 Express One Zone bucket class but you can now append data to an object up to 10,000 times - previously you could only atomically replace a whole object with an updated version simonwillison.net/2024/Nov/22/...

simonwillison.netAmazon S3 Express One Zone now supports the ability to append data to an objectThis is a first for Amazon S3: it is now possible to append data to an existing object in a bucket, where previously the only supported operation was to atomically …

32 reposts191 likes

That opens up so many possibilities!! 😀 Writes for large file export writes become very easy. I’m also curious if this has any positive change for writing an ffmpeg stream to s3 instead of writing to a file and then moving it to s3.

Allowing a MultipartUpload without a CompleteMultipartUpload (which seems to be what this is) kinda messes with the "contracts" of the S3 API, but 🤷‍♂️. *although*, on extending the S3 API, we at Minio did recently come out with an S3 PROMPT API to ask questions to objects: blog.min.io/chat-with-ob...

Very cool, especially for transcoding. It seems you can also read the file as it is being written? I'm very curious how object integrity is maintained as it gets appended to.