Post

There is so much to gzip. It's a perfect microcosm of the experience of Computering. Fractally complex and clever. Hamstrung by the limitations of computers at the time it was invented. Dramatic, backstabby origin story. Completely unused extension points. Both prescient and myopic in its design.

tl;dr gzip is a thin envelope for DEFLATE. With just that, you can do interesting things, e.g. stargz is spiritually a zip file masquerading as a targz. Going deeper, DEFLATE has enough structure that you can do very cool things, e.g. soci from aws folks uses an old trick to get random access.

Most of this is interesting to me because the world of containers and packages is mostly shuffling gzipped tarballs around. The culmination of this fever dream is oci.dag.dev I extracted some relevant bits to github.com/jonjohnsonjr... (if go is legible to you) Also github.com/google/crfs

I also wrote a poem about all this that only exists publicly in the subscriber-only content of an episode of the now-defunct Ship It podcast so I should probably publish that somewhere else.