Post

care to share any details about algorithms for the nerds? :)

Hopefully a post soon 👍 It's a simple dictionary compression scheme, with streaming training, and a format-specific tokenizer.

There are a lot of compression-time gains still left on the table, so we've put effort into making the decompressor broadly forwards-compatible. It'll be interesting to see what we can squeeze out of it over the next few months.

It’s actually a surprisingly simple and effective scheme, which is what I like about it 🙂 It’s JSON-oriented, so we chunk the input by `,` treating those as internable fragments.

Since fragment selection is a forwards-only scan, we can train on live ingest, and take a snapshot of the trained dictionary each time we start a new file.