Been working on this for a while now. My first research paper! It's in manuscript form, so feedback welcome!
We’re sharing a public manuscript from QRK Labs: Density-Adaptive Bitmask Encoding (DABE) for tokenisation At matched bitrate DABE outperforms BPE: 72% -> 89% token recovery, ~18 -> ~7 errors per 64-token chunk. Feedback welcome. Manuscript: qrk.ng/research/den... Summary: qrk.ng/research/den...
0 reposts0 likes