Post

@tsoding.bsky.social

There is probably an alternative universe where the programming language developers didn't use the word "token" to denote a lexeme and the whole weird cryptocurrency-esque discourse about LLM outputs has never happened.

18/10/26, 11:08 pm5 reposts89 likes

RadiantHostility (he/him)

@radianthostility.bsky.social

Tell me more this is extremely I need to know more about this. (I am an AI hater)

the fool (cachinnatory)

@agnoster.net

The fact that both cryptocurrency “tokens” and AI inference “tokens” share a word is an historical accident A “token” in AI is just a word or part of a word, originating from linguistics, and then passing into AI/ML A “token” in crypto is more like the common English term

the fool (cachinnatory)

@agnoster.net

The fact that both use GPUs (really just “the most efficient parallel compute we have”) to produce lots of “tokens” made a lot of folks think that AI and crypto are “basically the same”. Throw in the fact that technology is funded in the US by VC and attracts grifters, and, well, there you go

ai-nerd.bsky.social

@ai-nerd.bsky.social

the word is older than lexers anyway, Peirce drew the type/token distinction in 1906, so even that universe probably ends up with tokens somewhere

So Overfit

@sooverfit.bsky.social

and in that universe you get billed per million lexemes, which sounds like a linguistics department fundraiser