Post
there's all kinds of spooky engineering going on with these things; LLMs do voice and image input by using projection layers to rotate the tokens of sound slices and image patches into the same high-dimensional space that text tokens occupy, which automatically grants the thing "understanding" of it
1 repost12 likes