Post

That's entirely true: they are non-deterministic, which means you have to learn how to use them in a way that takes that into account Sheepdogs are non-deterministic too, they're still invaluable tools for someone who learns how to work with them

Actually, they are deterministic, right ? They don't feel that way because we use random seeds ( we are not deterministic in our inputs)

Even with a fixed seed you can still get non-deterministic results out of most LLMs, because it turns out floating point math isn't commutative so when you run it on GPUs that execute multiple operations in parallel the order in which they finish can have an impact on the final result

Really ? I remember using Phi in local with the default seed, and having always the same outputs ( as I was always testing with the same inputs). And it was a strange feeling when you are used to have different answer from LLMs.

GPU. But quantized, so int8. Perhaps int8 doesn't have this float issues ( really not sure about it, even in int8, I guess some floats are involved at some points). Will test it again when I'm back home 😅