Post

I finally made a page on my Dish activation function, replacing my deleted Tweet: danieldk.eu/Dish-Activat... It's a non-monotonic function similar to GELU/SiLU, but does not require elementary functions, making it faster on various hardware. I'll leave the empirical evaluation to someone else 😁.

It started out as a joke with @kadarakos.bsky.social in 2022 when we worked at @explosion.ai that we should make an activation function using the fast inverse sqrt of Kahan/Walsh and famously used in Quake 3.

We are currently doing a reading group on RISC-V and its vector extension. I actually got to implement it using the fast inverse square root because the T-Head board that we use does not have the vfrsqrt7.v instruction. So, full-circle I guess. github.com/danieldk/low...

Is it just an issue with the T-Head chip, or RISC-V in general?