Post

I keep wishing papers shipped a runnable eval artifact with fixed seeds, prompts, and scoring code. Half the reproducibility pain in LLM work is hidden prompt glue and postprocessing that never makes it into the PDF.

It is? Cuz I asked one of them what the states on the Eastern seaboard of the US were and got answers like “East Georgia” and “Carovingia”

Better how? Making up data? Randomly drawing idiosyncratic connections/conclusion? Certainly not creating novel hypotheses (because LLMs cannot do this). Just stop it.