Post

Jascha Sohl-Dickstein
@jascha.sohldickstein.com

Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish to those for which training diverges. Even better, a video: vimeo.com/903855670

31 reposts140 likes

Alexander Soare
@alexandersoare.bsky.social

Very cool! One thing I'm still not sure about is relevance to meta-learning (Discussion E). It's hard to guess from the looking at color maps, but although "best hparams are near the edge", is a perilous journey into fractal-land warranted for the reward vs good hparams in a more stable region?

Jascha Sohl-Dickstein
@jascha.sohldickstein.com

We would avoid it if we could! The challenge is that when you descend the meta-loss by gradient decent, you converge into the perilous region. It's hard to know how to exclude it. Our best approaches so far use stochastic finite difference gradient resonators (variants of ES) to descend a smoothed

Alexander Soare
@alexandersoare.bsky.social

"ES methods convolve the loss surface with a Gaussian distribution in the learnable parameter space, making it smoother and infinitely differentiable" - this very much reminds me of the score-matching interpretation of diffusion models. Taking a look over breakfast...

Jascha Sohl-Dickstein
@jascha.sohldickstein.com

The boundary between trainable and untrainable configurations is *fractal*! (and beautiful!) For details, and more pretty videos, see: blog post: sohl-dickstein.github.io/2024/02/12/f... paper: arxiv.org/abs/2402.06184

sohl-dickstein.github.ioNeural network training makes beautiful fractalsThis blog is intended to be a place to share ideas and results that are too weird, incomplete, or off-topic to turn into an academic paper, but that I think may be important. Let me know what you thin...