Post

Buzz Andersen
@andersen.buzz

Seems like the idea is basically: we thought neural networks weren't going to be interpretable because they're not as structurally simple as "this neuron corresponds to this concept." But actually, there is a higher level logic to how the concepts are organized across the network, so...we're good.

Colin
@colin-fraser.net

ok having read it, let me try. Background knowledge: a neural network, conceptually, has billions of little things called neurons. One goal of so-called "interpretability" would be to be able to look at each neuron and say what it does for the network.

Colin
@colin-fraser.net

However, there is an observed phenomenon called "polysemanticity" wherein it seems that sometimes a single neuron seems to be for multiple different things. An early example of this was a finding that a single neuron in this image model seems to be for both animals and cars.

Pelasgus
@pelasgus.bsky.social

Isn't that literally how genetics works? > The EDAR gene provides instructions for making a protein that acts as a key signaling receptor during embryonic development, controlling the growth and shaping of structures derived from the ectoderm—including hair follicles, sweat glands, teeth, and skin

Colin
@colin-fraser.net

One explanation for this would be the "superposition" hypothesis which holds that the network reuses some neurons for unrelated concepts. Animal and car pictures aren't very alike, this theory says, so it's economical to use the same neuron for both of them.

Colin
@colin-fraser.net

This is understood (by the author, at least) to be a disaster for interpretability since it makes it very hard to accomplish the goal of assigning one concept to one neuron. Perhaps this in fact makes interpretability impossible.