Jane Han

nature.comBehaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision - Nature CommunicationsHan et al. show that behavioral judgments of action meaning better capture brain activity than models of semantic and visual content across much of cortex, including the action observation network and...

Finally the ‘new’ paper in Nature Communications! 🧠🎬 @samnastase.bsky.social @haxbylab.bsky.social Action purpose & social content explain cortical representational geometry better than visual & word-embedding models, even in ventral temporal cortex. www.nature.com/articles/s41... 1/6 🧵👇

💛 So grateful to Jim Haxby, Sam Nastase, Ida Gobbini, and all my coauthors, participants & colleagues who made this work possible! It’s wonderful to see this part of my PhD at Dartmouth published. 5/6

🎓 A personal update, too: I completed my PhD at Dartmouth🌲 and joined Daniel Pine’s group at NIH/NIMH as a postdoctoral fellow! Excited to bring my interests in naturalistic cognition and social understanding into developmental affective neuroscience. 🐣🤍 6/6

📊 Action purpose & social content outperformed visual-content judgments, word embeddings, gaze & motion-energy models across much of cortex. Crucially, they explained unique variance: their contribution extended beyond what the other models captured. 3/6

🧠 Highlight: surprisingly, action meaning was strongly represented in ventral temporal cortex, which is traditionally associated with face & object perception. This motivates future work on how the action observation network and ventral visual pathway interact to support natural vision. 4/6

🎬 We measured fMRI responses to 90 real-world action clips. Using behavioral arrangement tasks, we modeled similarity in: 🎯 Action purpose (transitivity) 🤝 Social content 👀 People, objects & scenes Which models best captured neural representational geometry? 2/6

🚨 New paper out with @samnastase.bsky.social and @haxbylab.bsky.social! We use representational similarity analysis to test how well behavioral, semantic, and visual models capture cortical representational geometries when viewing naturalistic action videos: doi.org/10.1101/2024...

🤯 We were surprised at how well these behavioral-arrangement models capture cortical representational geometry, including in areas like VT—our findings suggest that high-level, behaviorally-relevant features of action understanding occupy a privileged role in cortical representation.

🙌 Again, another round of applause and huge thanks to the greatest mentor @sam, who kicked off this exciting project with his dissertation. Your guidance was pivotal. I sincerely would not have survived my PhD journey without @samnastase.bsky.social and @haxbylab.bsky.social ...!

💃 We found that, out of nine models, the behavioral models capturing the meaning of the actions depicted in the stimuli—the transitivity and sociality models—best captured neural representational geometry throughout much of the action observation network.

🥧 Variance partitioning revealed that behavioral models of transitivity and sociality captured a large portion of unique variance throughout the action observation network, and extending into ventral temporal cortex.

🎞️ Finally, we constructed semantic RDMs from word embeddings based on verbs and nonverbs in an annotation of the stimulus, a gaze RDM from a separate eye-tracking sample, and a visual motion energy RDM. This amounted to 3+ hours of fMRI data and 5+ hours of behavioral data per participant (N = 23)!

🧠 We tested all nine of these models against neural representational geometries (with hybrid hyperalignment based on a separate movie stimulus!) using both a searchlight analysis and in regions of interest.

👆 To capture behaviorally-relevant action features, we had participants perform two behavioral arrangement tasks where they organized the action videos according to their object-/goal-related features (transitivity) or their social features (sociality).

🖼️ We also included three other behavioral arrangement tasks where participants organized static images from the video stimuli according to their visual content: scene, person, and object features.