Yanchen Wang

Yanchen Wang

@ppwang.bsky.social

CS PhD Student @ Stanford Computational Neuroscience | Neural Data Science | Computer Vision

156 followers34 following11 posts

Pinned

Yanchen Wang
@ppwang.bsky.social

🎉 BEAST3D is a #NeurIPS2026 Oral! Can a moving animal's 3D shape predict what its neurons are doing, better than tracked keypoints? On average, yes. BEAST3D is a self-supervised vision transformer that reconstructs animals in 3D from unlabeled multi-camera lab video. 🧵

Yanchen Wang reposted

Lightning Pose
@lightningpose.bsky.social

Amazing work by @ppwang.bsky.social on self-supervised 3D representations of animals! We show, among other downstream tasks, how the self-supervised backbone can be incorporated into Lightning Pose for even stronger multi-view pose estimation ⚡

Yanchen Wang@ppwang.bsky.social

🎉 BEAST3D is a #NeurIPS2026 Oral! Can a moving animal's 3D shape predict what its neurons are doing, better than tracked keypoints? On average, yes. BEAST3D is a self-supervised vision transformer that reconstructs animals in 3D from unlabeled multi-camera lab video. 🧵

Yanchen Wang
@ppwang.bsky.social

🎉 BEAST3D is a #NeurIPS2026 Oral! Can a moving animal's 3D shape predict what its neurons are doing, better than tracked keypoints? On average, yes. BEAST3D is a self-supervised vision transformer that reconstructs animals in 3D from unlabeled multi-camera lab video. 🧵

View full thread
Yanchen Wang
@ppwang.bsky.social

🎉 BEAST3D is a #NeurIPS2026 Oral! Can a moving animal's 3D shape predict what its neurons are doing, better than tracked keypoints? On average, yes. BEAST3D is a self-supervised vision transformer that reconstructs animals in 3D from unlabeled multi-camera lab video. 🧵

View full thread
Yanchen Wang
@ppwang.bsky.social

For pose, we train Lightning Pose on just 100 labeled frames. BEAST3D has the lowest error on 3 of 4 datasets (VGGT wins on rat, but it's ~6x bigger), and its keypoint traces are smoother than DINOv3's. Multi-view alone isn't enough: VGGT and E-RayZer don't reliably beat single-view models.

Yanchen Wang
@ppwang.bsky.social

We also predict neural activity from the 3D point cloud. Bits per spike, BEAST3D vs 3D keypoints: Mouse facial motor nucleus (8 neurons): 0.39 vs 0.29 (p=0.01) Chickadee hippocampus (52 neurons): 0.11 vs 0.07 (not significant) Our 2D BEAST features score the same, but aren't tied to body locations.

Yanchen Wang
@ppwang.bsky.social

🎉 BEAST3D is a #NeurIPS2026 Oral! Can a moving animal's 3D shape predict what its neurons are doing, better than tracked keypoints? On average, yes. BEAST3D is a self-supervised vision transformer that reconstructs animals in 3D from unlabeled multi-camera lab video. 🧵

View full thread
Yanchen Wang
@ppwang.bsky.social

During training we hide one camera, predict 3D Gaussians from the other views (frozen DINOv3 features + known camera rays), render them from the hidden camera, and compare with the real frame. It also learns to segment the animal from SAM3 masks, so no mask model is needed at inference.

Yanchen Wang
@ppwang.bsky.social

We render each hidden camera from the others and measure foreground PSNR. Animals seen in training, BEAST3D vs the best baseline: Mouse 27.0 vs 13.6 dB Rat 22.4 vs 12.5 dB Chickadee 18.6 vs 10.2 dB Human 22.4 vs 13.2 dB On animals it hasn't seen, it's still 3-8 dB ahead.

Yanchen Wang
@ppwang.bsky.social

🎉 BEAST3D is a #NeurIPS2026 Oral! Can a moving animal's 3D shape predict what its neurons are doing, better than tracked keypoints? On average, yes. BEAST3D is a self-supervised vision transformer that reconstructs animals in 3D from unlabeled multi-camera lab video. 🧵

Yanchen Wang
@ppwang.bsky.social

Off-the-shelf 3D models like VGGT and E-RayZer are built for lots of overlapping views from unknown cameras. On a lab rig, with 4-6 cameras far apart and close to the animal, they break down. Lab cameras are calibrated, though, so BEAST3D just takes the calibration as input.

Yanchen Wang reposted

Lightning Pose
@lightningpose.bsky.social

Hi everybody, the Lightning Pose team is hiring a research software engineer to continue building frontend and backend tools for pose estimation, as well as other behavioral analysis approaches. Please reach out if you have any questions!

This post is unavailable.
Yanchen Wang
@ppwang.bsky.social

Check our new brain foundation model paper!

Cole Hurwitz@colehurwitz.bsky.social

Another step toward a foundation model of the mouse brain: "Neural Encoding and Decoding at Scale (NEDS)" Trained on neural and behavioral data from 70+ mice, NEDS achieves state-of-the-art prediction of behavior (decoding) and neural responses (encoding) on held-out animals. 🐀

Yanchen Wang
@ppwang.bsky.social

Check our paper!

Cole Hurwitz@colehurwitz.bsky.social

Ported over from X! What will a foundation model for the brain look like? 🧠 We argue that it must be able to solve a diverse set of tasks across multiple brain regions and animals. Check out our NeurIPS paper which introduces a multi-region, multi-animal, multi-task model arxiv.org/abs/2407.14668

Yanchen Wang reposted

Cole Hurwitz
@colehurwitz.bsky.social

Ok, so I wanted to try my hand at building a starter pack too :-). This pack includes developers of tools and methods for electrophysiology (focused on extracellular recordings). Please let me know if I missed anyone! Also, we need more people from X to switch to fill this out! go.bsky.app/DFWsuTJ

You are up to date.