Auditory-Visual Speech Association (AVISA)

twitch, twitch .... I just dosed off and dreamt I was a spider, or, hang on www.pnas.org/doi/10.1073/... ...a behavioral demonstration for two-stage sleep in a spider

Whisper automatic speech recognition and humans are similarly sensitive to accents but use sentence context differently https://pubmed.ncbi.nlm.nih.gov/42929669/

Visual-linguistic cues in manual communication: Neural and behavioral responses to Mouthings in sign-naïve perception www.sciencedirect.com/science/arti... EEG differences for manual signs with/without mouthings (linked to early & late neural modulations) + better accuracy & response speed

Dynamic facial movements shape how confident speakers appear Open Access royalsocietypublishing.org/rspb/article... Tested if dynamic facial movements shape how spoken responses are interpreted by systematically modulating listener inferences of speaker confidence & doubt - are you sure? lowers brow

Language prediction in multimodal contexts: The contribution of iconic gestures to anticipatory sentence comprehension www.sciencedirect.com/science/arti... Pre-reg EEG tested if early-onset gestures pre-activate the meaning of upcoming nouns -> gesture stroke 520 msec pre or 130 msec post noun ...

📚 Citation Classic "Second language speech learning: Theory findings and problems" Flege (1995) Citations: 5700+ Speech Learning Model explaining why L2 sounds are so... 🔗 https://scholar.google.com/scholar?q=Flege+1995+Second+language+speech+learning+theory+findings+and+problems #SpeechScience

Check out this new study by @avbarchet.bsky.social with Andrea Bruera, me @jonasobleser.bsky.social and Gesa Hartwigsen @mpicbs.bsky.social The study looks into which speech processing levels are affected by aging during multitaker listening situations

Phonological working memory but not beat gestures affect L2 word stress perception www.jbe-platform.com/content/jour... Study investigated if beat gestures facilitate L2 stress perception & how this relates to learners’ visuospatial, phonological & executive working memory ->spoiler alert,no benefit

Validating dynamic time warping as a measure of gesture form similarity link.springer.com/content/pdf/... Can DTW capture gesture form similarity? It seems that it can serve as a useful proxy for gesture form similarity - here a framework for preprocessing motion tracking data & calculating DTW 👍

Retrofitting language models to operate over bytes www.nature.com/articles/s41... "To represent text, most large language models use words or word fragments as the tokens, known as subword tokenization...introduce a general method for creating byte-level large language models through byteification"😮

Relationships between speech-in-noise comprehension, lip reading ability and cortical speech tracking in talking avatars as well as natural talking heads https://pubmed.ncbi.nlm.nih.gov/42854931/

Gesturing at the Boundary: CanConcurrentBeat Gestures BiasSpeech Segmentation? osf.io/preprints/ps... "whether a single beat gesture co-occurring with the ambiguous speech (i.e., proximalcontext) can similarly shift boundary placement remains unknown..." Huandi Chena & Hans Rutger Bosker find out :)

Infant Attention to Eyes Versus Mouth Varies With Mouth Movement Speed, Family History of Autism and Language Development link.springer.com/article/10.1... Investigated longitudinal patterns of attention to eyes & mouths in infants withwithout an elevated familial likelihood of autism &/or ADHD

Postdoc opening in my lab to work on NeuroAI approaches to improve prosthetic devices for human hearing. We offer a great training environment and strong mentorship. Prior expertise in machine learning and signal processing are essential. Apply here: careers.peopleclick.com/careerscp/cl...

Rethinking musicians as a model for plasticity and transfer www.cell.com/trends/cogni... Proposes repositioning correlational research from inferring training effects to constraining causal accounts, revealing neurocognitive organization & guiding studies with genetically informed designs 🎵 🧠 🤹‍♀️

And this one

AVSP 2026 has gone, so let's go back to where it started

Time after time: neural codes of speech comprehension https://pubmed.ncbi.nlm.nih.gov/42830249/

📚 Citation Classic "Word association norms mutual information and lexicography" Church & Hanks (1990) Citations: 7600+ Popularized mutual information for finding... 🔗 https://scholar.google.com/scholar?q=Church+Hanks+1990+Word+association+norms+mutual+information+and+lexicography #SpeechScience

Here we go! We can finally share what Sharjeel Shaikh @sharjeelshaikh has been leading the work on with us! Consider participating in our detection challenge

The [DRAFT] program for AVSP 2026 is out! Check out project.inria.fr/avsp2026/wor... A terrific day to be had: Great presentations throughout An excellent keynote project.inria.fr/avsp2026/key... A 50th anniversary event re. the McGurk & MacDonald effect (there may be cake🎂) A lipreading contest😦

Multimodal Communication in Autism Is Reorganized, Not Reduced: Evidence From Speech and Gesture Across Four Languages journals.sagepub.com/doi/10.1177/... Autistic & non-autistic children tell stories, speech & gesture analyzed wrt amount, diversity & complexity.

Exploring the acceptability of a lip-reading app in acute and critical care: Perspectives from patients with tracheostomies, relatives and healthcare professional www.sciencedirect.com/science/arti... Study aimed to explore the acceptability of SRAVI among tracheostomies patients relatives & staff

Multimodal Language Processing in Children With Developmental Language Disorder: The Influence of Verbal and Visuospatial Memory pubs.asha.org/doi/10.1044/... How do verbal & visuospatial memory capacities contribute to the processing & comprehension of pragmatic meanings in children with/out DLD?

📚 Citation Classic "Robust text-independent speaker identification using Gaussian mixture speaker models" Reynolds & Rose (1995) Citations: 7600+ GMM-based... 🔗 https://scholar.google.com/scholar?q=Reynolds+Rose+1995+Robust+text-independent+speaker+identification+Gaussian+mixture #SpeechScience

Larger language models better align with neural representations of natural language elifesciences.org/articles/101... Fitted electrode-wise (ECoG) encoding models from contextual embeddings of each hidden layer of LLMs to predict word-level neural signals. The early layers of larger LLMs do better..

Simultaneous speech and gesture decoding for multimodal communication in paralysis www.nature.com/articles/s41... Shows neural signals recorded with a single high-density electrocorticography implant can support simultaneous decoding of speech & gestures Commentary www.nature.com/articles/d41...

Visual speech enhances phoneme separability in human superior temporal gyrus https://www.biorxiv.org/content/10.64898/2026.09.10.750762v1

Impact of visual mouth image presentation on speech perception in noise in normal hearing subjects and cochlear implant users journals.plos.org/plosone/arti... Computer‑animated avatar provided no speech perception AV benefit; real‑speaker videos enhanced AV speech perception but no impact on effort

The Cortical Contribution to the Speech-Frequency–Following Response Is Not Modulated by Visual Information onlinelibrary.wiley.com/doi/10.1111/... "Our results suggest that visual modulation of the speech-FFR in the auditory cortex is, if existent, too small to be measurable"

Next time when you're thinking of zooming in on that meeting when driving ... "Fast cars, slow words? Tracking processing of concurrent speaking and driving in real time" link.springer.com/article/10.3...

The shrinking landscape of linguistic diversity in the age of large language models www.nature.com/articles/s41... Egads, the trumpet tongues of scholars now sound a single tune; gone, the curate's egg of runny then congealed prose, all is pabulum.

Rhythm Across Species: Situating Human Speech in a Comparative Framework nyaspubs.onlinelibrary.wiley.com/doi/10.1111/... Examined speech rhythm using units used for nonhuman animals: interonset intervals (IOI) found these durations are relatively similar across a sample of 48 languages - we fit in🥁

Go, @andreadevarda.bsky.social! A beautiful and comprehensive study! 🔑 findng: behav. measures are ~fully reducible to simple predictors of processing effort (surprisal, word length+frequency), but for 🧠 measures, LLM embeddings carry additional predictive power, likely capturing aspects of meaning.

The role of high-amplitude bursts of high-gamma activity in naturalistic speech and music listening direct.mit.edu/imag/article... Neuronal avalanches (top 1% of high-gamma activations) may provide an insight into stimulus-relevant, large-scale neural dynamics - its always the bursty noise makers

Evaluating perceptual evidence of cross-modal plasticity in older adults with hearing loss using an AV word mismatch task www.sciencedirect.com/science/arti... Audio/visual mismatch detection task noise/no noise; greater hearing loss not associated with visual benefits or altered gaze to face

📚 Citation Classic "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" Baevski et al (2020) Citations: 9100+ wav2vec 2.0 showed self-supervised pretraining slashes the need for labeled speech. 🔗 https://arxiv.org/abs/2006.11477 #SpeechScience

Sensory context improves language prediction in humans & LLMs www.pnas.org/doi/abs/10.1... Compared humans predicting language in reading, listening, watching AV videos with text-based LLM predictions. People do better, AV ~ AO > reading (some of it is prosody); LMMs do better when given this info👁️👂

Colin Cherry (1953) described how we can focus on one conversation in a noisy room full of people talking. This launched decades of research into selective attention and speech perception. 🎧 Cherry (1953) - foundational auditory perception study #SpeechScience #Perception

Assessing Visual Speech Communication Abilities: Cross-Situational Consistency in Lipreadability brill.com/view/journal... Visual speech abilities: Receivers consistently good or poor at lipreading regardless of the talker;OR, talkers consistently lipread accurately or not, regardless of listener 👁️😮

Viewpoint on Multisensory Time Perception brill.com/view/journal... Shi & van Wassenhove draw on decades of research & argue that the field has undergone an important shift from the centralized internal clock toward timing as an active inference process.

Voice perception reflects graded sensitivity to acoustic cues along the auditory hierarchy www.sciencedirect.com/science/arti... How like a human voice? Acoustic patterns over time; Perceptual & neural features align most in the auditory association area; voiceness: a hierarchical encoding framework

Music Is Quasi-Rhythmic Too: Comparing Bandwidth in Speech and Music Acoustic Modulation Spectra. nyaspubs.onlinelibrary.wiley.com/doi/10.1111/... Confirms findings that speech & music modulation spectra differ in their centre frequency -> but find that the modulation bandwidth is comparable 🎵 ↔️ 💬

Distinct neural processes link speech planning and execution www.nature.com/articles/s41... Busy day on the intracranial recordings research front: planning => prefrontal units => dynamically integrated => execution via speech motor regions representing speech sound units & transitional properties