Multisensory integration of speech sounds with letters vs. visual speech

Only visual speech induces the mismatch negativity

Research output: Contribution to journalArticleScientificpeer-review

13 Downloads (Pure)

Abstract

Numerous studies have demonstrated that the vision of lip movements can alter the perception of auditory speech syllables (McGurk effect). While there is ample evidence for integration of text and auditory speech, there are only a few studies on the orthographic equivalent of the McGurk effect. Here, we examined whether written text, like visual speech, can induce an illusory change in the perception of speech sounds on both the behavioural and neural levels. In a sound categorization task, we found that both text and visual speech changed the identity of speech sounds from an /aba/-/ada/ continuum, but the size of this audiovisual effect was considerably smaller for text than visual speech. To examine at which level in the information processing hierarchy these multisensory interactions occur, we recorded electroencephalography in an audiovisual mismatch negativity (MMN, a component of the event-related potential reflecting preattentive auditory change detection) paradigm in which deviant text or visual speech was used to induce an illusory change in a sequence of ambiguous sounds halfway between /aba/ and /ada/. We found that only deviant visual speech induced an MMN, but not deviant text, which induced a late P3-like positive potential. These results demonstrate that text has much weaker effects on sound processing than visual speech does, possibly because text has different biological roots than visual speech.

Original languageEnglish
Pages (from-to)1135-1145
JournalEuropean Journal of Neuroscience
Volume47
Issue number9
DOIs
Publication statusPublished - 2018

Fingerprint

Phonetics
Electroencephalography

Keywords

  • ATTENTION
  • AUDIOVISUAL INTEGRATION
  • CORTEX
  • CROSSMODAL BINDING
  • INFORMATION
  • MMN
  • McGurk-MMN
  • PERCEPTION
  • RECALIBRATION
  • SEEING VOICES
  • TEXT
  • event-related potentials
  • text-sound integration
  • visual speech-sound integration

Cite this

@article{4983e79c037448228d8170d0943f2a71,
title = "Multisensory integration of speech sounds with letters vs. visual speech: Only visual speech induces the mismatch negativity",
abstract = "Numerous studies have demonstrated that the vision of lip movements can alter the perception of auditory speech syllables (McGurk effect). While there is ample evidence for integration of text and auditory speech, there are only a few studies on the orthographic equivalent of the McGurk effect. Here, we examined whether written text, like visual speech, can induce an illusory change in the perception of speech sounds on both the behavioural and neural levels. In a sound categorization task, we found that both text and visual speech changed the identity of speech sounds from an /aba/-/ada/ continuum, but the size of this audiovisual effect was considerably smaller for text than visual speech. To examine at which level in the information processing hierarchy these multisensory interactions occur, we recorded electroencephalography in an audiovisual mismatch negativity (MMN, a component of the event-related potential reflecting preattentive auditory change detection) paradigm in which deviant text or visual speech was used to induce an illusory change in a sequence of ambiguous sounds halfway between /aba/ and /ada/. We found that only deviant visual speech induced an MMN, but not deviant text, which induced a late P3-like positive potential. These results demonstrate that text has much weaker effects on sound processing than visual speech does, possibly because text has different biological roots than visual speech.",
keywords = "ATTENTION, AUDIOVISUAL INTEGRATION, CORTEX, CROSSMODAL BINDING, INFORMATION, MMN, McGurk-MMN, PERCEPTION, RECALIBRATION, SEEING VOICES, TEXT, event-related potentials, text-sound integration, visual speech-sound integration",
author = "J.J. Stekelenburg and M.N. Keetels and J. Vroomen",
note = "{\circledC} 2018 The Authors. European Journal of Neuroscience published by Federation of European Neuroscience Societies and John Wiley & Sons Ltd.",
year = "2018",
doi = "10.1111/ejn.13908",
language = "English",
volume = "47",
pages = "1135--1145",
journal = "European Journal of Neuroscience",
issn = "0953-816X",
publisher = "Wiley-Blackwell",
number = "9",

}

TY - JOUR

T1 - Multisensory integration of speech sounds with letters vs. visual speech

T2 - Only visual speech induces the mismatch negativity

AU - Stekelenburg, J.J.

AU - Keetels, M.N.

AU - Vroomen, J.

N1 - © 2018 The Authors. European Journal of Neuroscience published by Federation of European Neuroscience Societies and John Wiley & Sons Ltd.

PY - 2018

Y1 - 2018

N2 - Numerous studies have demonstrated that the vision of lip movements can alter the perception of auditory speech syllables (McGurk effect). While there is ample evidence for integration of text and auditory speech, there are only a few studies on the orthographic equivalent of the McGurk effect. Here, we examined whether written text, like visual speech, can induce an illusory change in the perception of speech sounds on both the behavioural and neural levels. In a sound categorization task, we found that both text and visual speech changed the identity of speech sounds from an /aba/-/ada/ continuum, but the size of this audiovisual effect was considerably smaller for text than visual speech. To examine at which level in the information processing hierarchy these multisensory interactions occur, we recorded electroencephalography in an audiovisual mismatch negativity (MMN, a component of the event-related potential reflecting preattentive auditory change detection) paradigm in which deviant text or visual speech was used to induce an illusory change in a sequence of ambiguous sounds halfway between /aba/ and /ada/. We found that only deviant visual speech induced an MMN, but not deviant text, which induced a late P3-like positive potential. These results demonstrate that text has much weaker effects on sound processing than visual speech does, possibly because text has different biological roots than visual speech.

AB - Numerous studies have demonstrated that the vision of lip movements can alter the perception of auditory speech syllables (McGurk effect). While there is ample evidence for integration of text and auditory speech, there are only a few studies on the orthographic equivalent of the McGurk effect. Here, we examined whether written text, like visual speech, can induce an illusory change in the perception of speech sounds on both the behavioural and neural levels. In a sound categorization task, we found that both text and visual speech changed the identity of speech sounds from an /aba/-/ada/ continuum, but the size of this audiovisual effect was considerably smaller for text than visual speech. To examine at which level in the information processing hierarchy these multisensory interactions occur, we recorded electroencephalography in an audiovisual mismatch negativity (MMN, a component of the event-related potential reflecting preattentive auditory change detection) paradigm in which deviant text or visual speech was used to induce an illusory change in a sequence of ambiguous sounds halfway between /aba/ and /ada/. We found that only deviant visual speech induced an MMN, but not deviant text, which induced a late P3-like positive potential. These results demonstrate that text has much weaker effects on sound processing than visual speech does, possibly because text has different biological roots than visual speech.

KW - ATTENTION

KW - AUDIOVISUAL INTEGRATION

KW - CORTEX

KW - CROSSMODAL BINDING

KW - INFORMATION

KW - MMN

KW - McGurk-MMN

KW - PERCEPTION

KW - RECALIBRATION

KW - SEEING VOICES

KW - TEXT

KW - event-related potentials

KW - text-sound integration

KW - visual speech-sound integration

U2 - 10.1111/ejn.13908

DO - 10.1111/ejn.13908

M3 - Article

VL - 47

SP - 1135

EP - 1145

JO - European Journal of Neuroscience

JF - European Journal of Neuroscience

SN - 0953-816X

IS - 9

ER -