Speech-specific audiovisual integration modulates induced theta-band oscillations

Alma Lindborg*, Martijn Baart, Jeroen J. Stekelenburg, Jean Vroomen, Tobias S. Andersen

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4-8 Hz) oscillations. Here, we investigated whether the theta modulation merely reflects mismatch processing or, alternatively, audiovisual integration of speech. We used electroencephalographic recordings from two previously published studies using audiovisual sine-wave speech (SWS), a spectrally degraded speech signal sounding nonsensical to naive perceivers but perceived as speech by informed subjects. Earlier studies have shown that informed, but not naive subjects integrate SWS phonetically with visual speech. In an N1/P2 event-related potential paradigm, we found a significant difference in theta-band activity between informed and naive perceivers of audiovisual speech, suggesting that audiovisual integration modulates induced theta-band oscillations. In a McGurk mismatch negativity paradigm (MMN) where infrequent McGurk stimuli were embedded in a sequence of frequent audio-visually congruent stimuli we found no difference between congruent and McGurk stimuli. The infrequent stimuli in this paradigm are violating both the general prediction of stimulus content, and that of audiovisual congruence. Hence, we found no support for the hypothesis that audiovisual mismatch modulates induced theta-band oscillations. We also did not find any effects of audiovisual integration in the MMN paradigm, possibly due to the experimental design.

Original languageEnglish
Article numbere0219744
Number of pages15
JournalPLoS ONE
Volume14
Issue number7
DOIs
Publication statusPublished - 2019

Keywords

  • ELECTROPHYSIOLOGICAL EVIDENCE
  • MULTISENSORY INTEGRATION
  • CORTICAL OSCILLATIONS
  • SEEING-VOICES
  • VISUAL SPEECH
  • HEARING-LIPS
  • EEG
  • COMPREHENSION
  • FACILITATION
  • PERCEPTION

Cite this

@article{48d94ad2435d424e95b1f0430e149b48,
title = "Speech-specific audiovisual integration modulates induced theta-band oscillations",
abstract = "Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4-8 Hz) oscillations. Here, we investigated whether the theta modulation merely reflects mismatch processing or, alternatively, audiovisual integration of speech. We used electroencephalographic recordings from two previously published studies using audiovisual sine-wave speech (SWS), a spectrally degraded speech signal sounding nonsensical to naive perceivers but perceived as speech by informed subjects. Earlier studies have shown that informed, but not naive subjects integrate SWS phonetically with visual speech. In an N1/P2 event-related potential paradigm, we found a significant difference in theta-band activity between informed and naive perceivers of audiovisual speech, suggesting that audiovisual integration modulates induced theta-band oscillations. In a McGurk mismatch negativity paradigm (MMN) where infrequent McGurk stimuli were embedded in a sequence of frequent audio-visually congruent stimuli we found no difference between congruent and McGurk stimuli. The infrequent stimuli in this paradigm are violating both the general prediction of stimulus content, and that of audiovisual congruence. Hence, we found no support for the hypothesis that audiovisual mismatch modulates induced theta-band oscillations. We also did not find any effects of audiovisual integration in the MMN paradigm, possibly due to the experimental design.",
keywords = "ELECTROPHYSIOLOGICAL EVIDENCE, MULTISENSORY INTEGRATION, CORTICAL OSCILLATIONS, SEEING-VOICES, VISUAL SPEECH, HEARING-LIPS, EEG, COMPREHENSION, FACILITATION, PERCEPTION",
author = "Alma Lindborg and Martijn Baart and Stekelenburg, {Jeroen J.} and Jean Vroomen and Andersen, {Tobias S.}",
year = "2019",
doi = "10.1371/journal.pone.0219744",
language = "English",
volume = "14",
journal = "PLoS ONE",
issn = "1932-6203",
publisher = "PUBLIC LIBRARY SCIENCE",
number = "7",

}

Speech-specific audiovisual integration modulates induced theta-band oscillations. / Lindborg, Alma; Baart, Martijn; Stekelenburg, Jeroen J.; Vroomen, Jean; Andersen, Tobias S.

In: PLoS ONE, Vol. 14, No. 7, e0219744, 2019.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Speech-specific audiovisual integration modulates induced theta-band oscillations

AU - Lindborg, Alma

AU - Baart, Martijn

AU - Stekelenburg, Jeroen J.

AU - Vroomen, Jean

AU - Andersen, Tobias S.

PY - 2019

Y1 - 2019

N2 - Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4-8 Hz) oscillations. Here, we investigated whether the theta modulation merely reflects mismatch processing or, alternatively, audiovisual integration of speech. We used electroencephalographic recordings from two previously published studies using audiovisual sine-wave speech (SWS), a spectrally degraded speech signal sounding nonsensical to naive perceivers but perceived as speech by informed subjects. Earlier studies have shown that informed, but not naive subjects integrate SWS phonetically with visual speech. In an N1/P2 event-related potential paradigm, we found a significant difference in theta-band activity between informed and naive perceivers of audiovisual speech, suggesting that audiovisual integration modulates induced theta-band oscillations. In a McGurk mismatch negativity paradigm (MMN) where infrequent McGurk stimuli were embedded in a sequence of frequent audio-visually congruent stimuli we found no difference between congruent and McGurk stimuli. The infrequent stimuli in this paradigm are violating both the general prediction of stimulus content, and that of audiovisual congruence. Hence, we found no support for the hypothesis that audiovisual mismatch modulates induced theta-band oscillations. We also did not find any effects of audiovisual integration in the MMN paradigm, possibly due to the experimental design.

AB - Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4-8 Hz) oscillations. Here, we investigated whether the theta modulation merely reflects mismatch processing or, alternatively, audiovisual integration of speech. We used electroencephalographic recordings from two previously published studies using audiovisual sine-wave speech (SWS), a spectrally degraded speech signal sounding nonsensical to naive perceivers but perceived as speech by informed subjects. Earlier studies have shown that informed, but not naive subjects integrate SWS phonetically with visual speech. In an N1/P2 event-related potential paradigm, we found a significant difference in theta-band activity between informed and naive perceivers of audiovisual speech, suggesting that audiovisual integration modulates induced theta-band oscillations. In a McGurk mismatch negativity paradigm (MMN) where infrequent McGurk stimuli were embedded in a sequence of frequent audio-visually congruent stimuli we found no difference between congruent and McGurk stimuli. The infrequent stimuli in this paradigm are violating both the general prediction of stimulus content, and that of audiovisual congruence. Hence, we found no support for the hypothesis that audiovisual mismatch modulates induced theta-band oscillations. We also did not find any effects of audiovisual integration in the MMN paradigm, possibly due to the experimental design.

KW - ELECTROPHYSIOLOGICAL EVIDENCE

KW - MULTISENSORY INTEGRATION

KW - CORTICAL OSCILLATIONS

KW - SEEING-VOICES

KW - VISUAL SPEECH

KW - HEARING-LIPS

KW - EEG

KW - COMPREHENSION

KW - FACILITATION

KW - PERCEPTION

U2 - 10.1371/journal.pone.0219744

DO - 10.1371/journal.pone.0219744

M3 - Article

VL - 14

JO - PLoS ONE

JF - PLoS ONE

SN - 1932-6203

IS - 7

M1 - e0219744

ER -