Multisensory integration in speech processing: Neural mechanisms of cross-modal aftereffects

N. Kilian-Hütten, Elia Formisano, J. Vroomen

Research output: Chapter in Book/Report/Conference proceedingChapterScientificpeer-review

10 Downloads (Pure)

Abstract

Traditionally, perceptual neuroscience has focused on unimodal information processing. This is true also for investigations of speech processing, where the auditory modality was the natural focus of interest. Given the complexity of neuronal processing, this was a logical step, considering that the field was still in its infancy. However, it is clear that this restriction does not do justice to the way we perceive the world around us in everyday interactions. Very rarely is sensory information confined to one modality. Instead, we are constantly confronted with a stream of input to several or all senses and already in infancy, we match facial movements with their corresponding sounds (Campbell et al. 2001; Kuhl and Meltzoff 1982). Moreover, the information that is processed by our individual senses does not stay separated. Rather, the different channels interact and influence each other, affecting perceptual interpretations and constructions (Calvert 2001). Consequently, in the last 15–20 years, the perspective in cognitive science and perceptual neuroscience has shifted to include investigations of such multimodal integrative phenomena. Facilitating cross-modal effects have consistently been demonstrated behaviorally (Shimojo and Shams 2001). When multisensory input is congruent (e.g., semantically and/or temporally) it typically lowers detection thresholds (Frassinetti et al. 2002), shortens reaction times (Forster et al. 2002; Schröger and Widmann 1998), and decreases saccadic eye movement latencies (Hughes et al. 1994) as compared to unimodal exposure. When incongruent input is (artificially) added in a second modality, this usually has opposite consequences (Sekuler et al. 1997).
Original languageEnglish
Title of host publicationNeural mechanisms of language
EditorsM. Mody
Place of PublicationBoston, MA
PublisherSpringer Science
Pages105-127
ISBN (Electronic)978-1-4939-7325-5
ISBN (Print)978-1-4939-7323-1
DOIs
Publication statusPublished - 2017

Fingerprint

information processing
speech
exposure
science
effect
sound
justice
detection
world

Cite this

Kilian-Hütten, N., Formisano, E., & Vroomen, J. (2017). Multisensory integration in speech processing: Neural mechanisms of cross-modal aftereffects. In M. Mody (Ed.), Neural mechanisms of language (pp. 105-127). Boston, MA: Springer Science. https://doi.org/10.1007/978-1-4939-7325-5_6
Kilian-Hütten, N. ; Formisano, Elia ; Vroomen, J. / Multisensory integration in speech processing : Neural mechanisms of cross-modal aftereffects. Neural mechanisms of language. editor / M. Mody. Boston, MA : Springer Science, 2017. pp. 105-127
@inbook{4e0155cc592745cb9df78ac56842efb3,
title = "Multisensory integration in speech processing: Neural mechanisms of cross-modal aftereffects",
abstract = "Traditionally, perceptual neuroscience has focused on unimodal information processing. This is true also for investigations of speech processing, where the auditory modality was the natural focus of interest. Given the complexity of neuronal processing, this was a logical step, considering that the field was still in its infancy. However, it is clear that this restriction does not do justice to the way we perceive the world around us in everyday interactions. Very rarely is sensory information confined to one modality. Instead, we are constantly confronted with a stream of input to several or all senses and already in infancy, we match facial movements with their corresponding sounds (Campbell et al. 2001; Kuhl and Meltzoff 1982). Moreover, the information that is processed by our individual senses does not stay separated. Rather, the different channels interact and influence each other, affecting perceptual interpretations and constructions (Calvert 2001). Consequently, in the last 15–20 years, the perspective in cognitive science and perceptual neuroscience has shifted to include investigations of such multimodal integrative phenomena. Facilitating cross-modal effects have consistently been demonstrated behaviorally (Shimojo and Shams 2001). When multisensory input is congruent (e.g., semantically and/or temporally) it typically lowers detection thresholds (Frassinetti et al. 2002), shortens reaction times (Forster et al. 2002; Schr{\"o}ger and Widmann 1998), and decreases saccadic eye movement latencies (Hughes et al. 1994) as compared to unimodal exposure. When incongruent input is (artificially) added in a second modality, this usually has opposite consequences (Sekuler et al. 1997).",
author = "N. Kilian-H{\"u}tten and Elia Formisano and J. Vroomen",
year = "2017",
doi = "10.1007/978-1-4939-7325-5_6",
language = "English",
isbn = "978-1-4939-7323-1",
pages = "105--127",
editor = "M. Mody",
booktitle = "Neural mechanisms of language",
publisher = "Springer Science",

}

Kilian-Hütten, N, Formisano, E & Vroomen, J 2017, Multisensory integration in speech processing: Neural mechanisms of cross-modal aftereffects. in M Mody (ed.), Neural mechanisms of language. Springer Science, Boston, MA, pp. 105-127. https://doi.org/10.1007/978-1-4939-7325-5_6

Multisensory integration in speech processing : Neural mechanisms of cross-modal aftereffects. / Kilian-Hütten, N.; Formisano, Elia; Vroomen, J.

Neural mechanisms of language. ed. / M. Mody. Boston, MA : Springer Science, 2017. p. 105-127.

Research output: Chapter in Book/Report/Conference proceedingChapterScientificpeer-review

TY - CHAP

T1 - Multisensory integration in speech processing

T2 - Neural mechanisms of cross-modal aftereffects

AU - Kilian-Hütten, N.

AU - Formisano, Elia

AU - Vroomen, J.

PY - 2017

Y1 - 2017

N2 - Traditionally, perceptual neuroscience has focused on unimodal information processing. This is true also for investigations of speech processing, where the auditory modality was the natural focus of interest. Given the complexity of neuronal processing, this was a logical step, considering that the field was still in its infancy. However, it is clear that this restriction does not do justice to the way we perceive the world around us in everyday interactions. Very rarely is sensory information confined to one modality. Instead, we are constantly confronted with a stream of input to several or all senses and already in infancy, we match facial movements with their corresponding sounds (Campbell et al. 2001; Kuhl and Meltzoff 1982). Moreover, the information that is processed by our individual senses does not stay separated. Rather, the different channels interact and influence each other, affecting perceptual interpretations and constructions (Calvert 2001). Consequently, in the last 15–20 years, the perspective in cognitive science and perceptual neuroscience has shifted to include investigations of such multimodal integrative phenomena. Facilitating cross-modal effects have consistently been demonstrated behaviorally (Shimojo and Shams 2001). When multisensory input is congruent (e.g., semantically and/or temporally) it typically lowers detection thresholds (Frassinetti et al. 2002), shortens reaction times (Forster et al. 2002; Schröger and Widmann 1998), and decreases saccadic eye movement latencies (Hughes et al. 1994) as compared to unimodal exposure. When incongruent input is (artificially) added in a second modality, this usually has opposite consequences (Sekuler et al. 1997).

AB - Traditionally, perceptual neuroscience has focused on unimodal information processing. This is true also for investigations of speech processing, where the auditory modality was the natural focus of interest. Given the complexity of neuronal processing, this was a logical step, considering that the field was still in its infancy. However, it is clear that this restriction does not do justice to the way we perceive the world around us in everyday interactions. Very rarely is sensory information confined to one modality. Instead, we are constantly confronted with a stream of input to several or all senses and already in infancy, we match facial movements with their corresponding sounds (Campbell et al. 2001; Kuhl and Meltzoff 1982). Moreover, the information that is processed by our individual senses does not stay separated. Rather, the different channels interact and influence each other, affecting perceptual interpretations and constructions (Calvert 2001). Consequently, in the last 15–20 years, the perspective in cognitive science and perceptual neuroscience has shifted to include investigations of such multimodal integrative phenomena. Facilitating cross-modal effects have consistently been demonstrated behaviorally (Shimojo and Shams 2001). When multisensory input is congruent (e.g., semantically and/or temporally) it typically lowers detection thresholds (Frassinetti et al. 2002), shortens reaction times (Forster et al. 2002; Schröger and Widmann 1998), and decreases saccadic eye movement latencies (Hughes et al. 1994) as compared to unimodal exposure. When incongruent input is (artificially) added in a second modality, this usually has opposite consequences (Sekuler et al. 1997).

U2 - 10.1007/978-1-4939-7325-5_6

DO - 10.1007/978-1-4939-7325-5_6

M3 - Chapter

SN - 978-1-4939-7323-1

SP - 105

EP - 127

BT - Neural mechanisms of language

A2 - Mody, M.

PB - Springer Science

CY - Boston, MA

ER -

Kilian-Hütten N, Formisano E, Vroomen J. Multisensory integration in speech processing: Neural mechanisms of cross-modal aftereffects. In Mody M, editor, Neural mechanisms of language. Boston, MA: Springer Science. 2017. p. 105-127 https://doi.org/10.1007/978-1-4939-7325-5_6