Relative Contribution of Auditory and Visual Information to Mandarin Chinese Tone Identification by Native and Tone-naive Listeners

Yueqiao Han*, Martijn Goudbeek, Maria Mos, Marc Swerts

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Speech perception is a multisensory process: what we hear can be affected by what we see. For instance, the McGurk effect occurs when auditory speech is presented in synchrony with discrepant visual information. A large number of studies have targeted the McGurk effect at the segmental level of speech (mainly consonant perception), which tends to be visually salient (lip-reading based), while the present study aims to extend the existing body of literature to the suprasegmental level, that is, investigating a McGurk effect for the identification of tones in Mandarin Chinese. Previous studies have shown that visual information does play a role in Chinese tone perception, and that the different tones correlate with variable movements of the head and neck. We constructed various tone combinations of congruent and incongruent auditory-visual materials (10 syllables with 16 tone combinations each) and presented them to native speakers of Mandarin Chinese and speakers of tone-naive languages. In line with our previous work, we found that tone identification varies with individual tones, with tone 3 (the low-dipping tone) being the easiest one to identify, whereas tone 4 (the high-falling tone) was the most difficult one. We found that both groups of participants mainly relied on auditory input (instead of visual input), and that the auditory reliance for Chinese subjects was even stronger. The results did not show evidence for auditory-visual integration among native participants, while visual information is helpful for tone-naive participants. However, even for this group, visual information only marginally increases the accuracy in the tone identification task, and this increase depends on the tone in question.

Original languageEnglish
Article number0023830919889995
Number of pages21
JournalLanguage and Speech
DOIs
Publication statusPublished - 30 Dec 2019

Keywords

  • Auditory-visual perception
  • McGurk effect
  • Mandarin Chinese tone identification
  • native and tone-naive participants
  • congruent and incongruent
  • SPEECH-PERCEPTION
  • NORMAL-HEARING
  • LEXICAL TONES
  • LANGUAGE
  • ENGLISH
  • INTELLIGIBILITY
  • RECOGNITION
  • CLEAR
  • PITCH
  • LIPS

Cite this