Imposing Cognitive Constraints on Reference Production

The Interplay Between Speech and Gesture During Grounding

Ingrid Masson-Carro*, Martijn Goudbeek, Emiel Krahmer

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Past research has sought to elucidate how speakers and addressees establish common ground in conversation, yet few studies have focused on how visual cues such as co-speech gestures contribute to this process. Likewise, the effect of cognitive constraints on multimodal grounding remains to be established. This study addresses the relationship between the verbal and gestural modalities during grounding in referential communication. We report data from a collaborative task where repeated references were elicited, and a time constraint was imposed to increase cognitive load. Our results reveal no differential effects of repetition or cognitive load on the semantic-based gesture rate, suggesting that representational gestures and speech are closely coordinated during grounding. However, gestures and speech differed in their execution, especially under time pressure. We argue that speech and gesture are two complementary streams that might be planned in conjunction but that unfold independently in later stages of language production, with speakers emphasizing the form of their gestures, but not of their words, to better meet the goals of the collaborative task.

A collaborative referencing task is used to examine communication through both verbal and visual modalities in order to understand the cognitive constraints on multimodal grounding in conversation. Analyses of co-speech gesture show that while gesture and speech might be planned in conjunction, that they unfold independently in language production.

Original languageEnglish
Pages (from-to)819-836
Number of pages18
JournalTopics in Cognitive Science
Volume8
Issue number4
DOIs
Publication statusPublished - Oct 2016

Keywords

  • Cognitive load
  • Time pressure
  • Reduction
  • Referring expressions
  • Speech
  • Gesture
  • Grounding
  • Dialogue
  • Language
  • Speaker
  • Load

Cite this

@article{ced9e26d6a6644bbbe09eae90d435b48,
title = "Imposing Cognitive Constraints on Reference Production: The Interplay Between Speech and Gesture During Grounding",
abstract = "Past research has sought to elucidate how speakers and addressees establish common ground in conversation, yet few studies have focused on how visual cues such as co-speech gestures contribute to this process. Likewise, the effect of cognitive constraints on multimodal grounding remains to be established. This study addresses the relationship between the verbal and gestural modalities during grounding in referential communication. We report data from a collaborative task where repeated references were elicited, and a time constraint was imposed to increase cognitive load. Our results reveal no differential effects of repetition or cognitive load on the semantic-based gesture rate, suggesting that representational gestures and speech are closely coordinated during grounding. However, gestures and speech differed in their execution, especially under time pressure. We argue that speech and gesture are two complementary streams that might be planned in conjunction but that unfold independently in later stages of language production, with speakers emphasizing the form of their gestures, but not of their words, to better meet the goals of the collaborative task.A collaborative referencing task is used to examine communication through both verbal and visual modalities in order to understand the cognitive constraints on multimodal grounding in conversation. Analyses of co-speech gesture show that while gesture and speech might be planned in conjunction, that they unfold independently in language production.",
keywords = "Cognitive load, Time pressure, Reduction, Referring expressions, Speech, Gesture, Grounding, Dialogue, Language, Speaker, Load",
author = "Ingrid Masson-Carro and Martijn Goudbeek and Emiel Krahmer",
year = "2016",
month = "10",
doi = "10.1111/tops.12217",
language = "English",
volume = "8",
pages = "819--836",
journal = "Topics in Cognitive Science",
issn = "1756-8757",
publisher = "John Wiley & Sons Inc.",
number = "4",

}

Imposing Cognitive Constraints on Reference Production : The Interplay Between Speech and Gesture During Grounding. / Masson-Carro, Ingrid; Goudbeek, Martijn; Krahmer, Emiel.

In: Topics in Cognitive Science, Vol. 8, No. 4, 10.2016, p. 819-836.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Imposing Cognitive Constraints on Reference Production

T2 - The Interplay Between Speech and Gesture During Grounding

AU - Masson-Carro, Ingrid

AU - Goudbeek, Martijn

AU - Krahmer, Emiel

PY - 2016/10

Y1 - 2016/10

N2 - Past research has sought to elucidate how speakers and addressees establish common ground in conversation, yet few studies have focused on how visual cues such as co-speech gestures contribute to this process. Likewise, the effect of cognitive constraints on multimodal grounding remains to be established. This study addresses the relationship between the verbal and gestural modalities during grounding in referential communication. We report data from a collaborative task where repeated references were elicited, and a time constraint was imposed to increase cognitive load. Our results reveal no differential effects of repetition or cognitive load on the semantic-based gesture rate, suggesting that representational gestures and speech are closely coordinated during grounding. However, gestures and speech differed in their execution, especially under time pressure. We argue that speech and gesture are two complementary streams that might be planned in conjunction but that unfold independently in later stages of language production, with speakers emphasizing the form of their gestures, but not of their words, to better meet the goals of the collaborative task.A collaborative referencing task is used to examine communication through both verbal and visual modalities in order to understand the cognitive constraints on multimodal grounding in conversation. Analyses of co-speech gesture show that while gesture and speech might be planned in conjunction, that they unfold independently in language production.

AB - Past research has sought to elucidate how speakers and addressees establish common ground in conversation, yet few studies have focused on how visual cues such as co-speech gestures contribute to this process. Likewise, the effect of cognitive constraints on multimodal grounding remains to be established. This study addresses the relationship between the verbal and gestural modalities during grounding in referential communication. We report data from a collaborative task where repeated references were elicited, and a time constraint was imposed to increase cognitive load. Our results reveal no differential effects of repetition or cognitive load on the semantic-based gesture rate, suggesting that representational gestures and speech are closely coordinated during grounding. However, gestures and speech differed in their execution, especially under time pressure. We argue that speech and gesture are two complementary streams that might be planned in conjunction but that unfold independently in later stages of language production, with speakers emphasizing the form of their gestures, but not of their words, to better meet the goals of the collaborative task.A collaborative referencing task is used to examine communication through both verbal and visual modalities in order to understand the cognitive constraints on multimodal grounding in conversation. Analyses of co-speech gesture show that while gesture and speech might be planned in conjunction, that they unfold independently in language production.

KW - Cognitive load

KW - Time pressure

KW - Reduction

KW - Referring expressions

KW - Speech

KW - Gesture

KW - Grounding

KW - Dialogue

KW - Language

KW - Speaker

KW - Load

U2 - 10.1111/tops.12217

DO - 10.1111/tops.12217

M3 - Article

VL - 8

SP - 819

EP - 836

JO - Topics in Cognitive Science

JF - Topics in Cognitive Science

SN - 1756-8757

IS - 4

ER -