A multimodal parallel architecture: A cognitive framework for multimodal interactions

Neil Cohn*

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech-gesture and text-image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated "grammatical" sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoffs (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it "dominates" the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the "linguistic" system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality. (C) 2015 Elsevier B.V. All rights reserved.

Original languageEnglish
Pages (from-to)304-323
Number of pages20
JournalCognition
Volume146
DOIs
Publication statusPublished - Jan 2016
Externally publishedYes

Keywords

  • Multimodality
  • Visual language
  • Gesture
  • Comics
  • Narrative
  • Parallel architecture
  • Linguistic models
  • VISUAL NARRATIVE COMPREHENSION
  • SEQUENTIAL IMAGE COMPREHENSION
  • LANGUAGE COMPREHENSION
  • BIMODAL BILINGUALISM
  • WORDS
  • GRAMMAR
  • COMMUNICATION
  • DISCOURSE
  • INFERENCE
  • PICTURES

Cite this

@article{976e5060f23348e1b648c39b06b4d19b,
title = "A multimodal parallel architecture: A cognitive framework for multimodal interactions",
abstract = "Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech-gesture and text-image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated {"}grammatical{"} sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoffs (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it {"}dominates{"} the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the {"}linguistic{"} system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality. (C) 2015 Elsevier B.V. All rights reserved.",
keywords = "Multimodality, Visual language, Gesture, Comics, Narrative, Parallel architecture, Linguistic models, VISUAL NARRATIVE COMPREHENSION, SEQUENTIAL IMAGE COMPREHENSION, LANGUAGE COMPREHENSION, BIMODAL BILINGUALISM, WORDS, GRAMMAR, COMMUNICATION, DISCOURSE, INFERENCE, PICTURES",
author = "Neil Cohn",
year = "2016",
month = "1",
doi = "10.1016/j.cognition.2015.10.007",
language = "English",
volume = "146",
pages = "304--323",
journal = "Cognition",
issn = "0010-0277",
publisher = "Elsevier Science BV",

}

A multimodal parallel architecture : A cognitive framework for multimodal interactions. / Cohn, Neil.

In: Cognition, Vol. 146, 01.2016, p. 304-323.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - A multimodal parallel architecture

T2 - A cognitive framework for multimodal interactions

AU - Cohn, Neil

PY - 2016/1

Y1 - 2016/1

N2 - Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech-gesture and text-image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated "grammatical" sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoffs (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it "dominates" the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the "linguistic" system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality. (C) 2015 Elsevier B.V. All rights reserved.

AB - Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech-gesture and text-image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated "grammatical" sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoffs (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it "dominates" the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the "linguistic" system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality. (C) 2015 Elsevier B.V. All rights reserved.

KW - Multimodality

KW - Visual language

KW - Gesture

KW - Comics

KW - Narrative

KW - Parallel architecture

KW - Linguistic models

KW - VISUAL NARRATIVE COMPREHENSION

KW - SEQUENTIAL IMAGE COMPREHENSION

KW - LANGUAGE COMPREHENSION

KW - BIMODAL BILINGUALISM

KW - WORDS

KW - GRAMMAR

KW - COMMUNICATION

KW - DISCOURSE

KW - INFERENCE

KW - PICTURES

U2 - 10.1016/j.cognition.2015.10.007

DO - 10.1016/j.cognition.2015.10.007

M3 - Article

VL - 146

SP - 304

EP - 323

JO - Cognition

JF - Cognition

SN - 0010-0277

ER -