Beyond speech balloons and thought bubbles

The integration of text and image

Neil Cohn*

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Speech balloons and thought bubbles are among the most recognizable visual signs of the visual language used in comics. These enclosed graphic containers provide a way in which text and image can interface with each other. However, their stereotypical meanings as representing speech or thought betray much deeper semantic richness. This paper uses these graphic signs as a platform for examining the multimodal interfaces between text and image, and details four types of interfaces that characterize the connections between modalities: Inherent, Emergent, Adjoined, and Independent relationships. Each interface facilitates different levels of multimodal integration, tempered by principles of Gestalt grouping and underlying semantic features. This process allows the possibility of creating singular cohesive units of text and image that is on par with other multimodal interfaces, such as between speech and gesture.

Original languageEnglish
Pages (from-to)35-63
Number of pages29
JournalSemiotica : revue publ. par l'Association Internationale de Semiotique
Volume197
DOIs
Publication statusPublished - Oct 2013
Externally publishedYes

Keywords

  • visual language
  • comics
  • multimodality
  • graphic communication
  • thought bubbles
  • ORGANIZATION

Cite this

@article{d18e981ef72b4d58b58d38422e042acf,
title = "Beyond speech balloons and thought bubbles: The integration of text and image",
abstract = "Speech balloons and thought bubbles are among the most recognizable visual signs of the visual language used in comics. These enclosed graphic containers provide a way in which text and image can interface with each other. However, their stereotypical meanings as representing speech or thought betray much deeper semantic richness. This paper uses these graphic signs as a platform for examining the multimodal interfaces between text and image, and details four types of interfaces that characterize the connections between modalities: Inherent, Emergent, Adjoined, and Independent relationships. Each interface facilitates different levels of multimodal integration, tempered by principles of Gestalt grouping and underlying semantic features. This process allows the possibility of creating singular cohesive units of text and image that is on par with other multimodal interfaces, such as between speech and gesture.",
keywords = "visual language, comics, multimodality, graphic communication, thought bubbles, ORGANIZATION",
author = "Neil Cohn",
year = "2013",
month = "10",
doi = "10.1515/sem-2013-0079",
language = "English",
volume = "197",
pages = "35--63",
journal = "Semiotica : revue publ. par l'Association Internationale de Semiotique",
issn = "0037-1998",
publisher = "Walter de Gruyter GmbH",

}

Beyond speech balloons and thought bubbles : The integration of text and image. / Cohn, Neil.

In: Semiotica : revue publ. par l'Association Internationale de Semiotique, Vol. 197, 10.2013, p. 35-63.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Beyond speech balloons and thought bubbles

T2 - The integration of text and image

AU - Cohn, Neil

PY - 2013/10

Y1 - 2013/10

N2 - Speech balloons and thought bubbles are among the most recognizable visual signs of the visual language used in comics. These enclosed graphic containers provide a way in which text and image can interface with each other. However, their stereotypical meanings as representing speech or thought betray much deeper semantic richness. This paper uses these graphic signs as a platform for examining the multimodal interfaces between text and image, and details four types of interfaces that characterize the connections between modalities: Inherent, Emergent, Adjoined, and Independent relationships. Each interface facilitates different levels of multimodal integration, tempered by principles of Gestalt grouping and underlying semantic features. This process allows the possibility of creating singular cohesive units of text and image that is on par with other multimodal interfaces, such as between speech and gesture.

AB - Speech balloons and thought bubbles are among the most recognizable visual signs of the visual language used in comics. These enclosed graphic containers provide a way in which text and image can interface with each other. However, their stereotypical meanings as representing speech or thought betray much deeper semantic richness. This paper uses these graphic signs as a platform for examining the multimodal interfaces between text and image, and details four types of interfaces that characterize the connections between modalities: Inherent, Emergent, Adjoined, and Independent relationships. Each interface facilitates different levels of multimodal integration, tempered by principles of Gestalt grouping and underlying semantic features. This process allows the possibility of creating singular cohesive units of text and image that is on par with other multimodal interfaces, such as between speech and gesture.

KW - visual language

KW - comics

KW - multimodality

KW - graphic communication

KW - thought bubbles

KW - ORGANIZATION

U2 - 10.1515/sem-2013-0079

DO - 10.1515/sem-2013-0079

M3 - Article

VL - 197

SP - 35

EP - 63

JO - Semiotica : revue publ. par l'Association Internationale de Semiotique

JF - Semiotica : revue publ. par l'Association Internationale de Semiotique

SN - 0037-1998

ER -