Face to face: The eyes as an anchor in multimodal communication

Making eye contact with our conversational partners is what is most common in multimodal communication. Yet, little is known about this behavior. Prior studies have reported different findings on what we look at in the narrator's face. Some studies show eye gaze is usually focused on our conversational partner's eyes, other studies have shown evidence for eye gaze primarily on the narrator's mouth, and yet others find evidence for fixations on the narrator's nose bridge perhaps as a transition for eye gaze between the eyes and mouth. The current study aimed to shed light on these different findings by investigating eye gaze on a narrator's face in a fixed cognitive task. Experiment 1 monitored participants' eye gaze when looking at videos of a male and female human narrator. Experiment 2 used a virtual human, allowing manipulation of different parts of the narrator's face to validate the findings in Experiment 1. Gaze behavior on the human faces (Experiment 1) and the virtual human face (Experiment 2) of the narrator was similar, with the narrator's eyes attracting most fixations seemingly serving as an anchor for communication, particularly at the start and the end of a conversation. The mouth, in turn, served as a communicative cue when eye contact has been established. When lip movements were impaired in the virtual human, the eyes immediately took over as the anchor again. These findings can be explained by the theoretical framework of action ladders in multimodal language use. They shed light on cognitive and social psychological aspects of human-human multimodal communication, both in human and embodied conversational agents.