Problem Detection in Human-Machine Interactions based on Facial Expressions of Users

P. Barkhuysen, E.J. Krahmer, M.G.J. Swerts

    Research output: Contribution to journalArticleScientificpeer-review

    26 Citations (Scopus)
    224 Downloads (Pure)


    This paper describes research into audiovisual cues to communication problems in interactions between users and a spoken dialogue system. The study consists of two parts. First, we describe a series of three perception experiments in which subjects are offered film fragments (without any dialogue context) of speakers interacting with a spoken dialogue sys- tem. In half of these fragments, the speaker is or becomes aware of a communication problem. Subjects have to determine by forced choice which are the problematic fragments. In all three tests, subjects are capable of performing this task to some extent, but with varying levels of correct classifications. Second, we report results of an observational analysis in which we first attempt to relate the perceptual results to visual features of the stimuli presented to subjects, and second to find out which visual features actually are potential cues for error detection. Our major finding is that more problematic contexts lead to more dynamic facial expressions, in line with earlier claims that communication errors lead to marked speaker behaviour. We conclude that visual information from a user's face is potentially beneficial for problem detection.
    Original languageEnglish
    Pages (from-to)343-359
    JournalSpeech Communication
    Issue number3
    Publication statusPublished - 2005


    Dive into the research topics of 'Problem Detection in Human-Machine Interactions based on Facial Expressions of Users'. Together they form a unique fingerprint.

    Cite this