How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain

Emiel van Miltenburg

How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

Evaluations of image description systems are typically domain-general: generated descriptions for the held-out test images are either compared to a set of reference descriptions (using automated metrics), or rated by human judges on one or more Likert scales (for fluency, overall quality, and other quality criteria). While useful, these evaluations do not tell us anything about the kinds of image descriptions that systems are able to produce. Or, phrased differently, these evaluations do not tell us anything about the cognitive capabilities of image description systems. This paper proposes a different kind of assessment, that is able to quantify the extent to which these systems are able to describe humans. This assessment is based on a manual characterisation (a context-free grammar) of English entity labels in the PEOPLE domain, to determine the range of possible outputs. We examined 9 systems to see what kinds of labels they actually use. We found that these systems only use a small subset of at most 13 different kinds of modifiers (e.g. tall and short modify HEIGHT, sad and happy modify MOOD), but 27 kinds of modifiers are never used. Future research could study these semantic dimensions in more detail.

Original language	English
Title of host publication	Proceedings of the Second Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN)
Place of Publication	Barcelona, Spain
Publisher	Association for Computational Linguistics
Pages	30-36
Number of pages	7
Publication status	Published - 1 Dec 2020
Event	Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge - online, Saarbrucken, Germany Duration: 1 Dec 2020 → … Conference number: 2 https://www.lantern.uni-saarland.de/2020/

Workshop

Workshop	Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge
Abbreviated title	LANTERN
Country/Territory	Germany
City	Saarbrucken
Period	1/12/20 → …
Internet address	https://www.lantern.uni-saarland.de/2020/

Access to Document

https://www.aclweb.org/anthology/2020.lantern-1.4

Cite this

@inproceedings{506e4c4e9af348679884039a4437630a,

title = "How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain",

abstract = "Evaluations of image description systems are typically domain-general: generated descriptions for the held-out test images are either compared to a set of reference descriptions (using automated metrics), or rated by human judges on one or more Likert scales (for fluency, overall quality, and other quality criteria). While useful, these evaluations do not tell us anything about the kinds of image descriptions that systems are able to produce. Or, phrased differently, these evaluations do not tell us anything about the cognitive capabilities of image description systems. This paper proposes a different kind of assessment, that is able to quantify the extent to which these systems are able to describe humans. This assessment is based on a manual characterisation (a context-free grammar) of English entity labels in the PEOPLE domain, to determine the range of possible outputs. We examined 9 systems to see what kinds of labels they actually use. We found that these systems only use a small subset of at most 13 different kinds of modifiers (e.g. tall and short modify HEIGHT, sad and happy modify MOOD), but 27 kinds of modifiers are never used. Future research could study these semantic dimensions in more detail.",

author = "{van Miltenburg}, Emiel",

year = "2020",

month = dec,

day = "1",

language = "English",

pages = "30--36",

booktitle = "Proceedings of the Second Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN)",

publisher = "Association for Computational Linguistics",

note = "Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge, LANTERN ; Conference date: 01-12-2020",

url = "https://www.lantern.uni-saarland.de/2020/",

}

van Miltenburg, E 2020, How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain. in Proceedings of the Second Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN). Association for Computational Linguistics, Barcelona, Spain, pp. 30-36, Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge, Saarbrucken, Germany, 1/12/20. <https://www.aclweb.org/anthology/2020.lantern-1.4>

How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain. / van Miltenburg, Emiel.
Proceedings of the Second Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN). Barcelona, Spain: Association for Computational Linguistics, 2020. p. 30-36.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain

AU - van Miltenburg, Emiel

N1 - Conference code: 2

PY - 2020/12/1

Y1 - 2020/12/1

N2 - Evaluations of image description systems are typically domain-general: generated descriptions for the held-out test images are either compared to a set of reference descriptions (using automated metrics), or rated by human judges on one or more Likert scales (for fluency, overall quality, and other quality criteria). While useful, these evaluations do not tell us anything about the kinds of image descriptions that systems are able to produce. Or, phrased differently, these evaluations do not tell us anything about the cognitive capabilities of image description systems. This paper proposes a different kind of assessment, that is able to quantify the extent to which these systems are able to describe humans. This assessment is based on a manual characterisation (a context-free grammar) of English entity labels in the PEOPLE domain, to determine the range of possible outputs. We examined 9 systems to see what kinds of labels they actually use. We found that these systems only use a small subset of at most 13 different kinds of modifiers (e.g. tall and short modify HEIGHT, sad and happy modify MOOD), but 27 kinds of modifiers are never used. Future research could study these semantic dimensions in more detail.

AB - Evaluations of image description systems are typically domain-general: generated descriptions for the held-out test images are either compared to a set of reference descriptions (using automated metrics), or rated by human judges on one or more Likert scales (for fluency, overall quality, and other quality criteria). While useful, these evaluations do not tell us anything about the kinds of image descriptions that systems are able to produce. Or, phrased differently, these evaluations do not tell us anything about the cognitive capabilities of image description systems. This paper proposes a different kind of assessment, that is able to quantify the extent to which these systems are able to describe humans. This assessment is based on a manual characterisation (a context-free grammar) of English entity labels in the PEOPLE domain, to determine the range of possible outputs. We examined 9 systems to see what kinds of labels they actually use. We found that these systems only use a small subset of at most 13 different kinds of modifiers (e.g. tall and short modify HEIGHT, sad and happy modify MOOD), but 27 kinds of modifiers are never used. Future research could study these semantic dimensions in more detail.

M3 - Conference contribution

SP - 30

EP - 36

BT - Proceedings of the Second Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN)

PB - Association for Computational Linguistics

CY - Barcelona, Spain

T2 - Workshop on Beyond Vision and LANguage: inTEgrating Real-world kNowledge

Y2 - 1 December 2020

ER -

How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain

Abstract

Workshop

Access to Document

Fingerprint

Cite this