Underreporting of errors in NLG output, and what to do about it

Emiel van Miltenburg; Miruna Clinciu; Ondřej Dušek; Dimitra Gkatzia; Stephanie Inglis; Leo Leppänen; Saad Mahamood; Emma Manning; Stephanie Schoch; Craig Thomson; Luou Wen

Underreporting of errors in NLG output, and what to do about it

Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, Luou Wen

Language, Communication and Cognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

Original language	English
Title of host publication	Proceedings of the 14th International Conference on Natural Language Generation
Place of Publication	Aberdeen, Scotland, UK
Publisher	Association for Computational Linguistics
Pages	140-153
Number of pages	14
Publication status	Published - 1 Aug 2021
Event	The 14th International Conference on Natural Language Generation - Aberdeen (online), Aberdeen, United Kingdom Duration: 20 Sep 2021 → 24 Sep 2021 Conference number: 14

Conference

Conference	The 14th International Conference on Natural Language Generation
Abbreviated title	INLG
Country/Territory	United Kingdom
City	Aberdeen
Period	20/09/21 → 24/09/21

Access to Document

https://aclanthology.org/2021.inlg-1.14

Commendation for Outstanding Position Paper
van Miltenburg, Emiel (Recipient), 21 Sep 2021
Prize

Cite this

van Miltenburg, E., Clinciu, M., Dušek, O., Gkatzia, D., Inglis, S., Leppänen, L., Mahamood, S., Manning, E., Schoch, S., Thomson, C., & Wen, L. (2021). Underreporting of errors in NLG output, and what to do about it. In Proceedings of the 14th International Conference on Natural Language Generation (pp. 140-153). Association for Computational Linguistics. https://aclanthology.org/2021.inlg-1.14

van Miltenburg, Emiel ; Clinciu, Miruna ; Dušek, Ondřej ; Gkatzia, Dimitra ; Inglis, Stephanie ; Leppänen, Leo ; Mahamood, Saad ; Manning, Emma ; Schoch, Stephanie ; Thomson, Craig ; Wen, Luou. / Underreporting of errors in NLG output, and what to do about it. Proceedings of the 14th International Conference on Natural Language Generation. Aberdeen, Scotland, UK : Association for Computational Linguistics, 2021. pp. 140-153

@inproceedings{cb5ec7f25f5a44ee950fd1f20209b93b,

title = "Underreporting of errors in NLG output, and what to do about it",

abstract = "We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.",

author = "{van Miltenburg}, Emiel and Miruna Clinciu and Ond{\v r}ej Du{\v s}ek and Dimitra Gkatzia and Stephanie Inglis and Leo Lepp{\"a}nen and Saad Mahamood and Emma Manning and Stephanie Schoch and Craig Thomson and Luou Wen",

year = "2021",

month = aug,

day = "1",

language = "English",

pages = "140--153",

booktitle = "Proceedings of the 14th International Conference on Natural Language Generation",

publisher = "Association for Computational Linguistics",

note = "The 14th International Conference on Natural Language Generation, INLG ; Conference date: 20-09-2021 Through 24-09-2021",

}

van Miltenburg, E, Clinciu, M, Dušek, O, Gkatzia, D, Inglis, S, Leppänen, L, Mahamood, S, Manning, E, Schoch, S, Thomson, C & Wen, L 2021, Underreporting of errors in NLG output, and what to do about it. in Proceedings of the 14th International Conference on Natural Language Generation. Association for Computational Linguistics, Aberdeen, Scotland, UK, pp. 140-153, The 14th International Conference on Natural Language Generation, Aberdeen, United Kingdom, 20/09/21. <https://aclanthology.org/2021.inlg-1.14>

Underreporting of errors in NLG output, and what to do about it. / van Miltenburg, Emiel; Clinciu, Miruna; Dušek, Ondřej; Gkatzia, Dimitra; Inglis, Stephanie; Leppänen, Leo; Mahamood, Saad; Manning, Emma; Schoch, Stephanie; Thomson, Craig; Wen, Luou.

Proceedings of the 14th International Conference on Natural Language Generation. Aberdeen, Scotland, UK : Association for Computational Linguistics, 2021. p. 140-153.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Underreporting of errors in NLG output, and what to do about it

AU - van Miltenburg, Emiel

AU - Clinciu, Miruna

AU - Dušek, Ondřej

AU - Gkatzia, Dimitra

AU - Inglis, Stephanie

AU - Leppänen, Leo

AU - Mahamood, Saad

AU - Manning, Emma

AU - Schoch, Stephanie

AU - Thomson, Craig

AU - Wen, Luou

N1 - Conference code: 14

PY - 2021/8/1

Y1 - 2021/8/1

N2 - We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

AB - We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

M3 - Conference contribution

SP - 140

EP - 153

BT - Proceedings of the 14th International Conference on Natural Language Generation

PB - Association for Computational Linguistics

CY - Aberdeen, Scotland, UK

T2 - The 14th International Conference on Natural Language Generation

Y2 - 20 September 2021 through 24 September 2021

ER -

Underreporting of errors in NLG output, and what to do about it

Abstract

Conference

Access to Document

Prizes

Commendation for Outstanding Position Paper

Cite this