A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019)

Anouck Braggaar; Frédéric Tomas; Peter Blomsma; Saar Hommes; Nadine Braun; Emiel van Miltenburg; Chris van der Lee; Martijn Goudbeek; Emiel Krahmer

A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019)

Anouck Braggaar, Frédéric Tomas, Peter Blomsma, Saar Hommes, Nadine Braun, Emiel van Miltenburg, Chris van der Lee, Martijn Goudbeek, Emiel Krahmer

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

In this paper, we describe our reproduction effort of the paper: Towards Best Experiment Design for Evaluating Dialogue System Output by Santhanam and Shaikh (2019) for the 2022 ReproGen shared task. We aim to produce the same results, using different human evaluators, and a different implementation of the automatic metrics used in the original paper. Although overall the study posed some challenges to reproduce (e.g. difficulties with reproduction of automatic metrics and statistics), in the end we did find that the results generally replicate the findings of Santhanam and Shaikh (2019) and seem to follow similar trends.

Original language	English
Title of host publication	Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges
Place of Publication	Waterville, Maine, USA and virtual meeting
Publisher	Association for Computational Linguistics
Pages	86-93
Number of pages	8
ISBN (Print)	978-1-955917-60-5
Publication status	Published - Jul 2022

Keywords

Reproduction study
NLP research
Human evaluation task
Dialogue system output

Access to Document

https://aclanthology.org/2022.inlg-genchal.13

Cite this

Braggaar, A., Tomas, F., Blomsma, P., Hommes, S., Braun, N., van Miltenburg, E., van der Lee, C., Goudbeek, M., & Krahmer, E. (2022). A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019). In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges (pp. 86-93). Association for Computational Linguistics. https://aclanthology.org/2022.inlg-genchal.13

Braggaar, Anouck ; Tomas, Frédéric ; Blomsma, Peter et al. / A reproduction study of methods for evaluating dialogue system output : Replicating Santhanam and Shaikh (2019). Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. Waterville, Maine, USA and virtual meeting : Association for Computational Linguistics, 2022. pp. 86-93

@inproceedings{0a74ace0f2e74d07b6d4ac8bafbaa9df,

title = "A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019)",

abstract = "In this paper, we describe our reproduction effort of the paper: Towards Best Experiment Design for Evaluating Dialogue System Output by Santhanam and Shaikh (2019) for the 2022 ReproGen shared task. We aim to produce the same results, using different human evaluators, and a different implementation of the automatic metrics used in the original paper. Although overall the study posed some challenges to reproduce (e.g. difficulties with reproduction of automatic metrics and statistics), in the end we did find that the results generally replicate the findings of Santhanam and Shaikh (2019) and seem to follow similar trends.",

keywords = "Reproduction study, NLP research, Human evaluation task, Dialogue system output",

author = "Anouck Braggaar and Fr{\'e}d{\'e}ric Tomas and Peter Blomsma and Saar Hommes and Nadine Braun and {van Miltenburg}, Emiel and {van der Lee}, Chris and Martijn Goudbeek and Emiel Krahmer",

year = "2022",

month = jul,

language = "English",

isbn = "978-1-955917-60-5",

pages = "86--93",

booktitle = "Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges",

publisher = "Association for Computational Linguistics",

}

Braggaar, A , Tomas, F, Blomsma, P, Hommes, S , Braun, N , van Miltenburg, E , van der Lee, C, Goudbeek, M & Krahmer, E 2022, A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019). in Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. Association for Computational Linguistics, Waterville, Maine, USA and virtual meeting, pp. 86-93. <https://aclanthology.org/2022.inlg-genchal.13>

A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019). / Braggaar, Anouck ; Tomas, Frédéric; Blomsma, Peter et al.
Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. Waterville, Maine, USA and virtual meeting: Association for Computational Linguistics, 2022. p. 86-93.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - A reproduction study of methods for evaluating dialogue system output

T2 - Replicating Santhanam and Shaikh (2019)

AU - Braggaar, Anouck

AU - Tomas, Frédéric

AU - Blomsma, Peter

AU - Hommes, Saar

AU - Braun, Nadine

AU - van Miltenburg, Emiel

AU - van der Lee, Chris

AU - Goudbeek, Martijn

AU - Krahmer, Emiel

PY - 2022/7

Y1 - 2022/7

N2 - In this paper, we describe our reproduction effort of the paper: Towards Best Experiment Design for Evaluating Dialogue System Output by Santhanam and Shaikh (2019) for the 2022 ReproGen shared task. We aim to produce the same results, using different human evaluators, and a different implementation of the automatic metrics used in the original paper. Although overall the study posed some challenges to reproduce (e.g. difficulties with reproduction of automatic metrics and statistics), in the end we did find that the results generally replicate the findings of Santhanam and Shaikh (2019) and seem to follow similar trends.

AB - In this paper, we describe our reproduction effort of the paper: Towards Best Experiment Design for Evaluating Dialogue System Output by Santhanam and Shaikh (2019) for the 2022 ReproGen shared task. We aim to produce the same results, using different human evaluators, and a different implementation of the automatic metrics used in the original paper. Although overall the study posed some challenges to reproduce (e.g. difficulties with reproduction of automatic metrics and statistics), in the end we did find that the results generally replicate the findings of Santhanam and Shaikh (2019) and seem to follow similar trends.

KW - Reproduction study

KW - NLP research

KW - Human evaluation task

KW - Dialogue system output

M3 - Conference contribution

SN - 978-1-955917-60-5

SP - 86

EP - 93

BT - Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges

PB - Association for Computational Linguistics

CY - Waterville, Maine, USA and virtual meeting

ER -

Braggaar A , Tomas F, Blomsma P, Hommes S , Braun N , van Miltenburg E et al. A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019). In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. Waterville, Maine, USA and virtual meeting: Association for Computational Linguistics. 2022. p. 86-93

A reproduction study of methods for evaluating dialogue system output: Replicating Santhanam and Shaikh (2019)

Abstract

Keywords

Access to Document

Fingerprint

Cite this