Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation

Emiel van Miltenburg; Chris van der Lee; Thiago Castro-Ferreira; Emiel Krahmer

Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation

Emiel van Miltenburg, Chris van der Lee, Thiago Castro-Ferreira, Emiel Krahmer

Language, Communication and Cognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

NLG researchers often use uncontrolled corpora to train and evaluate their systems, using textual similarity metrics, such as BLEU. This position paper argues in favour of two alternative evaluation strategies, using grammars or rule-based systems. These strategies are particularly useful to identify the strengths and weaknesses of different systems. We contrast our proposals with the (extended) WebNLG dataset, which is revealed to have a skewed distribution of predicates. We predict that this distribution affects the quality of the predictions for systems trained on this data. However, this hypothesis can only be thoroughly tested (without any confounds) once we are able to systematically manipulate the skewness of the data, using a rule-based approach.

Original language	English
Title of host publication	Proceedings of the 1st Workshop on Evaluating NLG Evaluation
Place of Publication	Dublin, Ireland
Publisher	Association for Computational Linguistics
Pages	17-27
Number of pages	11
Publication status	Published - 1 Dec 2020
Event	Workshop on Evaluating NLG Evaluation - Online, Dublin, Ireland Duration: 18 Dec 2020 → … https://evalnlg-workshop.github.io/

Conference

Conference	Workshop on Evaluating NLG Evaluation
Country/Territory	Ireland
City	Dublin
Period	18/12/20 → …
Internet address	https://evalnlg-workshop.github.io/

Access to Document

https://www.aclweb.org/anthology/2020.evalnlgeval-1.3

Cite this

@inproceedings{ea98daa9081d4a58bbd0ba12912ec9aa,

title = "Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation",

abstract = "NLG researchers often use uncontrolled corpora to train and evaluate their systems, using textual similarity metrics, such as BLEU. This position paper argues in favour of two alternative evaluation strategies, using grammars or rule-based systems. These strategies are particularly useful to identify the strengths and weaknesses of different systems. We contrast our proposals with the (extended) WebNLG dataset, which is revealed to have a skewed distribution of predicates. We predict that this distribution affects the quality of the predictions for systems trained on this data. However, this hypothesis can only be thoroughly tested (without any confounds) once we are able to systematically manipulate the skewness of the data, using a rule-based approach.",

author = "{van Miltenburg}, Emiel and {van der Lee}, Chris and Thiago Castro-Ferreira and Emiel Krahmer",

year = "2020",

month = dec,

day = "1",

language = "English",

pages = "17--27",

booktitle = "Proceedings of the 1st Workshop on Evaluating NLG Evaluation",

publisher = "Association for Computational Linguistics",

note = "Workshop on Evaluating NLG Evaluation ; Conference date: 18-12-2020",

url = "https://evalnlg-workshop.github.io/",

}

van Miltenburg, E , van der Lee, C, Castro-Ferreira, T & Krahmer, E 2020, Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation. in Proceedings of the 1st Workshop on Evaluating NLG Evaluation. Association for Computational Linguistics, Dublin, Ireland, pp. 17-27, Workshop on Evaluating NLG Evaluation, Dublin, Ireland, 18/12/20. <https://www.aclweb.org/anthology/2020.evalnlgeval-1.3>

Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation. / van Miltenburg, Emiel ; van der Lee, Chris; Castro-Ferreira, Thiago; Krahmer, Emiel.

Proceedings of the 1st Workshop on Evaluating NLG Evaluation. Dublin, Ireland : Association for Computational Linguistics, 2020. p. 17-27.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation

AU - van Miltenburg, Emiel

AU - van der Lee, Chris

AU - Castro-Ferreira, Thiago

AU - Krahmer, Emiel

PY - 2020/12/1

Y1 - 2020/12/1

N2 - NLG researchers often use uncontrolled corpora to train and evaluate their systems, using textual similarity metrics, such as BLEU. This position paper argues in favour of two alternative evaluation strategies, using grammars or rule-based systems. These strategies are particularly useful to identify the strengths and weaknesses of different systems. We contrast our proposals with the (extended) WebNLG dataset, which is revealed to have a skewed distribution of predicates. We predict that this distribution affects the quality of the predictions for systems trained on this data. However, this hypothesis can only be thoroughly tested (without any confounds) once we are able to systematically manipulate the skewness of the data, using a rule-based approach.

AB - NLG researchers often use uncontrolled corpora to train and evaluate their systems, using textual similarity metrics, such as BLEU. This position paper argues in favour of two alternative evaluation strategies, using grammars or rule-based systems. These strategies are particularly useful to identify the strengths and weaknesses of different systems. We contrast our proposals with the (extended) WebNLG dataset, which is revealed to have a skewed distribution of predicates. We predict that this distribution affects the quality of the predictions for systems trained on this data. However, this hypothesis can only be thoroughly tested (without any confounds) once we are able to systematically manipulate the skewness of the data, using a rule-based approach.

M3 - Conference contribution

SP - 17

EP - 27

BT - Proceedings of the 1st Workshop on Evaluating NLG Evaluation

PB - Association for Computational Linguistics

CY - Dublin, Ireland

T2 - Workshop on Evaluating NLG Evaluation

Y2 - 18 December 2020

ER -

Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation

Abstract

Conference

Access to Document

Fingerprint

Cite this