Automatic Construction of Evaluation Suites for Natural Language Generation Datasets

Simon Mille; Kaustubh Dhole; Saad Mahamood; Laura Perez-Beltrachini; Varun Prashant Gangal; Mihir Kale; Emiel van Miltenburg; Sebastian Gehrmann

Automatic Construction of Evaluation Suites for Natural Language Generation Datasets

Simon Mille, Kaustubh Dhole, Saad Mahamood, Laura Perez-Beltrachini, Varun Prashant Gangal, Mihir Kale, Emiel van Miltenburg, Sebastian Gehrmann

Language, Communication and Cognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

15 Downloads (Pure)

Abstract

Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets are constructed as an i.i.d. sample from the overall data, this approach overly simplifies the complexity of language and encourages overfitting to the head of the data distribution. As such, rare language phenomena or text about underrepresented groups are not equally included in the evaluation. To encourage more in-depth model analyses, researchers have proposed the use of multiple test sets, also called challenge sets, that assess specific capabilities of a model. In this paper, we develop a framework based on this idea which is able to generate controlled perturbations and identify subsets in text-to-scalar, text-to-text, or data-to-text settings. By applying this framework to the GEM generation benchmark, we develop evaluation suites made of 80 challenge sets, demonstrate the kinds of analyses that it enables, and shed light onto the limits of current generation models.

Original language	English
Title of host publication	Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021)
Publication status	Published - 2021
Event	Conference on Neural Information Processing Systems 2021: Datasets and Benchmarks - Hybrid Duration: 28 Nov 2021 → 9 Dec 2021 Conference number: 35 https://neurips.cc/

Conference

Conference	Conference on Neural Information Processing Systems 2021
Abbreviated title	NeurIPS 2021
Period	28/11/21 → 9/12/21
Internet address	https://neurips.cc/

Access to Document

NeurIPS Datasets and Benchmarks-2021-automatic-construction-of-evaluation-suites-for-natural-language-generation-datasets-Paper-round1Final published version, 659 KB

Cite this

@inproceedings{c19e8a972ed84f198bedb221b12f5ba4,

title = "Automatic Construction of Evaluation Suites for Natural Language Generation Datasets",

abstract = "Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets are constructed as an i.i.d. sample from the overall data, this approach overly simplifies the complexity of language and encourages overfitting to the head of the data distribution. As such, rare language phenomena or text about underrepresented groups are not equally included in the evaluation. To encourage more in-depth model analyses, researchers have proposed the use of multiple test sets, also called challenge sets, that assess specific capabilities of a model. In this paper, we develop a framework based on this idea which is able to generate controlled perturbations and identify subsets in text-to-scalar, text-to-text, or data-to-text settings. By applying this framework to the GEM generation benchmark, we develop evaluation suites made of 80 challenge sets, demonstrate the kinds of analyses that it enables, and shed light onto the limits of current generation models.",

author = "Simon Mille and Kaustubh Dhole and Saad Mahamood and Laura Perez-Beltrachini and Gangal, {Varun Prashant} and Mihir Kale and Miltenburg, {Emiel van} and Sebastian Gehrmann",

year = "2021",

language = "English",

booktitle = "Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021)",

note = "Conference on Neural Information Processing Systems 2021 : Datasets and Benchmarks , NeurIPS 2021 ; Conference date: 28-11-2021 Through 09-12-2021",

url = "https://neurips.cc/",

}

Mille, S, Dhole, K, Mahamood, S, Perez-Beltrachini, L, Gangal, VP, Kale, M, Miltenburg, EV & Gehrmann, S 2021, Automatic Construction of Evaluation Suites for Natural Language Generation Datasets. in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021). Conference on Neural Information Processing Systems 2021, 28/11/21.

Automatic Construction of Evaluation Suites for Natural Language Generation Datasets. / Mille, Simon; Dhole, Kaustubh; Mahamood, Saad et al.
Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021). 2021.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Automatic Construction of Evaluation Suites for Natural Language Generation Datasets

AU - Mille, Simon

AU - Dhole, Kaustubh

AU - Mahamood, Saad

AU - Perez-Beltrachini, Laura

AU - Gangal, Varun Prashant

AU - Kale, Mihir

AU - Miltenburg, Emiel van

AU - Gehrmann, Sebastian

N1 - Conference code: 35

PY - 2021

Y1 - 2021

N2 - Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets are constructed as an i.i.d. sample from the overall data, this approach overly simplifies the complexity of language and encourages overfitting to the head of the data distribution. As such, rare language phenomena or text about underrepresented groups are not equally included in the evaluation. To encourage more in-depth model analyses, researchers have proposed the use of multiple test sets, also called challenge sets, that assess specific capabilities of a model. In this paper, we develop a framework based on this idea which is able to generate controlled perturbations and identify subsets in text-to-scalar, text-to-text, or data-to-text settings. By applying this framework to the GEM generation benchmark, we develop evaluation suites made of 80 challenge sets, demonstrate the kinds of analyses that it enables, and shed light onto the limits of current generation models.

AB - Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets are constructed as an i.i.d. sample from the overall data, this approach overly simplifies the complexity of language and encourages overfitting to the head of the data distribution. As such, rare language phenomena or text about underrepresented groups are not equally included in the evaluation. To encourage more in-depth model analyses, researchers have proposed the use of multiple test sets, also called challenge sets, that assess specific capabilities of a model. In this paper, we develop a framework based on this idea which is able to generate controlled perturbations and identify subsets in text-to-scalar, text-to-text, or data-to-text settings. By applying this framework to the GEM generation benchmark, we develop evaluation suites made of 80 challenge sets, demonstrate the kinds of analyses that it enables, and shed light onto the limits of current generation models.

M3 - Conference contribution

BT - Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021)

T2 - Conference on Neural Information Processing Systems 2021

Y2 - 28 November 2021 through 9 December 2021

ER -

Automatic Construction of Evaluation Suites for Natural Language Generation Datasets

Abstract

Conference

Access to Document

Fingerprint

Cite this