The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Sebastian Gehrmann; Tosin Adewumi; Karmanya Aggarwal; Pawan Sasanka Ammanamanchi; Anuoluwapo Aremu; Antoine Bosselut; Khyathi Raghavi Chandu; Miruna-Adriana Clinciu; Dipanjan Das; Kaustubh Dhole; Wanyu Du; Esin Durmus; Ondřej Dušek; Chris Chinenye Emezue; Varun Gangal; Cristina Garbacea; Tatsunori Hashimoto; Yufang Hou; Yacine Jernite; Harsh Jhamtani; Yangfeng Ji; Shailza Jolly; Mihir Kale; Dhruv Kumar; Faisal Ladhak; Aman Madaan; Mounica Maddela; Khyati Mahajan; Saad Mahamood; Bodhisattwa Prasad Majumder; Pedro Henrique Martins; Angelina McMillan-Major; Simon Mille; Emiel van Miltenburg; Moin Nadeem; Shashi Narayan; Vitaly Nikolaev; Andre Niyongabo Rubungo; Salomey Osei; Ankur Parikh; Laura Perez-Beltrachini; Niranjan Ramesh Rao; Vikas Raunak; Juan Diego Rodriguez; Sashank Santhanam; João Sedoc; Thibault Sellam; Samira Shaikh; Anastasia Shimorina; Marco Antonio Sobrevilla Cabezudo; Hendrik Strobelt; Nishant Subramani; Wei Xu; Diyi Yang; Akhila Yerukola; Jiawei Zhou

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh JhamtaniYangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, Jiawei Zhou

Language, Communication and Cognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

Original language	English
Title of host publication	Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
Place of Publication	Online
Publisher	Association for Computational Linguistics
Pages	96-120
Number of pages	25
Publication status	Published - 1 Aug 2021
Event	Workshop on Natural Language Generation, Evaluation, and Metrics - Berkeley Hotel, Thailand Duration: 5 Aug 2021 → 6 Aug 2021 Conference number: 1 https://www.aclweb.org/portal/content/first-workshop-generation-evaluation-and-metrics-acl-2021

Conference

Conference	Workshop on Natural Language Generation, Evaluation, and Metrics
Abbreviated title	GEM2021
Country/Territory	Thailand
Period	5/08/21 → 6/08/21
Internet address	https://www.aclweb.org/portal/content/first-workshop-generation-evaluation-and-metrics-acl-2021

Access to Document

https://aclanthology.org/2021.gem-1.10

Cite this

Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M-A., Das, D., Dhole, K., Du, W., Durmus, E., Dušek, O., Emezue, C. C., Gangal, V., Garbacea, C., Hashimoto, T., Hou, Y., Jernite, Y., ... Zhou, J. (2021). The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) (pp. 96-120). Association for Computational Linguistics. https://aclanthology.org/2021.gem-1.10

Gehrmann, Sebastian ; Adewumi, Tosin ; Aggarwal, Karmanya ; Ammanamanchi, Pawan Sasanka ; Aremu, Anuoluwapo ; Bosselut, Antoine ; Chandu, Khyathi Raghavi ; Clinciu, Miruna-Adriana ; Das, Dipanjan ; Dhole, Kaustubh ; Du, Wanyu ; Durmus, Esin ; Dušek, Ondřej ; Emezue, Chris Chinenye ; Gangal, Varun ; Garbacea, Cristina ; Hashimoto, Tatsunori ; Hou, Yufang ; Jernite, Yacine ; Jhamtani, Harsh ; Ji, Yangfeng ; Jolly, Shailza ; Kale, Mihir ; Kumar, Dhruv ; Ladhak, Faisal ; Madaan, Aman ; Maddela, Mounica ; Mahajan, Khyati ; Mahamood, Saad ; Majumder, Bodhisattwa Prasad ; Martins, Pedro Henrique ; McMillan-Major, Angelina ; Mille, Simon ; van Miltenburg, Emiel ; Nadeem, Moin ; Narayan, Shashi ; Nikolaev, Vitaly ; Niyongabo Rubungo, Andre ; Osei, Salomey ; Parikh, Ankur ; Perez-Beltrachini, Laura ; Rao, Niranjan Ramesh ; Raunak, Vikas ; Rodriguez, Juan Diego ; Santhanam, Sashank ; Sedoc, João ; Sellam, Thibault ; Shaikh, Samira ; Shimorina, Anastasia ; Sobrevilla Cabezudo, Marco Antonio ; Strobelt, Hendrik ; Subramani, Nishant ; Xu, Wei ; Yang, Diyi ; Yerukola, Akhila ; Zhou, Jiawei. / The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021). Online : Association for Computational Linguistics, 2021. pp. 96-120

@inproceedings{c42a48c622ca4ea49e886a43cdf0b3f9,

title = "The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics",

abstract = "We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.",

author = "Sebastian Gehrmann and Tosin Adewumi and Karmanya Aggarwal and Ammanamanchi, {Pawan Sasanka} and Anuoluwapo Aremu and Antoine Bosselut and Chandu, {Khyathi Raghavi} and Miruna-Adriana Clinciu and Dipanjan Das and Kaustubh Dhole and Wanyu Du and Esin Durmus and Ond{\v r}ej Du{\v s}ek and Emezue, {Chris Chinenye} and Varun Gangal and Cristina Garbacea and Tatsunori Hashimoto and Yufang Hou and Yacine Jernite and Harsh Jhamtani and Yangfeng Ji and Shailza Jolly and Mihir Kale and Dhruv Kumar and Faisal Ladhak and Aman Madaan and Mounica Maddela and Khyati Mahajan and Saad Mahamood and Majumder, {Bodhisattwa Prasad} and Martins, {Pedro Henrique} and Angelina McMillan-Major and Simon Mille and {van Miltenburg}, Emiel and Moin Nadeem and Shashi Narayan and Vitaly Nikolaev and {Niyongabo Rubungo}, Andre and Salomey Osei and Ankur Parikh and Laura Perez-Beltrachini and Rao, {Niranjan Ramesh} and Vikas Raunak and Rodriguez, {Juan Diego} and Sashank Santhanam and Jo{\~a}o Sedoc and Thibault Sellam and Samira Shaikh and Anastasia Shimorina and {Sobrevilla Cabezudo}, {Marco Antonio} and Hendrik Strobelt and Nishant Subramani and Wei Xu and Diyi Yang and Akhila Yerukola and Jiawei Zhou",

year = "2021",

month = aug,

day = "1",

language = "English",

pages = "96--120",

booktitle = "Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)",

publisher = "Association for Computational Linguistics",

note = "Workshop on Natural Language Generation, Evaluation, and Metrics , GEM2021 ; Conference date: 05-08-2021 Through 06-08-2021",

url = "https://www.aclweb.org/portal/content/first-workshop-generation-evaluation-and-metrics-acl-2021",

}

Gehrmann, S, Adewumi, T, Aggarwal, K, Ammanamanchi, PS, Aremu, A, Bosselut, A, Chandu, KR, Clinciu, M-A, Das, D, Dhole, K, Du, W, Durmus, E, Dušek, O, Emezue, CC, Gangal, V, Garbacea, C, Hashimoto, T, Hou, Y, Jernite, Y, Jhamtani, H, Ji, Y, Jolly, S, Kale, M, Kumar, D, Ladhak, F, Madaan, A, Maddela, M, Mahajan, K, Mahamood, S, Majumder, BP, Martins, PH, McMillan-Major, A, Mille, S, van Miltenburg, E, Nadeem, M, Narayan, S, Nikolaev, V, Niyongabo Rubungo, A, Osei, S, Parikh, A, Perez-Beltrachini, L, Rao, NR, Raunak, V, Rodriguez, JD, Santhanam, S, Sedoc, J, Sellam, T, Shaikh, S, Shimorina, A, Sobrevilla Cabezudo, MA, Strobelt, H, Subramani, N, Xu, W, Yang, D, Yerukola, A & Zhou, J 2021, The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. in Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021). Association for Computational Linguistics, Online, pp. 96-120, Workshop on Natural Language Generation, Evaluation, and Metrics , Thailand, 5/08/21. <https://aclanthology.org/2021.gem-1.10>

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. / Gehrmann, Sebastian; Adewumi, Tosin; Aggarwal, Karmanya; Ammanamanchi, Pawan Sasanka; Aremu, Anuoluwapo; Bosselut, Antoine; Chandu, Khyathi Raghavi; Clinciu, Miruna-Adriana; Das, Dipanjan; Dhole, Kaustubh; Du, Wanyu; Durmus, Esin; Dušek, Ondřej; Emezue, Chris Chinenye; Gangal, Varun; Garbacea, Cristina; Hashimoto, Tatsunori; Hou, Yufang; Jernite, Yacine; Jhamtani, Harsh; Ji, Yangfeng; Jolly, Shailza; Kale, Mihir; Kumar, Dhruv; Ladhak, Faisal; Madaan, Aman; Maddela, Mounica; Mahajan, Khyati; Mahamood, Saad; Majumder, Bodhisattwa Prasad; Martins, Pedro Henrique; McMillan-Major, Angelina; Mille, Simon; van Miltenburg, Emiel; Nadeem, Moin; Narayan, Shashi; Nikolaev, Vitaly; Niyongabo Rubungo, Andre; Osei, Salomey; Parikh, Ankur; Perez-Beltrachini, Laura; Rao, Niranjan Ramesh; Raunak, Vikas; Rodriguez, Juan Diego; Santhanam, Sashank; Sedoc, João; Sellam, Thibault; Shaikh, Samira; Shimorina, Anastasia; Sobrevilla Cabezudo, Marco Antonio; Strobelt, Hendrik; Subramani, Nishant; Xu, Wei; Yang, Diyi; Yerukola, Akhila; Zhou, Jiawei.

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021). Online : Association for Computational Linguistics, 2021. p. 96-120.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

AU - Gehrmann, Sebastian

AU - Adewumi, Tosin

AU - Aggarwal, Karmanya

AU - Ammanamanchi, Pawan Sasanka

AU - Aremu, Anuoluwapo

AU - Bosselut, Antoine

AU - Chandu, Khyathi Raghavi

AU - Clinciu, Miruna-Adriana

AU - Das, Dipanjan

AU - Dhole, Kaustubh

AU - Du, Wanyu

AU - Durmus, Esin

AU - Dušek, Ondřej

AU - Emezue, Chris Chinenye

AU - Gangal, Varun

AU - Garbacea, Cristina

AU - Hashimoto, Tatsunori

AU - Hou, Yufang

AU - Jernite, Yacine

AU - Jhamtani, Harsh

AU - Ji, Yangfeng

AU - Jolly, Shailza

AU - Kale, Mihir

AU - Kumar, Dhruv

AU - Ladhak, Faisal

AU - Madaan, Aman

AU - Maddela, Mounica

AU - Mahajan, Khyati

AU - Mahamood, Saad

AU - Majumder, Bodhisattwa Prasad

AU - Martins, Pedro Henrique

AU - McMillan-Major, Angelina

AU - Mille, Simon

AU - van Miltenburg, Emiel

AU - Nadeem, Moin

AU - Narayan, Shashi

AU - Nikolaev, Vitaly

AU - Niyongabo Rubungo, Andre

AU - Osei, Salomey

AU - Parikh, Ankur

AU - Perez-Beltrachini, Laura

AU - Rao, Niranjan Ramesh

AU - Raunak, Vikas

AU - Rodriguez, Juan Diego

AU - Santhanam, Sashank

AU - Sedoc, João

AU - Sellam, Thibault

AU - Shaikh, Samira

AU - Shimorina, Anastasia

AU - Sobrevilla Cabezudo, Marco Antonio

AU - Strobelt, Hendrik

AU - Subramani, Nishant

AU - Xu, Wei

AU - Yang, Diyi

AU - Yerukola, Akhila

AU - Zhou, Jiawei

N1 - Conference code: 1

PY - 2021/8/1

Y1 - 2021/8/1

N2 - We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

AB - We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

M3 - Conference contribution

SP - 96

EP - 120

BT - Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

PB - Association for Computational Linguistics

CY - Online

T2 - Workshop on Natural Language Generation, Evaluation, and Metrics

Y2 - 5 August 2021 through 6 August 2021

ER -

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Abstract

Conference

Access to Document

Fingerprint

Cite this