Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent.

Markus Freitag; Nitika Mathur; Chi-kiu Lo; Eleftherios Avramidis; Ricardo Rei; Brian Thompson; Tom Kocmi; Frédéric Blain; Daniel Deutsch; Craig Stewart; Chrysoula Zerva; Sheila Castilho; Alon Lavie; George F. Foster

Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent.

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, George F. Foster

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation
Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year’s success, we also included a challenge
set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering
fewer tasks and calculating a global score by weighted averaging across the various tasks.

Original language	English
Title of host publication	Proceedings of the Eighth Conference on Machine Translation
Pages	578-628
Number of pages	51
Publication status	Published - 2023

Access to Document

https://aclanthology.org/2023.wmt-1.51Licence: CC BY

Cite this

Freitag, M., Mathur, N., Lo, C., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., Stewart, C., Zerva, C., Castilho, S., Lavie, A., & Foster, G. F. (2023). Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent. In Proceedings of the Eighth Conference on Machine Translation (pp. 578-628) https://aclanthology.org/2023.wmt-1.51

@inproceedings{5065ce796d334448bfaff820cfead214,

title = "Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent.",

abstract = "This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year{\textquoteright}s success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics{\textquoteright} ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks. ",

author = "Markus Freitag and Nitika Mathur and Chi-kiu Lo and Eleftherios Avramidis and Ricardo Rei and Brian Thompson and Tom Kocmi and Fr{\'e}d{\'e}ric Blain and Daniel Deutsch and Craig Stewart and Chrysoula Zerva and Sheila Castilho and Alon Lavie and Foster, \{George F.\}",

note = "DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.",

year = "2023",

language = "English",

pages = "578--628",

booktitle = "Proceedings of the Eighth Conference on Machine Translation",

}

Freitag, M, Mathur, N, Lo, C, Avramidis, E, Rei, R, Thompson, B, Kocmi, T, Blain, F, Deutsch, D, Stewart, C, Zerva, C, Castilho, S, Lavie, A & Foster, GF 2023, Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent. in Proceedings of the Eighth Conference on Machine Translation. pp. 578-628. <https://aclanthology.org/2023.wmt-1.51>

TY - GEN

T1 - Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent.

AU - Freitag, Markus

AU - Mathur, Nitika

AU - Lo, Chi-kiu

AU - Avramidis, Eleftherios

AU - Rei, Ricardo

AU - Thompson, Brian

AU - Kocmi, Tom

AU - Blain, Frédéric

AU - Deutsch, Daniel

AU - Stewart, Craig

AU - Zerva, Chrysoula

AU - Castilho, Sheila

AU - Lavie, Alon

AU - Foster, George F.

N1 - DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.

PY - 2023

Y1 - 2023

N2 - This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year’s success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks.

AB - This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year’s success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks.

M3 - Conference contribution

SP - 578

EP - 628

BT - Proceedings of the Eighth Conference on Machine Translation

ER -

Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent.

Abstract

Access to Document

Fingerprint

Cite this