Findings of the WMT 2023 Shared Task on Quality Estimation.

Frédéric Blain; Chrysoula Zerva; Ricardo Ribeiro; Nuno Miguel Guerreiro; Diptesh Kanojia; José G. C. de Souza; Beatriz Silva; Tânia Vaz; Yan Jingxuan; Fatemeh Azadi; Constantin Orasan; André Martins

doi:10.18653/v1/2023.wmt-1.52

Findings of the WMT 2023 Shared Task on Quality Estimation.

Frédéric Blain
, Chrysoula Zerva
, Ricardo Ribeiro
, Nuno Miguel Guerreiro
, Diptesh Kanojia
, José G. C. de Souza
, Beatriz Silva
, Tânia Vaz
, Yan Jingxuan
, Fatemeh Azadi
, Constantin Orasan
, André Martins

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

150 Downloads (Pure)

Abstract

We report the results of the WMT 2023 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to
reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the provided data to new language pairs: we specifically target low-resource languages and provide training, development and test data for English-Hindi, English-Tamil, English-Telegu and English-Gujarati as well as a zero-shot test set for English-Farsi. Further, we introduce a novel fine-grained error prediction task aspiring to motivate research towards more detailed quality predictions

Original language	English
Title of host publication	Proceedings of the Eighth Conference on Machine Translation
Editors	Philipp Koehn, Barry Haddow, Tom Kocmi, Christof Monz
Publisher	Association for Computational Linguistics
Pages	629-653
Number of pages	25
DOIs	https://doi.org/10.18653/v1/2023.wmt-1.52
Publication status	Published - 2023
Event	Eight conference on machine translation - , Singapore Duration: 6 Dec 2023 → 7 Dec 2023

Conference

Conference	Eight conference on machine translation
Abbreviated title	WMT23
Country/Territory	Singapore
Period	6/12/23 → 7/12/23

Access to Document

10.18653/v1/2023.wmt-1.52Licence: CC BY

2023.wmt-1.52Licence: Unspecified

Cite this

Blain, F., Zerva, C., Ribeiro, R., Guerreiro, N. M., Kanojia, D., Souza, J. G. C. D., Silva, B., Vaz, T., Jingxuan, Y., Azadi, F., Orasan, C., & Martins, A. (2023). Findings of the WMT 2023 Shared Task on Quality Estimation. In P. Koehn, B. Haddow, T. Kocmi, & C. Monz (Eds.), Proceedings of the Eighth Conference on Machine Translation (pp. 629-653). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.52

@inproceedings{e4e631e57ea64302b5251b0b362e4753,

title = "Findings of the WMT 2023 Shared Task on Quality Estimation.",

abstract = "We report the results of the WMT 2023 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the provided data to new language pairs: we specifically target low-resource languages and provide training, development and test data for English-Hindi, English-Tamil, English-Telegu and English-Gujarati as well as a zero-shot test set for English-Farsi. Further, we introduce a novel fine-grained error prediction task aspiring to motivate research towards more detailed quality predictions",

author = "Fr{\'e}d{\'e}ric Blain and Chrysoula Zerva and Ricardo Ribeiro and Guerreiro, \{Nuno Miguel\} and Diptesh Kanojia and Souza, \{Jos{\'e} G. C. de\} and Beatriz Silva and T{\^a}nia Vaz and Yan Jingxuan and Fatemeh Azadi and Constantin Orasan and Andr{\'e} Martins",

note = "DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.; Eight conference on machine translation, WMT23 ; Conference date: 06-12-2023 Through 07-12-2023",

year = "2023",

doi = "10.18653/v1/2023.wmt-1.52",

language = "English",

pages = "629--653",

editor = "Koehn, \{Philipp \} and Barry Haddow and Tom Kocmi and Christof Monz",

booktitle = "Proceedings of the Eighth Conference on Machine Translation",

publisher = "Association for Computational Linguistics",

}

Blain, F, Zerva, C, Ribeiro, R, Guerreiro, NM, Kanojia, D, Souza, JGCD, Silva, B, Vaz, T, Jingxuan, Y, Azadi, F, Orasan, C & Martins, A 2023, Findings of the WMT 2023 Shared Task on Quality Estimation. in P Koehn, B Haddow, T Kocmi & C Monz (eds), Proceedings of the Eighth Conference on Machine Translation. Association for Computational Linguistics, pp. 629-653, Eight conference on machine translation, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.wmt-1.52

Findings of the WMT 2023 Shared Task on Quality Estimation. / Blain, Frédéric; Zerva, Chrysoula; Ribeiro, Ricardo et al.
Proceedings of the Eighth Conference on Machine Translation. ed. / Philipp Koehn; Barry Haddow; Tom Kocmi; Christof Monz. Association for Computational Linguistics, 2023. p. 629-653.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Findings of the WMT 2023 Shared Task on Quality Estimation.

AU - Blain, Frédéric

AU - Zerva, Chrysoula

AU - Ribeiro, Ricardo

AU - Guerreiro, Nuno Miguel

AU - Kanojia, Diptesh

AU - Souza, José G. C. de

AU - Silva, Beatriz

AU - Vaz, Tânia

AU - Jingxuan, Yan

AU - Azadi, Fatemeh

AU - Orasan, Constantin

AU - Martins, André

N1 - DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.

PY - 2023

Y1 - 2023

N2 - We report the results of the WMT 2023 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the provided data to new language pairs: we specifically target low-resource languages and provide training, development and test data for English-Hindi, English-Tamil, English-Telegu and English-Gujarati as well as a zero-shot test set for English-Farsi. Further, we introduce a novel fine-grained error prediction task aspiring to motivate research towards more detailed quality predictions

AB - We report the results of the WMT 2023 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the provided data to new language pairs: we specifically target low-resource languages and provide training, development and test data for English-Hindi, English-Tamil, English-Telegu and English-Gujarati as well as a zero-shot test set for English-Farsi. Further, we introduce a novel fine-grained error prediction task aspiring to motivate research towards more detailed quality predictions

U2 - 10.18653/v1/2023.wmt-1.52

DO - 10.18653/v1/2023.wmt-1.52

M3 - Conference contribution

SP - 629

EP - 653

BT - Proceedings of the Eighth Conference on Machine Translation

A2 - Koehn, Philipp

A2 - Haddow, Barry

A2 - Kocmi, Tom

A2 - Monz, Christof

PB - Association for Computational Linguistics

T2 - Eight conference on machine translation

Y2 - 6 December 2023 through 7 December 2023

ER -

Findings of the WMT 2023 Shared Task on Quality Estimation.

Abstract

Conference

Access to Document

Fingerprint

Cite this