[2405.18348] Can Automatic Metrics Assess High-Quality Translations?

View a PDF of the paper titled Can Automated Metrics Assess Excessive-High quality Translations?, by Sweta Agrawal and three different authors

View PDF
HTML (experimental)

Summary:Automated metrics for evaluating translation high quality are sometimes validated by measuring how nicely they correlate with human assessments. Nonetheless, correlation strategies are inclined to seize solely the flexibility of metrics to distinguish between good and unhealthy source-translation pairs, overlooking their reliability in distinguishing various translations for a similar supply. On this paper, we verify that that is certainly the case by displaying that present metrics are insensitive to nuanced variations in translation high quality. This impact is most pronounced when the standard is excessive and the variance amongst options is low. Given this discovering, we shift in direction of detecting high-quality right translations, an necessary drawback in sensible decision-making eventualities the place a binary examine of correctness is prioritized over a nuanced analysis of high quality. Utilizing the MQM framework because the gold customary, we systematically stress-test the flexibility of present metrics to determine translations with no errors as marked by people. Our findings reveal that present metrics usually over or underestimate translation high quality, indicating vital room for enchancment in automated analysis strategies.