Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense

View a PDF of the paper titled Thank You, Stingray: Multilingual Giant Language Fashions Can Not (But) Disambiguate Cross-Lingual Phrase Sense, by Samuel Cahyawijaya and Ruochen Zhang and Holy Lovenia and Jan Christian Blaise Cruz and Elisa Gilbert and Hiroki Nomoto and Alham Fikri Aji

View PDF
HTML (experimental)

Summary:Multilingual giant language fashions (LLMs) have gained prominence, however issues come up relating to their reliability past English. This examine addresses the hole in cross-lingual semantic analysis by introducing a novel benchmark for cross-lingual sense disambiguation, StingrayBench. On this paper, we show utilizing false associates — phrases which might be orthographically comparable however have fully totally different meanings in two languages — as a attainable strategy to pinpoint the limitation of cross-lingual sense disambiguation in LLMs. We accumulate false associates in 4 language pairs, particularly Indonesian-Malay, Indonesian-Tagalog, Chinese language-Japanese, and English-German; and problem LLMs to differentiate using them in context. In our evaluation of varied fashions, we observe they are typically biased towards higher-resource languages. We additionally suggest new metrics for quantifying the cross-lingual sense bias and comprehension primarily based on our benchmark. Our work contributes to growing extra numerous and inclusive language modeling, selling fairer entry for the broader multilingual group.