[2311.09335] Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization

View a PDF of the paper titled Investigating Hallucinations in Pruned Giant Language Fashions for Abstractive Summarization, by George Chrysostomou and three different authors

View PDF
HTML (experimental)

Summary:Regardless of the exceptional efficiency of generative massive language fashions (LLMs) on abstractive summarization, they face two vital challenges: their appreciable measurement and tendency to hallucinate. Hallucinations are regarding as a result of they erode reliability and lift questions of safety. Pruning is a method that reduces mannequin measurement by eradicating redundant weights, enabling extra environment friendly sparse inference. Pruned fashions yield downstream activity efficiency akin to the unique, making them supreme options when working on a restricted finances. Nevertheless, the impact that pruning has upon hallucinations in abstractive summarization with LLMs has but to be explored. On this paper, we offer an in depth empirical research throughout 5 summarization datasets, two state-of-the-art pruning strategies, and 5 instruction-tuned LLMs. Surprisingly, we discover that hallucinations are much less prevalent from pruned LLMs than the unique fashions. Our evaluation means that pruned fashions are likely to rely extra on the supply doc for abstract era. This results in the next lexical overlap between the generated abstract and the supply doc, which could possibly be a purpose for the discount in hallucination danger.