[2406.11109] Investigating Annotator Bias in Large Language Models for Hate Speech Detection

View a PDF of the paper titled Investigating Annotator Bias in Giant Language Fashions for Hate Speech Detection, by Amit Das and 9 different authors

View PDF
HTML (experimental)

Summary:Information annotation, the follow of assigning descriptive labels to uncooked information, is pivotal in optimizing the efficiency of machine studying fashions. Nonetheless, it’s a resource-intensive course of vulnerable to biases launched by annotators. The emergence of refined Giant Language Fashions (LLMs), like ChatGPT presents a novel alternative to modernize and streamline this advanced process. Whereas present analysis extensively evaluates the efficacy of LLMs, as annotators, this paper delves into the biases current in LLMs, particularly GPT 3.5 and GPT 4o when annotating hate speech information. Our analysis contributes to understanding biases in 4 key classes: gender, race, faith, and incapacity. Particularly concentrating on extremely weak teams inside these classes, we analyze annotator biases. Moreover, we conduct a complete examination of potential components contributing to those biases by scrutinizing the annotated information. We introduce our customized hate speech detection dataset, HateSpeechCorpus, to conduct this analysis. Moreover, we carry out the identical experiments on the ETHOS (Mollas et al., 2022) dataset additionally for comparative evaluation. This paper serves as a vital useful resource, guiding researchers and practitioners in harnessing the potential of LLMs for dataannotation, thereby fostering developments on this crucial subject. The HateSpeechCorpus dataset is out there right here: this https URL