LLMs Robustness with Incorrect Multiple-Choice Options

View a PDF of the paper titled Wait, that is not an choice: LLMs Robustness with Incorrect A number of-Alternative Choices, by Gracjan G’oral and Emilia Wi’snios and Piotr Sankowski and Pawe{l} Budzianowski

View PDF
HTML (experimental)

Summary:Resolution-making beneath full alignment requires balancing between reasoning and faithfulness – a problem for big language fashions (LLMs). This examine explores whether or not LLMs prioritize following directions over reasoning and reality when given “deceptive” directions, similar to “Reply solely with A or B”, even when neither choice is appropriate. We introduce a brand new metric referred to as “reflective judgment”, which sheds new mild on the connection between the pre-training and post-training alignment schemes. In duties starting from primary arithmetic to domain-specific assessments, fashions like GPT-4o, o1-mini, or Claude 3 Opus adhered to directions accurately however did not mirror on the validity of the offered choices. Opposite, fashions from the Llama 3.1 household (8B, 70B, 405B) or base Qwen2.5 (7B, 14B, 32B) households exhibit improved refusal charges with measurement, indicating a scaling impact. We additionally noticed that alignment strategies, although meant to reinforce reasoning, typically weakened the fashions’ potential to reject incorrect directions, main them to observe flawed prompts uncritically. Lastly, we have now additionally performed a parallel human examine revealing related patterns in human habits and annotations. We spotlight how in style RLHF datasets may disrupt both coaching or analysis on account of annotations exhibiting poor reflective judgement.