Improve Factuality via Implicit Self-consistency

View a PDF of the paper titled Integrative Decoding: Enhance Factuality through Implicit Self-consistency, by Yi Cheng and 13 different authors

View PDF
HTML (experimental)

Summary:Self-consistency-based approaches, which contain repeatedly sampling a number of outputs and deciding on essentially the most constant one as the ultimate response, show to be remarkably efficient in bettering the factual accuracy of huge language fashions. Nonetheless, current strategies often have strict constraints on the duty format, largely limiting their applicability. On this paper, we current Integrative Decoding (ID), to unlock the potential of self-consistency in open-ended technology duties. ID operates by setting up a set of inputs, every prepended with a beforehand sampled response, after which processes them concurrently, with the subsequent token being chosen by aggregating of all their corresponding predictions at every decoding step. In essence, this straightforward method implicitly incorporates self-consistency within the decoding goal. Intensive analysis reveals that ID persistently enhances factuality over a variety of language fashions, with substantial enhancements on the TruthfulQA (+11.2%), Biographies (+15.4%) and LongFact (+8.5%) benchmarks. The efficiency features amplify progressively because the variety of sampled responses will increase, indicating the potential of ID to scale up with repeated sampling.