Enabling Early Exit Inference and Self-Speculative Decoding

View a PDF of the paper titled LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding, by Mostafa Elhoushi and 12 different authors

View PDF
HTML (experimental)

Summary:We current LayerSkip, an end-to-end resolution to speed-up inference of enormous language fashions (LLMs). First, throughout coaching we apply layer dropout, with low dropout charges for earlier layers and better dropout charges for later layers, and an early exit loss the place all transformer layers share the identical exit. Second, throughout inference, we present that this coaching recipe will increase the accuracy of early exit at earlier layers, with out including any auxiliary layers or modules to the mannequin. Third, we current a novel self-speculative decoding resolution the place we exit at early layers and confirm and proper with remaining layers of the mannequin. Our proposed self-speculative decoding method has much less reminiscence footprint than different speculative decoding approaches and advantages from shared compute and activations of the draft and verification phases. We run experiments on totally different Llama mannequin sizes on several types of coaching: pretraining from scratch, continuous pretraining, finetuning on particular knowledge area, and finetuning on particular activity. We implement our inference resolution and present speedups of as much as 2.16x on summarization for CNN/DM paperwork, 1.82x on coding, and a pair of.0x on TOPv2 semantic parsing activity. We open supply our code and checkpoints at this https URL.