2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability

View a PDF of the paper titled 3-in-1: 2D Rotary Adaptation for Environment friendly Finetuning, Environment friendly Batching and Composability, by Baohao Liao and Christof Monz

View PDF
HTML (experimental)

Summary:Parameter-efficient finetuning (PEFT) strategies successfully adapt massive language fashions (LLMs) to numerous downstream duties, decreasing storage and GPU reminiscence calls for. Regardless of these benefits, a number of functions pose new challenges to PEFT past mere parameter effectivity. One notable problem includes the environment friendly deployment of LLMs outfitted with a number of task- or user-specific adapters, notably when completely different adapters are wanted for distinct requests throughout the identical batch. One other problem is the interpretability of LLMs, which is essential for understanding how LLMs operate. Earlier research launched varied approaches to deal with completely different challenges. On this paper, we introduce a novel methodology, RoAd, which employs a simple 2D rotation to adapt LLMs and addresses all of the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimum efficiency on GLUE, eight commonsense reasoning duties and 4 arithmetic reasoning duties with $<0.1%$ trainable parameters; (2) RoAd facilitates the environment friendly serving of requests requiring completely different adapters inside a batch, with an overhead similar to element-wise multiplication as an alternative of batch matrix multiplication; (3) RoAd enhances LLM’s interpretability by means of integration inside a framework of distributed interchange intervention, demonstrated through composition experiments.