Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model

View a PDF of the paper titled Quest: Question-centric Knowledge Synthesis Method for Lengthy-context Scaling of Giant Language Mannequin, by Chaochen Gao and three different authors

View PDF
HTML (experimental)

Summary:Latest developments in giant language fashions (LLMs) have highlighted the significance of extending context lengths for dealing with complicated duties. Whereas conventional strategies for coaching on lengthy contexts typically use filtered lengthy paperwork, these approaches result in area imbalances, limiting mannequin efficiency. To handle this, strategies like random doc concatenation (Normal) and similarity-based strategies (KNN, ICLM) have been developed. Nevertheless, they both sacrifice semantic coherence or variety. To steadiness each points, we introduce Quest, a query-centric knowledge synthesis methodology aggregating semantically related but numerous paperwork. Quest makes use of a generative mannequin to foretell potential queries for every doc, grouping paperwork with comparable queries and key phrases. In depth experiments reveal Quest’s superior efficiency on long-context duties, attaining outstanding outcomes with context lengths of as much as 1M tokens and confirming its scalability throughout numerous mannequin sizes.