Benchmarking LLM Proficiency in Scientific Literature Analysis

View a PDF of the paper titled SciAssess: Benchmarking LLM Proficiency in Scientific Literature Evaluation, by Hengxing Cai and 22 different authors

View PDF
HTML (experimental)

Summary:Latest breakthroughs in Giant Language Fashions (LLMs) have revolutionized scientific literature evaluation. Nevertheless, current benchmarks fail to adequately consider the proficiency of LLMs on this area, significantly in eventualities requiring higher-level skills past mere memorization and the dealing with of multimodal information. In response to this hole, we introduce SciAssess, a benchmark particularly designed for the excellent analysis of LLMs in scientific literature evaluation. It goals to totally assess the efficacy of LLMs by evaluating their capabilities in Memorization (L1), Comprehension (L2), and Evaluation & Reasoning (L3). It encompasses quite a lot of duties drawn from numerous scientific fields, together with biology, chemistry, materials, and drugs. To make sure the reliability of SciAssess, rigorous high quality management measures have been applied, making certain accuracy, anonymization, and compliance with copyright requirements. SciAssess evaluates 11 LLMs, highlighting their strengths and areas for enchancment. We hope this analysis helps the continuing growth of LLM functions in scientific literature evaluation. SciAssess and its assets can be found at url{this https URL}.