arXiv:2409.16718v1 Announce Sort: cross
Summary: Current advances in fine-tuning Imaginative and prescient-Language Fashions (VLMs) have witnessed the success of immediate tuning and adapter tuning, whereas the basic mannequin fine-tuning on inherent parameters appears to be missed. It’s believed that fine-tuning the parameters of VLMs with few-shot samples corrupts the pre-trained information since fine-tuning the CLIP mannequin even degrades efficiency. On this paper, we revisit this viewpoint, and suggest a brand new perspective: fine-tuning the particular parameters as an alternative of all will uncover the facility of basic mannequin fine-tuning on VLMs. By way of our meticulous research, we suggest ClipFit, a easy but efficient technique to fine-tune CLIP with out introducing any overhead of additional parameters. We exhibit that by solely fine-tuning the particular bias phrases and normalization layers, ClipFit can enhance the efficiency of zero-shot CLIP by 7.27% common harmonic imply accuracy. Lastly, to know how fine-tuning in CLIPFit impacts the pre-trained fashions, we performed in depth experimental analyses w.r.t. adjustments in inner parameters and representations. We discovered that low-level textual content bias layers and the primary layer normalization layer change way more than different layers. The code is accessible at url{https://github.com/minglllli/CLIPFit}.
Source link
#VisionLanguage #Mannequin #FineTuning #Easy #ParameterEfficient #Modification
Unlock the potential of cutting-edge AI options with our complete choices. As a number one supplier within the AI panorama, we harness the facility of synthetic intelligence to revolutionize industries. From machine studying and information analytics to pure language processing and pc imaginative and prescient, our AI options are designed to reinforce effectivity and drive innovation. Discover the limitless potentialities of AI-driven insights and automation that propel your online business ahead. With a dedication to staying on the forefront of the quickly evolving AI market, we ship tailor-made options that meet your particular wants. Be a part of us on the forefront of technological development, and let AI redefine the best way you use and achieve a aggressive panorama. Embrace the long run with AI excellence, the place potentialities are limitless, and competitors is surpassed.
