Bridging Vision and Genomics for Biodiversity Monitoring at Scale

View a PDF of the paper titled CLIBD: Bridging Imaginative and prescient and Genomics for Biodiversity Monitoring at Scale, by ZeMing Gong and 6 different authors

View PDF
HTML (experimental)

Summary:Measuring biodiversity is essential for understanding ecosystem well being. Whereas prior works have developed machine studying fashions for taxonomic classification of photographic photographs and DNA individually, on this work, we introduce a multimodal method combining each, utilizing CLIP-style contrastive studying to align photographs, barcode DNA, and text-based representations of taxonomic labels in a unified embedding house. This permits for correct classification of each identified and unknown insect species with out task-specific fine-tuning, leveraging contrastive studying for the primary time to fuse DNA and picture knowledge. Our technique surpasses earlier single-modality approaches in accuracy by over 8% on zero-shot studying duties, showcasing its effectiveness in biodiversity research.