Improved Visual Grounding through Self-Consistent Explanations

With #CVPR2024 coming soon, check out the conversation of Harpreet Sahota’s (Hacker-in-Residence @Voxel51) and Jason Corso (Co-Founder and Chief Science Officer) with Dr. Paola Cascante-Bonilla and Ruozhen He about their paper "Improved Visual Grounding through Self-Consistent Explanations" The main contribution is introducing a weakly supervised strategy called SelfEQ, which leverages paraphrases to consistently improve the model's ability to localize objects in images. 😡 The Problem Vision-and-language models trained to match images with text struggle with the precise localization of objects, especially when the textual descriptions vary slightly (e.g., "frisbee" vs. "disc"). The challenge is to improve these models' grounding abilities without relying on extensive object location annotations. Current methods often require additional finetuning with a bounding box or segment annotations or depend on pretrained object detectors. These approaches are limited by their need for detailed annotations and can lack consistency when handling varied textual descriptions. Specific Issues: 📖 Lack of Detail: Existing models may not handle diverse vocabulary well, leading to inconsistent localization. 👯‍♀️ Inconsistency: Models may fail to provide consistent visual explanations for paraphrased textual inputs referring to the same object. 😄 The Solution SelfEQ encourages self-consistent visual explanations for paraphrased text inputs. This method involves generating paraphrases using a large language model and finetuning the vision-and-language model to ensure that the original and paraphrased texts map to the same image regions. 🧐 How It Works - Start with an existing method: The ALBEF model, which aligns images and text using image-text pairs without object location annotations, serves as the foundation. - Improvements by SelfEQ: 1. Paraphrase Generation: A LLM (Vicuna-13B) creates paraphrases for text descriptions. 2. Self-Consistency Tuning Finetunes the model using GradCAM to ensure consistent visual attention maps for original and paraphrased texts. 🤔 Why It's Better - Expanded Vocabulary: The model can handle a broader range of textual descriptions. - Improved Localization: SelfEQ enhances the precision and consistency of object localization without requiring bounding box annotations. - Efficiency: The approach leverages weak supervision, reducing the need for detailed annotations and making the finetuning process more efficient. 🔑 Key Contributions - Novel Objective: Introduces a self-consistency equivalence tuning objective to improve visual grounding. - Paraphrase Utilization: Employs large language models to generate high-quality paraphrases, enhancing the model's vocabulary handling. ---- If you'll be at the show in Seattle next week, swing by the Voxel51 booth (that's booth 1519). #deeplearning #artificialintelligence