Exposing and Improving Fine-Grained Visual Grounding Abilities of Lightweight Multimodal LLMs

Lightweight multimodal LLMs can localize whole objects effectively, yet often struggle when a query targets a small object part or fine-grained visual detail. This talk presents a reasoning-guided framework that teaches compact models to ground parts through an explicit coarse-to-fine process: first locating the parent object, then identifying the requested part. A part-aware reinforcement-learning objective provides stage-wise rewards for object accuracy, part containment, and the consistency of the model’s self-critique. Using these techniques, a compact 4B-parameter model achieves state-of-the-art zero-shot part grounding while preserving its object-level performance. These advances can be used to enable lightweight MLLMs to support detail-oriented tasks in biology and robotics. Kazi Mehrab is a CS PhD candidate at Virginia Tech, currently focusing on multimodal LLMs and computer vision tasks, including visual perception, reasoning and grounding. Kazi also likes to apply AI techniques to scientific domains.