Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigation

Visual Language Navigation (VLN) is a fundamental task within the field of Embodied AI, focusing on the ability of agents to navigate complex environments based on natural language instructions. Despite the progress made by existing methods, these methods often present some common challenges. First, they rely on pre-trained backbone models for visual perception, which struggle with the dynamic viewpoints in VLN scenarios. Second, the performance is limited when using pre-trained LLMs or VLMs without fine-tuning, due to the absence of VLN domain knowledge. Third, while fine-tuning LLMs and VLMs can improve results, their computational costs are higher than those without fine-tuning. To address these limitations, we propose Weakly-supervised Partial Contrastive Learning (WPCL), a method that enhances an agent’s ability to identify objects from dynamic viewpoints in VLN scenarios by effectively integrating pre-trained VLM knowledge into the perception process, without requiring VLM fine-tuning. Our method enhances the agent’s ability to interpret and respond to environmental cues while ensuring computational efficiency. Experimental results have shown that our method outperforms the baseline methods on multiple benchmarks, which validates the effectiveness, robustness, and generalizability of our method.

Authors

• Ruoyu Wang 0038
• Tong Yu 0001
• Junda Wu
• Yao Liu 0017
• Julian J. McAuley
• Lina Yao 0001

Keywords

• Visualization
• Navigation
• Computational modeling
• Natural languages
• Contrastive learning
• Benchmark testing
• Robustness
• Computational efficiency
• Object recognition
• Visual perception
• Self-supervised Learning
• Vision-language Navigation
• Computational Cost
• Dynamic Point Of View
• Multiple Benchmarks
• Positive Samples
• Negative Samples
• Object Detection
• Mutual Information
• Temperature Parameters
• Common Objects
• Activity Prediction
• Proven Effectiveness
• Navigation Task
• Process Of Agents
• Foundation Model
• Viewpoint Changes
• List Of Objects
• Pre-trained Language Models
• Supervisory Signal
• Fine-tuning Step