TIME: Temporal-Sensitive Multi-Dimensional Instruction Tuning and Robust Benchmarking for Video-LLMs

Video large language models have achieved remarkable performance in tasks such as video question answering, however, their temporal understanding remains suboptimal. To address this limitation, we curate a dedicated instruction fine-tuning dataset that focuses on enhancing temporal comprehension across five key dimensions. In order to reduce reliance on costly temporal annotations, we introduce a multi-task prompt fine-tuning approach that seamlessly integrates temporal-sensitive tasks into existing instruction datasets without requiring additional annotations. Furthermore, we develop a novel benchmark for temporal-sensitive video understanding that not only fills the gaps in dimension coverage left by existing benchmarks but also rigorously filters out potential shortcuts, ensuring a more accurate evaluation. Extensive experimental results demonstrate that our approach significantly enhances the temporal understanding of video-LLMs while avoiding reliance on shortcuts.

Authors

• Yunxiao Wang Shandong University, Jinan, China
• Meng Liu School of Computer, National University of Defense Technology
• Wenqi Liu
• Xuemeng Song Shandong University
• Bin Wen
• Fan Yang Nanyang Technological University
• Tingting Gao Kuaishou Technology
• Di Zhang Shanghai Artificial Intelligence Laboratory Fudan University
• Guorui Zhou Kuaishou Technology
• Liqiang Nie Harbin Institute of Technology (Shenzhen)

Keywords

No keywords are indexed for this paper.