Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting

In this work, we introduce CAT-V (Caption Anything in Video), a training-free framework for fine-grained object-centric video captioning of user-selected instances. CAT-V combines (i) a SAMURAI-based Segmenter for precise object masks across frames, (ii) a TRACE-Uni Temporal Analyzer for event boundary detection and coarse event descriptions, and (iii) an InternVL-2.5 Captioner that, conditioned on spatiotemporal visual prompts and chain-of-thought (CoT) guidance, produces detailed, temporally coherent captions about object attributes, actions, states, interactions, and context. The system supports point, box, and region prompts and maintains temporal sensitivity by tracking object states across segments. In contrast to vanilla video captioning that is overly abstract and dense video captioning that is often terse, CAT-V enables object-level specificity with spatial accuracy and temporal coherence, without additional training data.

Authors

• Yolo Yunlong Tang University of Rochester
• Jing Bi University of Rochester
• Chao Huang Sun Yat-Sen University
• Susan Liang
• Daiki Shimada Sony Group Corporation
• Hang Hua University of Rochester
• Yunzhong Xiao
• Yizhi Song
• Pinxin Liu
• Mingqian Feng University of Rochester
• Junjia Guo University of Rochester
• Zhuo Liu University of Rochester
• Luchuan Song
• Ali Vosoughi
• Jinxi He University of Rochester
• Liu He
• Zeliang Zhang
• Jiebo Luo University of Rochester
• Chenliang Xu University of Rochester

Keywords

No keywords are indexed for this paper.