Otter: Mitigating Background Distractions of Wide-Angle Few-Shot Action Recognition with Enhanced RWKV

Wide-angle videos in few-shot action recognition (FSAR) effectively express actions within specific scenarios. However, without a global understanding of both subjects and background, recognizing actions in such samples remains challenging because of the background distractions. Receptance Weighted Key Value (RWKV), which learns interaction between various dimensions, shows promise for global modeling. While directly applying RWKV to wide-angle FSAR may fail to highlight subjects due to excessive background information. Additionally, temporal relation degraded by frames with similar backgrounds is difficult to reconstruct, further impacting performance. Therefore, we design the CompOund SegmenTation and Temporal REconstructing RWKV (Otter). Specifically, the Compound Segmentation Module (CSM) is devised to segment and emphasize key patches in each frame, effectively highlighting subjects against background information. The Temporal Reconstruction Module (TRM) is incorporated into the temporal-enhanced prototype construction to enable bidirectional scanning, allowing better reconstruct temporal relation. Furthermore, a regular prototype is combined with the temporal-enhanced prototype to simultaneously enhance subject emphasis and temporal modeling, improving wide-angle FSAR performance. Extensive experiments on benchmarks such as SSv2, Kinetics, UCF101, and HMDB51 demonstrate that Otter achieves state-of-the-art performance. Extra evaluation on the VideoBadminton dataset further validates the superiority of Otter in wide-angle FSAR.

Authors

• Wenbo Huang Southeast University, Nanjing 211189, Jiangsu, China
• Jinghui Zhang Southeast University, Nanjing 211189, Jiangsu, China
• Zhenghao Chen University of Sydney
• Guang Li Hokkaido University, Sapporo 060-0808, Hokkaido, Japan
• Lei Zhang
• Yang Cao University of Science and Technology of China
• Fang Dong Southeast University, Nanjing 211189, Jiangsu, China
• Takahiro Ogawa Hokkaido University, Sapporo 060-0808, Hokkaido, Japan
• Miki Haseyama Hokkaido University, Sapporo 060-0808, Hokkaido, Japan

Keywords

No keywords are indexed for this paper.