Sequence-to-Segment Networks for Segment Detection

Detecting segments of interest from an input sequence is a challenging problem which often requires not only good knowledge of individual target segments, but also contextual understanding of the entire input sequence and the relationships between the target segments. To address this problem, we propose the Sequence-to-Segment Network (S$^2$N), a novel end-to-end sequential encoder-decoder architecture. S$^2$N first encodes the input into a sequence of hidden states that progressively capture both local and holistic information. It then employs a novel decoding architecture, called Segment Detection Unit (SDU), that integrates the decoder state and encoder hidden states to detect segments sequentially. During training, we formulate the assignment of predicted segments to ground truth as bipartite matching and use the Earth Mover's Distance to calculate the localization errors. We experiment with S$^2$N on temporal action proposal generation and video summarization and show that S$^2$N achieves state-of-the-art performance on both tasks.

Authors

• Zijun Wei
• Boyu Wang University of Western Ontario
• Minh Hoai Nguyen
• Jianming Zhang Adobe Research
• Zhe Lin Adobe Research
• Xiaohui Shen
• Radomir Mech Adobe Systems Incorporated
• Dimitris Samaras

Keywords

No keywords are indexed for this paper.