Author name cluster

Zhipeng Zhang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

18 papers

2 author rows

AAAI Conference 2026 Conference Paper

HDGS: Hierarchical Dynamic Gaussian Splatting for Urban Driving Scenes

Fudong Ge
Jin Gao
Hanshi Wang
Yiwei Zhang
Ke Wang
Weiming Hu
Zhipeng Zhang

This paper tackles the challenging task of achieving storage-efficient yet high-fidelity motion representation in large-scale dynamic 3D Gaussian Splatting. Our motivation stems from the truth that existing urban-scale methods, which rely on massive and unstructured individual Gaussians for scene modeling, face a critical scalability bottleneck. Inspired by recent advances in the 3DGS-based compression beyond autonomous driving, we address this challenge by leveraging the compression capability of anchor-driven methods. However, this is non-trivial as our exploratory experiments reveal that the direct application of this paradigm to dynamic, large-scale urban scenes results in performance degradation. We attribute this phenomenon to the hierarchical anchor design that severely loses dynamic information. To this end, we propose Hierarchical Dynamic Gaussian Splatting (HDGS), a novel framework designed to adapt the anchor-based Gaussian paradigm to 4D urban environments. We first establish a local support network to reinforce inter-anchor consistency, mitigating geometric and appearance fractures caused by supervision attenuation in deep hierarchies. Then, we handle heterogeneous object motion via coarse-to-fine decomposition, where high-level anchors model coarse dynamics and low-level anchors refine them with residual deformations. Third, we introduce a hybrid supervision scheme that fuses global geometric constraints and local pixel-level cues to alleviate geometrically inconsistent reconstruction under sparse LiDAR. Extensive experiments show that HDGS reduces storage by 69.0% while maintaining or even improving rendering fidelity compared to state-of-the-art methods.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Integrating Diverse Assignment Strategies into DETRs

Yiwei Zhang
Jin Gao
Hanshi Wang
Fudong Ge
Guan Luo
Weiming Hu
Zhipeng Zhang

Label assignment is a critical component in object detectors, particularly within DETR-style frameworks where the one-to-one matching strategy, despite its end-to-end elegance, suffers from slow convergence due to sparse supervision. While recent works have explored one-to-many assignments to enrich supervisory signals, they often introduce complex, architecture-specific modifications and typically focus on a single auxiliary strategy, lacking a unified and scalable design. In this paper, we first systematically investigate the effects of ``one-to-many'' supervision and reveal a surprising insight that performance gains are driven not by the sheer quantity of supervision, but by the diversity of the assignment strategies employed. This finding suggests that a more elegant, parameter-efficient approach is attainable. Building on this insight, we propose LoRA-DETR, a flexible and lightweight framework that seamlessly integrates diverse assignment strategies into any DETR-style detector. Our method augments the primary network with multiple Low-Rank Adaptation (LoRA) branches during training, each instantiating a different one-to-many assignment rule. These branches act as auxiliary modules that inject rich, varied supervisory gradients into the main model and are discarded during inference, thus incurring no additional computational cost. This design promotes robust joint optimization while maintaining the architectural simplicity of the original detector. Extensive experiments on different baselines validate the effectiveness of our approach. Our work presents a new paradigm for enhancing detectors, demonstrating that diverse ``one-to-many'' supervision can be integrated to achieve state-of-the-art results without compromising model elegance.

PDF Details DOI

AAAI Conference 2026 Conference Paper

RPE-PAD: Relative Pose Estimation for Pose-agnostic Anomaly Detection

Zhipeng Zhang
Mengzan Qi
Rongkang Ma
Yingying Fang
Guixu Zhang
Tieyong Zeng
Zhi Li

Pose-agnostic Anomaly Detection (PAD) aims to detect anomalies when the poses of query images are unknown and differ from those in the training set. Therefore, accurately estimating the camera poses for the query images in the test set is critical for this task. Existing query-specific framework methods require re-optimizing a new set of parameters for each query image, limiting their generalization and increasing computational burden. To overcome these limitations, we propose a novel method, Relative Pose Estimation for Pose-agnostic Anomaly Detection (RPE-PAD), which enhances both generalization and efficiency with a query-independent framework. Specifically, we propose a Random View Synthesis Scheme (RVSS) that generates new poses by adding Gaussian perturbations to the original poses, then renders the corresponding views to augment the dataset. To estimate the relative camera pose between two input images, we introduce an Iterative Relative Pose Refinement Network (IRPRN), which incorporates a hierarchical coarse-to-fine refinement strategy. Furthermore, we employ a Multi-Pair Training Strategy (MPTS) to train the proposed IRPRN, leveraging multiple image pairs to expand the relative pose transformation space during training. Extensive experiments demonstrate that our method achieves robust anomaly detection performance while significantly improving inference efficiency.

PDF Details DOI

EAAI Journal 2026 Journal Article

The research on railway inspection visual localization method based on multilevel feature matching

Peigang Li
Zexuan Liu
Qing Zhang
Zhipeng Zhang
Lu Wang
Jingqi Li
Jianjun Chen

Accurate localization is essential for railway inspection operations. This study addresses the limitations of existing localization methods concerning complexity, accuracy, and cost by proposing a two-stage Visual Position Recognition (VPR) method suitable for railway personnel and train passengers. This method is based on deep learning with a 34-layer residual network (ResNet34) and an epipolar geometry constraint model. Initially, a “visual-mileage” database is constructed by linking “far-near” trackside scenes to their corresponding mileage. The proposed two-stage localization framework consists of: (1) a preliminary localization stage that utilizes an improved ResNet34 and Network Vector of Locally Aggregated Descriptors (NetVLAD) for feature aggregation to extract visual features, enhanced through a triplet loss function for better discrimination; (2) a precise localization stage that employs epipolar geometry to estimate relative pose for high-accuracy localization. Experimental results from a 2. 38-km campus test route and a 30-km railway field test demonstrate a retrieval accuracy of 87. 5 % for the correct target location within the top 5 candidate images (TOP-5) and 83. 9 % within the top 1 candidate image (TOP-1). The average positioning error is 5. 67 m, with 90 % of errors being less than or equal to 10 m, thereby meeting the precision requirements for railway inspections. Our research represents the first application of Artificial Intelligence-driven visual localization in railway inspection positioning, innovatively incorporating train passengers into the railway inspection process and providing a novel solution for intelligent railway system inspections.

Details DOI

NeurIPS Conference 2025 Conference Paper

Each Complexity Deserves a Pruning Policy

Hanshi Wang
Yuhao Xu
Zekun Xu
Jin Gao
Yufan Liu
Weiming Hu
Ke Wang
Zhipeng Zhang

The established redundancy in visual tokens within large vision–language models (LVLMs) allows for pruning to effectively reduce their substantial computational demands. Empirical evidence from previous works indicates that visual tokens in later decoder stages receive less attention than shallow layers. Then, previous methods typically employ heuristics layer-specific pruning strategies where, although the number of tokens removed may differ across decoder layers, the overall pruning schedule is fixed and applied uniformly to all input samples and tasks, failing to align token elimination with the model’s holistic reasoning trajectory. Cognitive science indicates that human visual processing often begins with broad exploration to accumulate evidence before narrowing focus as the target becomes distinct. Our experiments reveal an analogous pattern in LVLMs. This observation strongly suggests that neither a fixed pruning schedule nor a heuristics layer-wise strategy can optimally accommodate the diverse complexities inherent in different inputs. To overcome this limitation, we introduce Complexity-Adaptive Pruning (AutoPrune), which is a training-free, plug-and-play framework that tailors pruning policies to varying sample and task complexities. Specifically, AutoPrune quantifies the mutual information between visual and textual tokens, and then projects this signal to a budget-constrained logistic retention curve. Each such logistic curve, defined by its unique shape, is shown to effectively correspond with the specific complexity of different tasks, and can easily guarantee adherence to a pre-defined computational constraints. We evaluate AutoPrune not only on standard vision-language tasks but also on Vision-Language-Action (VLA) models for autonomous driving. Notably, when applied to LLaVA-1. 5-7B, our method prunes 89% of visual tokens and reduces inference FLOPs by 76. 8%, but still retaining 96. 7% of the original accuracy averaged over all tasks. This corresponds to a 9. 1% improvement over the recent work PDrop (CVPR'2025), demonstrating the effectivenes. Code is available at https: //github. com/AutoLab-SAI-SJTU/AutoPrune.

PDF Details

NeurIPS Conference 2025 Conference Paper

EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models

Yantai Yang
Yuhao Wang
Zichen Wen
Luo Zhongwei
Chang Zou
Zhipeng Zhang
Chuan Wen
Linfeng Zhang

Vision-Language-Action (VLA) models, particularly diffusion-based architectures, demonstrate transformative potential for embodied intelligence but are severely hampered by high computational and memory demands stemming from extensive inherent and inference-time redundancies. While existing acceleration efforts often target isolated inefficiencies, such piecemeal solutions typically fail to holistically address the varied computational and memory bottlenecks across the entire VLA pipeline, thereby limiting practical deployability. We introduce EfficientVLA, a structured and training-free inference acceleration framework that systematically eliminates these barriers by cohesively exploiting multifaceted redundancies. EfficientVLA synergistically integrates three targeted strategies: (1) pruning of functionally inconsequential layers from the language module, guided by an analysis of inter-layer redundancies; (2) optimizing the visual processing pathway through a task-aware strategy that selects a compact, diverse set of visual tokens, balancing task-criticality with informational coverage; and (3) alleviating temporal computational redundancy within the iterative diffusion-based action head by strategically caching and reusing key intermediate features. We apply our method to a standard VLA model CogACT, yielding a $1. 93\times$ inference speedup and reduces FLOPs to 28. 9%, with only a 0. 6% success rate drop in the SIMPLER benchmark.

PDF Details

NeurIPS Conference 2025 Conference Paper

LoRATv2: Enabling Low-Cost Temporal Modeling in One-Stream Trackers

Liting Lin
Heng Fan
Zhipeng Zhang
Yuqing Huang
Yaowei Wang
Yong Xu
Haibin Ling

Transformer-based algorithms, such as LoRAT, have significantly enhanced object-tracking performance. However, these approaches rely on a standard attention mechanism, which incurs quadratic token complexity, making real-time inference computationally expensive. In this paper, we introduce LoRATv2, a novel tracking framework that addresses these limitations with three main contributions. First, LoRATv2 integrates frame-wise causal attention, which ensures full self-attention within each frame while enabling causal dependencies across frames, significantly reducing computational overhead. Moreover, key-value (KV) caching is employed to efficiently reuse past embeddings for further speedup. Second, building on LoRAT's parameter-efficient fine-tuning, we propose Stream-Specific LoRA Adapters (SSLA). As frame-wise causal attention introduces asymmetry in how streams access temporal information, SSLA assigns dedicated LoRA modules to the template and each search stream, with the main ViT backbone remaining frozen. This allows specialized adaptation for each stream's role in temporal tracking. Third, we introduce a two-phase progressive training strategy, which first trains a single-search-frame tracker and then gradually extends it to multi-search-frame inputs by introducing additional LoRA modules. This curriculum-based learning paradigm improves long-term tracking while maintaining training efficiency. In extensive experiments on multiple benchmarks, LoRATv2 achieves state-of-the-art performance, substantially improved efficiency, and a superior performance-to-FLOPs ratio over state-of-the-art trackers. The code is available at https: //github. com/LitingLin/LoRATv2.

PDF Details

NeurIPS Conference 2025 Conference Paper

Online Segment Any 3D Thing as Instance Tracking

Hanshi Wang
Cai Zijian
Jin Gao
Yiwei Zhang
Weiming Hu
Ke Wang
Zhipeng Zhang

Online, real-time, and fine-grained 3D segmentation constitutes a fundamental capability for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries to aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions. Nevertheless, perception, whether human or robotic, is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. This deficiency in temporal reasoning can exacerbate issues such as the over-segmentation commonly produced by VFMs, necessitating more handcrafted post-processing. Therefore, to further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as an instance tracking problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model in developing a holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-art, surpassing ESAM by 2. 8 AP on ScanNet200 and delivering consistent gains on ScanNet, SceneNN, and 3RScan datasets, corroborating that identity-aware temporal reasoning is a crucial, previously underemphasized component for robust 3D segmentation in real-time embodied intelligence. Code is at https: //github. com/AutoLab-SAI-SJTU/AutoSeg3D.

PDF Details

AAAI Conference 2025 Conference Paper

ScoreNet: Consistency-driven Framework with Multi-side Information Fusion for Session-based Recommendation

Piao Tong
Qiao Liu
Zhipeng Zhang
Yuke Wang
Tian Lan

Fusing side information in session-based recommendation is crucial for improving the performance of next-item prediction by providing additional context. Recent methods optimize attention weights by combining item and side information embeddings. However, semantic heterogeneity between item IDs and side information introduces computational noise in attention calculation, leading to inconsistencies in user interest modeling and reducing the accuracy of candidate item scores. These methods also often fail to leverage session-based re-interaction patterns, limiting improvements in score prediction during the decoding phase. To address these challenges, we propose ScoreNet, a consistency-driven framework with multi-side information fusion for session-based recommendation. ScoreNet explicitly models users' persistent preferences, generating consistent decoding scores for candidate items within a unified framework. It incorporates a multi-path re-engagement network to capture re-interaction behavior patterns in a semantic-agnostic manner, enhancing side information fusion while avoiding semantic interference. Additionally, a position-enhanced consistent scoring network redistributes attention scores within sessions, improving prediction accuracy, especially for items with limited interactions. Extensive experiments on three real-world datasets demonstrate that ScoreNet outperforms state-of-the-art models.

PDF Details DOI

EAAI Journal 2025 Journal Article

Spatial-temporal modeling for multi-scale wind speed predictions in rail transit networks: A graph neural network-based methodology

Yujie Huang
Zhipeng Zhang
Wenqiang Zhao
Hao Hu

The operation safety and efficiency of rail transit network are vulnerable to strong winds. An essential requirement exists for employing field monitoring data to estimate wind speed trends and make subsequent emergency decisions in time. However, the current warning systems primarily focus on wind speeds at individual monitoring spots or single lines, while the spatial dependency of wind patterns within a larger space context is ignored. To bridge this research gap, this study proposes a pioneering network-scale wind speed prediction approach based on spatial-temporal graph neural networks. First, a simulated wind stress vector field is devised, leveraging time-lagged cross correlation, to enhance the input wind sensors graph. This novel approach improves upon the traditional predefined graph structure, which relies solely on geographical distances. Second, diffusion convolution recurrent neural network is applied to model the spatial-temporal wind diffusion patterns and predict future wind speed. By carrying out comparison analysis on two real-world railroad datasets with different spatial and temporal granularity, the proposed model demonstrates its applicability in multi-scale wind predictions, significantly outperforming widely used baselines, particularly in multi-step ahead predictions. In addition, ablation study is conducted to assess and interpret the contributions of each component within the model. The proposed method in this study, along with obtained prediction results, is expected to support the network dispatching and delay assessment during extreme weather conditions, thereby enhancing the effectiveness of wind warning systems and strengthening rail safety management.

Details DOI

ICRA Conference 2025 Conference Paper

The Devil is in the Quality: Exploring Informative Samples for Semi-Supervised Monocular 3D Object Detection

Zhipeng Zhang
Zhenyu Li 0007
Hanshi Wang
Yuan He
Ke Wang
Heng Fan 0001

This paper tackles the challenging problem of semi-supervised monocular 3D object detection with a general framework. In specific, having observed that the bottleneck of this task lies in lacking reliable and informative samples from unlabeled data for detector learning, we introduce a novel simple yet effective ‘Augment and Criticize’ pipeline that mines abundant informative samples for robust detection. To be more specific, in the ‘Augment’ stage, we present the Augmentation-based Prediction aGgregation (APG), which applies automatically learned transformations to unlabeled images and aggregates detections from various augmented views as pseudo labels. Since not all the pseudo labels from APG are beneficially informative, the subsequent ‘Criticize’ phase is introduced. Particularly, we present the Critical Retraining Strategy (CRS) that, unlike simply filtering pseudo labels using a fixed threshold, employs a learnable network to evaluate the contribution of unlabeled images at different training timestamps. This way, the noisy samples prohibitive to model evolution can be effectively suppressed. In order to validate ‘Augment-Criticize’, we apply it to MonoDLE [1] and MonoFlex [2], and the two new detectors, dubbed 3DSeMo DLE and 3DSeMo FLEX, achieve state-of-the-art results with consistent improvements, evidencing its effectiveness and generality.

Details

ICML Conference 2024 Conference Paper

Image Fusion via Vision-Language Model

Zixiang Zhao
Lilun Deng
Haowen Bai
Yukun Cui
Zhipeng Zhang
Yulun Zhang 0001
Haotong Qin
Dongdong Chen

Image fusion integrates essential information from multiple images into a single composite, enhancing structures, textures, and refining imperfections. Existing methods predominantly focus on pixel-level and semantic visual features for recognition, but often overlook the deeper text-level semantic information beyond vision. Therefore, we introduce a novel fusion paradigm named image Fusion via vIsion-Language Model (FILM), for the first time, utilizing explicit textual information from source images to guide the fusion process. Specifically, FILM generates semantic prompts from images and inputs them into ChatGPT for comprehensive textual descriptions. These descriptions are fused within the textual domain and guide the visual information fusion, enhancing feature extraction and contextual understanding, directed by textual semantic information via cross-attention. FILM has shown promising results in four image fusion tasks: infrared-visible, medical, multi-exposure, and multi-focus image fusion. We also propose a vision-language dataset containing ChatGPT-generated paragraph descriptions for the eight image fusion datasets across four fusion tasks, facilitating future research in vision-language model-based image fusion. Code and dataset are available at https: //github. com/Zhaozixiang1228/IF-FILM.

Details

NeurIPS Conference 2024 Conference Paper

In this paper, we propose a novel benchmark, named VastTrack, aiming to facilitate the development of general visual tracking via encompassing abundant classes and videos. VastTrack consists of a few attractive properties: (1) Vast Object Category. In particular, it covers targets from 2, 115 categories, significantly surpassing object classes of existing popular benchmarks (e. g. , GOT-10k with 563 classes and LaSOT with 70 categories). Through providing such vast object classes, we expect to learn more general object tracking. (2) Larger scale. Compared with current benchmarks, VastTrack provides 50, 610 videos with 4. 2 million frames, which makes it to date the largest dataset in term of the number of videos, and hence could benefit training even more powerful visual trackers in the deep learning era. (3) Rich Annotation. Besides conventional bounding box annotations, VastTrack also provides linguistic descriptions with more than 50K sentences for the videos. Such rich annotations of VastTrack enable the development of both vision-only and vision-language tracking. In order to ensure precise annotation, each frame in the videos is manually labeled with multi-stage of careful inspections and refinements. To understand performance of existing trackers and to provide baselines for future comparison, we extensively evaluate 25 representative trackers. The results, not surprisingly, display significant drops compared to those on current datasets due to lack of abundant categories and videos from diverse scenarios for training, and more efforts are urgently required to improve general visual tracking. Our VastTrack, the toolkit, and evaluation results are publicly available at https: //github. com/HengLan/VastTrack.

PDF Details DOI

ECAI Conference 2023 Conference Paper

Time-Series Data Imputation via Realistic Masking-Guided Tri-Attention Bi-GRU

Yiqun Zhang 0006
An Zeng
Dan Pan 0001
Yuzhu Ji
Zhipeng Zhang
Jing Lin

Time series data with missing values are ubiquitous in real applications due to various unforeseen faults during data generation, storage, and transmission. Time-Series Data Imputation (TSDI) is thus crucial to many temporal data analysis tasks. However, existing works usually consider only one of the following two issues: (1) intra-feature temporal dependency, and (2) inter-feature correlation, leading to the overlook of complex coupling information in imputation. To achieve more accurate TDSI, we design a novel imputation model called TABiG, which delicately preserves the short-term, long-term, and inter-feature dependencies by attention mechanisms in a delay error-reduced bi-directional architecture. That is, it leverages GRU to model short-term temporal dependencies and adopts self-attention mechanisms hierarchically to capture long-term temporal dependencies and inter-feature correlations. The multiple self-attention mechanisms are nested in a bi-directional structure to alleviate the problem of delay errors in RNN-like structures. To facilitate model training with higher generalization, a masking strategy that mimics various extreme real missing situations beyond the simple random ones has been adopted for generating self-supervised learning tasks. Comprehensive experiments demonstrate that TABiG significantly outperforms most state-of-the-art imputation counterparts. Complementary results and source code can be accessed at https: //github. com/Zhang2112105189/TABiG

Details

NeurIPS Conference 2022 Conference Paper

Divert More Attention to Vision-Language Tracking

Mingzhe Guo
Zhipeng Zhang
Heng Fan
Liping Jing

Relying on Transformer for complex visual feature learning, object tracking has witnessed the new standard for state-of-the-arts (SOTAs). However, this advancement accompanies by larger training data and longer training period, making tracking increasingly expensive. In this paper, we demonstrate that the Transformer-reliance is not necessary and the pure ConvNets are still competitive and even better yet more economical and friendly in achieving SOTA tracking. Our solution is to unleash the power of multimodal vision-language (VL) tracking, simply using ConvNets. The essence lies in learning novel unified-adaptive VL representations with our modality mixer (ModaMixer) and asymmetrical ConvNet search. We show that our unified-adaptive VL representation, learned purely with the ConvNets, is a simple yet strong alternative to Transformer visual features, by unbelievably improving a CNN-based Siamese tracker by 14. 5% in SUC on challenging LaSOT (50. 7%$\rightarrow$65. 2%), even outperforming several Transformer-based SOTA trackers. Besides empirical results, we theoretically analyze our approach to evidence its effectiveness. By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope to open more possibilities for future tracking beyond Transformer. Code and models are released at https: //github. com/JudasDie/SOTS.

PDF Details

IJCAI Conference 2022 Conference Paper

Learning Target-aware Representation for Visual Tracking via Informative Interactions

Mingzhe Guo
Zhipeng Zhang
Heng Fan
Liping Jing
Yilin Lyu
Bing Li
Weiming Hu

We introduce a novel backbone architecture to improve target-perception ability of feature representation for tracking. Having observed de facto frameworks perform feature matching simply using the backbone outputs for target localization, there is no direct feedback from the matching module to the backbone network, especially the shallow layers. Concretely, only the matching module can directly access the target information, while the representation learning of candidate frame is blind to the reference target. Therefore, the accumulated target-irrelevant interference in shallow stages may degrade the feature quality of deeper layers. In this paper, we approach the problem by conducting multiple branch-wise interactions inside the Siamese-like backbone networks (InBN). The core of InBN is a general interaction modeler (GIM) that injects the target information to different stages of the backbone network, leading to better target-perception of candidate feature representation with negligible computation cost. The proposed GIM module and InBN mechanism are general and applicable to different backbone types including CNN and Transformer for improvements, as evidenced on multiple benchmarks. In particular, the CNN version improves the baseline with 3. 2/6. 9 absolute gains of SUC on LaSOT/TNL2K. The Transformer version obtains SUC of 65. 7/52. 0 on LaSOT/TNL2K, which are on par with recent SOTAs.

PDF Details DOI

AAAI Conference 2022 Conference Paper

One More Check: Making “Fake Background” Be Tracked Again

Chao Liang
Zhipeng Zhang
Xue Zhou
Bing Li
Weiming Hu

The one-shot multi-object tracking, which integrates object detection and ID embedding extraction into a unified network, has achieved groundbreaking results in recent years. However, current one-shot trackers solely rely on singleframe detections to predict candidate bounding boxes, which may be unreliable when facing disastrous visual degradation, e. g. , motion blur, occlusions. Once a target bounding box is mistakenly classified as background by the detector, the temporal consistency of its corresponding tracklet will be no longer maintained. In this paper, we set out to restore the bounding boxes misclassified as “fake background” by proposing a re-check network. The re-check network innovatively expands the role of ID embedding from data association to motion forecasting by effectively propagating previous tracklets to the current frame with a small overhead. Note that the propagation results are yielded by an independent and efficient embedding search, preventing the model from overrelying on detection results. Eventually, it helps to reload the “fake background” and repair the broken tracklets. Building on a strong baseline CSTrack, we construct a new one-shot tracker and achieve favorable gains by 70. 7 → 76. 4, 70. 6 → 76. 3 MOTA on MOT16 and MOT17, respectively. It also reaches a new state-of-the-art MOTA and IDF1 performance. Code is released at https: //github. com/JudasDie/SOTS.

PDF Details

NeurIPS Conference 2022 Conference Paper

SwinTrack: A Simple and Strong Baseline for Transformer Tracking

Liting Lin
Heng Fan
Zhipeng Zhang
Yong Xu
Haibin Ling

Recently Transformer has been largely explored in tracking and shown state-of-the-art (SOTA) performance. However, existing efforts mainly focus on fusing and enhancing features generated by convolutional neural networks (CNNs). The potential of Transformer in representation learning remains under-explored. In this paper, we aim to further unleash the power of Transformer by proposing a simple yet efficient fully-attentional tracker, dubbed SwinTrack, within classic Siamese framework. In particular, both representation learning and feature fusion in SwinTrack leverage the Transformer architecture, enabling better feature interactions for tracking than pure CNN or hybrid CNN-Transformer frameworks. Besides, to further enhance robustness, we present a novel motion token that embeds historical target trajectory to improve tracking by providing temporal context. Our motion token is lightweight with negligible computation but brings clear gains. In our thorough experiments, SwinTrack exceeds existing approaches on multiple benchmarks. Particularly, on the challenging LaSOT, SwinTrack sets a new record with 0. 713 SUC score. It also achieves SOTA results on other benchmarks. We expect SwinTrack to serve as a solid baseline for Transformer tracking and facilitate future research. Our codes and results are released at https: //github. com/LitingLin/SwinTrack.

PDF Details