Liu Liu

AAAI Conference 2026 Conference Paper

CrystalDiT: Simple Diffusion Transformers for Crystal Generation

• Xiaohan Yi
• Guikun Xu
• Zhong Zhang
• Liu Liu
• Yatao Bian
• Xi Xiao
• Peilin Zhao

We present CrystalDiT, a diffusion transformer for crystal structure generation that achieves state-of-the-art performance by challenging the trend of architectural complexity. Instead of intricate, multi-stream designs, CrystalDiT employs a unified transformer that imposes a powerful inductive bias: treating lattice and atomic properties as a single, interdependent system. Combined with a periodic table-based atomic representation and a balanced training strategy, our approach achieves 8.78% SUN (Stable, Unique, Novel) rate on MP-20, substantially outperforming recent methods including FlowMM (4.21%) and MatterGen (3.66%). Notably, CrystalDiT generates 63.28% unique and novel structures while maintaining comparable stability rates, demonstrating that architectural simplicity can be more effective than complexity for materials discovery. Our results suggest that in data-limited scientific domains, carefully designed simple architectures outperform sophisticated alternatives that are prone to overfitting.

AAAI Conference 2026 Conference Paper

Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds

• Xianhui Meng
• Yukang Huo
• Li Zhang
• Liu Liu
• Haonan Jiang
• Yan Zhong
• Pingrui Zhang
• Cewu Lu

Articulated objects are prevalent in daily life and robotic manipulation tasks. However, compared to rigid objects, pose tracking for articulated objects remains an underexplored problem due to their inherent kinematic constraints. To address these challenges, this work proposes a novel point-pair-based pose tracking framework, termed PPF-Tracker. The proposed framework first performs quasi-canonicalization of point clouds in the SE(3) Lie group space, and then models articulated objects using Point Pair Features (PPF) to predict pose voting parameters by leveraging the invariance properties of SE(3). Finally, semantic information of joint axes is incorporated to impose unified kinematic constraints across all parts of the articulated object. PPF-Tracker is systematically evaluated on both synthetic datasets and real-world scenarios, demonstrating strong generalization across diverse and challenging environments. Experimental results highlight the effectiveness and robustness of PPF-Tracker in multi-frame pose tracking of articulated objects. We believe this work can foster advances in robotics, embodied intelligence, and augmented reality.

AAAI Conference 2026 Conference Paper

FRBAT: Conditionally-Visible Physical Backdoor Attack via Fluorescence

• Yalun Wu
• Liu Liu
• Endong Tong
• Yingxiao Xiang
• Xiaoting Lyu
• Zhen Han
• Jiqiang Liu

Deep neural networks are increasingly vulnerable to physically deployable backdoor attacks, which manipulate real-world objects to induce targeted model failures. However, current physical backdoor attacks predominantly rely on perpetually visible triggers appended to target objects. These methods inevitably expose attack traces during the deployment phase, risking human suspicion prior to activation. In this paper, we propose a conditionally-visible physical backdoor attack, which can only be activated under specific optical conditions and thereby overcomes the risk of being detected after deployment and before the attack. Specifically, to ensure robust and reliable activation, we design irregular polygonal pattern as triggers to against across environmental variations. Moreover, we introduce a dual-phase mechanism (dormant and activated) to enable stealthy deployment. Our trigger remains invisible and dormant under non-attack conditions, leaving no physical traces. It activates instantaneously under specific illumination, inducing the target model to perform the desired behavior. We conduct experiments on traffic sign recognition tasks to compare our attack with six digital and seven physical attacks, and assess its performance against potential defenses. Extensive experimental results demonstrate the effectiveness, stealthiness, and robustness of our attack.

AAAI Conference 2026 Conference Paper

IGFuse: Interactive 3D Gaussian Scene Reconstruction via Multi-Scans Fusion

• Wenhao Hu
• Zesheng Li
• Haonan Zhou
• Liu Liu
• Xuexiang Wen
• Zhizhong Su
• Xi Li
• Gaoang Wang

Reconstructing complete and interactive 3D scenes remains a fundamental challenge in computer vision and robotics, particularly due to persistent object occlusions and limited sensor coverage. Even multi-view observations from a single scene scan often fail to capture the full structural details. Existing approaches typically rely on multi-stage pipelines—such as segmentation, background completion, and inpainting—or require per-object dense scanning, both of which are error-prone, and not easily scalable. We propose IGFuse, a novel framework that reconstructs interactive Gaussian scene by fusing observations from multiple scans, where natural object rearrangement between captures reveal previously occluded regions. Our method constructs segmentation-aware Gaussian fields and enforces bi-directional photometric and semantic consistency across scans. To handle spatial misalignments, we introduce a pseudo-intermediate scene state for symmetric alignment, alongside collaborative co-pruning strategies to refine geometry. IGFuse enables high-fidelity rendering and object-level scene manipulation without dense observations or complex pipelines. Extensive experiments validate the framework’s strong generalization to novel scene configurations, demonstrating its effectiveness for real-world 3D reconstruction and real-to-simulation transfer.

AAAI Conference 2026 Conference Paper

MINGLE: VLMs for Semantically Complex Region Detection in Urban Scenes

• Liu Liu
• Alexandra Schild
• Marco Cipriano
• Fatimeh Al Ghannam
• Freya Tan
• Gerard de Melo
• Andres Sevtsuk

Understanding group-level social interactions in public spaces is crucial for urban planning, informing the design of socially vibrant and inclusive environments. Detecting such interactions from images involves interpreting subtle visual cues such as relations, proximity and co-movement – semantically complex signals that go beyond traditional object detection. To address this challenge, we introduce a social group region detection task, which requires inferring and spatially grounding visual regions defined by abstract interpersonal relations. We propose MINGLE (Modeling INterpersonal Group-Level Engagement), a modular three-stage pipeline that integrates: (1) off-the-shelf human detection and depth estimation, (2) VLM-based reasoning to classify pairwise social affiliation, and (3) a lightweight spatial aggregation algorithm to localize socially connected groups. To support this task and encourage future research, we present a new dataset of 100K urban street-view images annotated with bounding boxes and labels for both individuals and socially interacting groups. The annotations combine human-created labels and outputs from the MINGLE pipeline, ensuring semantic richness and broad coverage of real world scenarios.

AAAI Conference 2026 Conference Paper

Remodeling Semantic Relationships in Vision-Language Fine-Tuning

• Xiangyang Wu
• Liu Liu
• Baosheng Yu
• Jiayan Qiu
• Zhenwei Shi

Vision-language fine-tuning has emerged as an efficient paradigm for constructing multimodal foundation models. While textual context often highlights semantic relationships within an image, existing fine-tuning methods typically overlook this information when aligning vision and language, thus leading to suboptimal performance. Toward solving this problem, we propose a method that can improve multimodal alignment and fusion based on both semantics and relationships.Specifically, we first extract multilevel semantic features from different vision encoder to capture more visual cues of the relationships. Then, we learn to project the vision features to group related semantics, among which are more likely to have relationships. Finally, we fuse the visual features with the textual by using inheritable cross-attention, where we globally remove the redundant visual relationships by discarding visual-language feature pairs with low correlation. We evaluate our proposed method on eight foundation models and two downstream tasks, visual question answering and image captioning, and show that it outperforms all existing methods.

AAAI Conference 2026 Conference Paper

SIAM: Towards Generalizable Articulated Object Modeling via Single Robot-Object Interaction

• Yuyan Liu
• Li Zhang
• Di Wu
• Yan Zhang
• Anran Huang
• Zhi Wang
• Liu Liu
• Dan Guo

Articulated object modeling, which represents interconnected rigid bodies with their geometry, part segmentation, articulation tree, and physical properties, is crucial for robotic perception and manipulation. Recently existing methods like SAGCI leverage Interactive Perception (IP) to refine models through robot interaction. However, SAGCI suffers from prior-dependency (requiring initialization), neglects kinematic/dynamic constraints, and generates non-watertight meshes. To overcome these limitations, we propose SIAM, a novel framework for efficient and generalizable Single-Interaction Articulated Modeling. Given an initial point cloud, SIAM first enables minimal robot interaction to trigger object motion. It then precisely segments parts by analyzing point cloud differences pre- and post-interaction. For joint parameter estimation, we introduce an optimization incorporating novel kinematic energy constraints, enhancing physical consistency. Finally, we reconstruct a high-quality, topologically watertight mesh by learning 3D Gaussian Primitives from multi-view RGB-D observations under deformation. Extensive experiments on the PartNet-Mobility benchmark demonstrate state-of-the-art articulation modeling performance. Successful real-world deployment with an xArm robot further validates the framework's practicality and transferability. SIAM achieves accurate, prior-free modeling with significantly reduced interaction cost.

EAAI Journal 2025 Journal Article

Advancing the safety of intelligent rail transit systems: A segmentation network for efficient end-of-track degradation feature extraction

• Tao Ye
• Haoran Chen
• Guopeng Liu
• Liu Liu
• Hongbin Ren
• Xiaosong Li
• Xi Zhang

Accurate segmentation of rail track is crucial for the safe autonomous driving of intelligent trains. Current train operations struggle with insufficient precision in rail track segmentation, primarily due to poor end-of-track segmentation performance caused by degradation of track-end features. To address these challenges, we propose Rail Track End Wise Network (RTEW-Net), an effective rail track end wise segmentation method. This network utilizes Full-Transformer Module (FTM) for effective track feature extraction and integrates the Global Response Normalization (GRN) module to handle drastic lighting changes. Additionally, we designed the Wise Weigh Maintain (WWM) method to enhance feature learning and retain track features. To validate its effectiveness, we constructed the RailMixed2024 (RM2024) dataset. Our model achieves high-precision global rail track segmentation and optimizes end detection. Experimental results demonstrate that RTEW-Net exhibits outstanding performance on the RM2024 and RailSem19 datasets, establishing it as the state-of-the-art (SOTA) in this field.

EAAI Journal 2025 Journal Article

Erratum to “Advancing the safety of intelligent rail transit systems: A segmentation network for efficient end-of-track degradation feature extraction” [Eng. Appl. Artif. Intellig. 159 (2025) 111527]

• Tao Ye
• Haoran Chen
• Guopeng Liu
• Liu Liu
• Hongbin Ren
• Xiaosong Li
• Xi Zhang

IROS Conference 2025 Conference Paper

GeoFlow-SLAM: A Robust Tightly-Coupled RGBD-Inertial and Legged Odometry Fusion SLAM for Dynamic Legged Robotics

• Tingyang Xiao
• Xiaolin Zhou
• Liu Liu
• Wei Sui
• Wei Feng
• Jiaxiong Qiu
• Xinjie Wang
• Zhizhong Su

This paper presents GeoFlow-SLAM, a robust and effective Tightly-Coupled RGBD-Inertial and Legged Odometry Fusion SLAM for legged robotics undergoing aggressive and high-frequency motions. By integrating geometric consistency, legged odometry constraints, and dual-stream optical flow (GeoFlow), our method addresses three critical challenges: feature matching and pose initialization failures during fast locomotion and visual feature scarcity in texture-less scenes. Specifically, in rapid motion scenarios, feature matching is notably enhanced by leveraging dual-stream optical flow, which combines prior map points and poses. Additionally, we propose a robust pose initialization method for fast locomotion and IMU error in legged robots, integrating IMU/Legged odometry, inter-frame Perspective-n-Point (PnP), and Generalized Iterative Closest Point (GICP). Furthermore, a novel optimization framework that tightly couples depth-to-map and GICP geometric constraints is first introduced to improve the robustness and accuracy in long-duration, visually texture-less environments. The proposed algorithms achieve state-of-the-art (SOTA) on collected legged robots and open-source datasets. To further promote research and development, the open-source datasets and code will be made publicly available at https://github.com/HorizonRobotics/GeoFlowSlam.

AAAI Conference 2025 Conference Paper

HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting

• Feng Shibo
• Peilin Zhao
• Liu Liu
• Pengcheng Wu
• Zhiqi Shen

Generative models have gained significant attention in multivariate time series forecasting (MTS), particularly due to their ability to generate high-fidelity samples. Forecasting the probability distribution of multivariate time series is a challenging yet practical task. Although some recent attempts have been made to handle this task, two major challenges persist: 1) some existing generative methods underperform in high-dimensional multivariate time series forecasting, which is hard to scale to higher dimensions; 2) The inherent high-dimensional multivariate attributes constrain the forecasting lengths of existing generative models. In this paper, we point out that discrete token representations can model high-dimensional MTS with faster inference time, and forecast the target with the long-term trends of itself can extend the forecasting length with high accuracy. Motivated by this, we propose a vector quantized framework called Hierarchical Discrete Transformer (HDT) that models time series into discrete token representations with l2 normalization enhanced vector quantized strategy, in which we transform the MTS forecasting into discrete tokens generation. To address the limitations of generative models in long-term forecasting, we propose a hierarchical discrete Transformer. This model captures the discrete long-term trend of the target at the low level and leverages this trend as a condition to generate the discrete representation of the target at the high level that introduces the features of target itself for extending the forecasting length in high-dimensional MTS. Extensive experiments on five popular MTS datasets verify the effectiveness of our proposed method. The source code will be released.

IJCAI Conference 2025 Conference Paper

Injecting Imbalance Sensitivity for Multi-Task Learning

• Zhipeng Zhou
• Liu Liu
• Peilin Zhao
• Wei Gong

Multi-task learning (MTL) has emerged as a promising approach for deploying deep learning models in real-life applications. Recent studies have proposed optimization-based learning paradigms to establish task-shared representations in MTL. However, our paper empirically argues that these studies, specifically gradient-based ones, primarily emphasize the conflict issue while neglecting the potentially more significant impact of imbalance/dominance in MTL. In line with this perspective, we enhance the existing baseline method by injecting imbalance-sensitivity through the imposition of constraints on the projected norms. To demonstrate the effectiveness of our proposed IMbalance-sensitive Gradient (IMGrad) descent method, we evaluate it on multiple mainstream MTL benchmarks, encompassing supervised learning tasks as well as reinforcement learning. The experimental results consistently demonstrate competitive performance.

NeurIPS Conference 2025 Conference Paper

Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDP

• Yuxin Pan
• Zhiguang Cao
• Chengyang GU
• Liu Liu
• Peilin Zhao
• Yize Chen
• Fangzhen Lin

Existing neural methods for multi-task vehicle routing problems (VRPs) typically learn unified solvers to handle multiple constraints simultaneously. However, they often underutilize the compositional structure of VRP variants, each derivable from a common set of basis VRP variants. This critical oversight causes unified solvers to miss out the potential benefits of basis solvers, each specialized for a basis VRP variant. To overcome this limitation, we propose a framework that enables unified solvers to perceive the shared-component nature across VRP variants by proactively reusing basis solvers, while mitigating the exponential growth of trained neural solvers. Specifically, we introduce a State-Decomposable MDP (SDMDP) that reformulates VRPs by expressing the state space as the Cartesian product of basis state spaces associated with basis VRP variants. More crucially, this formulation inherently yields the optimal basis policy for each basis VRP variant. Furthermore, a Latent Space-based SDMDP extension is developed by incorporating both the optimal basis policies and a learnable mixture function to enable the policy reuse in the latent space. Under mild assumptions, this extension provably recovers the optimal unified policy of SDMDP through the mixture function that computes the state embedding as a mapping from the basis state embeddings generated by optimal basis policies. For practical implementation, we introduce the Mixture-of-Specialized-Experts Solver (MoSES), which realizes basis policies through specialized Low-Rank Adaptation (LoRA) experts, and implements the mixture function via an adaptive gating mechanism. Extensive experiments conducted across VRP variants showcase the superiority of MoSES over prior methods.

IJCAI Conference 2025 Conference Paper

Pre-defined Keypoints Promote Category-level Articulation Pose Estimation via Multi-Modal Alignment

• Wenbo Xu
• Li Zhang
• Liu Liu
• Yan Zhong
• Haonan Jiang
• Xue Wang
• Rujing Wang

Articulations are essential in everyday interactions, yet traditional RGB-based pose estimation methods often struggle with issues such as lighting variations and shadows. To overcome these challenges, we propose a novel Pre-defined keypoint based framework for category-level articulation pose estimation via multi-modal Alignment, coined PAGE. Specifically, we first propose a customized keypoint estimation method, aiming to avoid the divergent distance pattern between heuristically generated keypoints and visible points. In addition, to reduce the mutual information redundancy between point clouds and RGB images, we design the geometry-color alignment, which fuses the features after aligning two modalities. This is followed by decoding the radius for each visible point, and applying our proposal integration scoring strategy to predict keypoints. Ultimately, the framework outputs the per-part 6D pose of the articulation. We conduct extensive experiments to evaluate PAGE across a variety of datasets, from synthetic to real-world scenarios, demonstrating its robustness and superior performance.

AAAI Conference 2025 Conference Paper

R^2-Art: Category-Level Articulation Pose Estimation from Single RGB Image via Cascade Render Strategy

• Li Zhang
• Haonan Jiang
• Yukang Huo
• Yan Zhong
• Jianan Wang
• Xue Wang
• Rujing Wang
• Liu Liu

Human life is filled with articulated objects. Previous works for estimating the pose of category-level articulated objects rely on costly 3D point clouds or RGB-D images. In this paper, our goal is to estimate category-level articulation poses from a single RGB image, where we propose R2-Art, a novel category-level Articulation pose estimation framework from a single RGB image and a cascade Render strategy. Given an RGB image as input, R2-Art estimates per-part 6D pose for the articulation. Specifically, we design parallel regression branches tailored to generate camera-to-root translation and rotation. Using the predicted joint states, we perform PC prior transformation and deformation with a joint-centric modeling approach. For further refinement, a cascade render strategy is proposed for projecting the 3D deformed prior onto the 2D mask. Extensive experiments are provided to validate our R2-Art on various datasets ranging from synthetic datasets to real-world scenarios, demonstrating the superior performance and robustness of the R2-Art. We believe that this work has the potential to be applied in many fields including robotics, embodied intelligence, and augmented reality.

NeurIPS Conference 2025 Conference Paper

REArtGS: Reconstructing and Generating Articulated Objects via 3D Gaussian Splatting with Geometric and Motion Constraints

• Di Wu
• Liu Liu
• Zhou Linli
• Anran Huang
• Liangtu Song
• Qiaojun Yu
• Qi Wu
• Cewu Lu

Articulated objects, as prevalent entities in human life, their 3D representations play crucial roles across various applications. However, achieving both high-fidelity textured surface reconstruction and dynamic generation for articulated objects remains challenging for existing methods. In this paper, we present REArtGS, a novel framework that introduces additional geometric and motion constraints to 3D Gaussian primitives, enabling realistic surface reconstruction and generation for articulated objects. Specifically, given multi-view RGB images of arbitrary two states of articulated objects, we first introduce an unbiased Signed Distance Field (SDF) guidance to regularize Gaussian opacity fields, enhancing geometry constraints and improving surface reconstruction quality. Then we establish deformable fields for 3D Gaussians constrained by the kinematic structures of articulated objects, achieving unsupervised generation of surface meshes in unseen states. Extensive experiments on both synthetic and real datasets demonstrate our approach achieves high-quality textured surface reconstruction for given states, and enables high-fidelity surface generation for unseen states. Project site: https: //sites. google. com/view/reartgs/home.

NeurIPS Conference 2025 Conference Paper

SIGMA: Refining Large Language Model Reasoning via Sibling-Guided Monte Carlo Augmentation

• Yanwei Ren
• Haotian Zhang
• Fuxiang Wu
• Jiayan Qiu
• Jiaxing Huang
• Baosheng Yu
• Liu Liu

Enhancing large language models by simply scaling up datasets has begun to yield diminishing returns, shifting the spotlight to data quality. Monte Carlo Tree Search (MCTS) has emerged as a powerful technique for generating high-quality chain-of-thought data, yet conventional approaches typically retain only the top-scoring trajectory from the search tree, discarding sibling nodes that often contain valuable partial insights, recurrent error patterns, and alternative reasoning strategies. This unconditional rejection of non-optimal reasoning branches may waste vast amounts of informative data in the whole search tree. We propose SIGMA (Sibling Guided Monte Carlo Augmentation), a novel framework that reintegrates these discarded sibling nodes to refine LLM reasoning. SIGMA forges semantic links among sibling nodes along each search path and applies a two-stage refinement: a critique model identifies overlooked strengths and weaknesses across the sibling set, and a revision model conducts text-based backpropagation to refine the top-scoring trajectory in light of this comparative feedback. By recovering and amplifying the underutilized but valuable signals from non-optimal reasoning branches, SIGMA substantially improves reasoning trajectories. On the challenging MATH benchmark, our SIGMA-tuned 7B model achieves 54. 92\% accuracy using only 30K samples, outperforming state-of-the-art models trained on 590K samples. This result highlights that our sibling-guided optimization not only significantly reduces data usage but also significantly boosts LLM reasoning.

NeurIPS Conference 2025 Conference Paper

Storyboard-guided Alignment for Fine-grained Video Action Recognition

• Enqi Liu
• Liyuan Pan
• Yan Yang
• Yiran Zhong
• Zhijing Wu
• Xinxiao Wu
• Liu Liu

Fine-grained video action recognition can be formulated as a video–text matching problem. Previous approaches primarily rely on global video semantics to consolidate video embeddings, often leading to misaligned video–text pairs due to inaccurate atomic-level action understanding. This inaccuracy arises due to i) videos with distinct global semantics may share similar atomic actions or visual appearances, and ii) atomic actions can be momentary, gradual, or not directly aligned with overarching video semantics. Inspired by storyboarding, where a script is segmented into individual shots, we propose a multi-granularity framework, SFAR. SFAR generates fine-grained descriptions of common atomic actions for each global semantic using a large language model. Unlike existing works that refine global semantics with auxiliary video frames, SFAR introduces a filtering metric to ensure correspondence between the descriptions and the global semantics, eliminating the need for direct video involvement and thereby enabling more nuanced recognition of subtle actions. By leveraging both global semantics and fine-grained descriptions, our SFAR effectively identifies prominent frames within videos, thereby improving the accuracy of embedding aggregation. Extensive experiments on various video action recognition datasets demonstrate the competitive performance of our SFAR in supervised, few-shot, and zero-shot settings.

NeurIPS Conference 2024 Conference Paper

EfficientCAPER: An End-to-End Framework for Fast and Robust Category-Level Articulated Object Pose Estimation

• Xinyi Yu
• Haonan Jiang
• Li Zhang
• Lin Y. Wu
• Linin Ou
• Liu Liu

Human life is populated with articulated objects. Pose estimation for category-level articulated objects is a significant challenge due to their inherent complexity and diverse kinematic structures. Current methods for this task usually meet the problems of insufficient consideration of kinematic constraints, self-occlusion, and optimization requirements. In this paper, we propose EfficientCAPER, an end-to-end Category-level Articulated object Pose EstimatoR, eliminating the need for optimization functions as post-processing and utilizing the kinematic structure for joint-centric pose modeling, thus enhancing the efficiency and applicability. Given a partial point cloud as input, the EfficientCAPER firstly estimates the pose for the free part of an articulated object using decoupled rotation representation. Next, we canonicalize the input point cloud to estimate constrained parts' poses by predicting the joint parameters and states as replacements. Evaluations on three diverse datasets, ArtImage, ReArtMix, and RobotArm, show EfficientCAPER's effectiveness and generalization ability to real-world scenarios. The framework exhibits excellent static pose estimation performance for articulated objects, contributing to the advancement of category-level pose estimation. Codes will be made publicly available.

AAAI Conference 2024 Conference Paper

KPA-Tracker: Towards Robust and Real-Time Category-Level Articulated Object 6D Pose Tracking

• Liu Liu
• Anran Huang
• Qi Wu
• Dan Guo
• Xun Yang
• Meng Wang

Our life is populated with articulated objects. Current category-level articulation estimation works largely focus on predicting part-level 6D poses on static point cloud observations. In this paper, we tackle the problem of category-level online robust and real-time 6D pose tracking of articulated objects, where we propose KPA-Tracker, a novel 3D KeyPoint based Articulated object pose Tracker. Given an RGB-D image or a partial point cloud at the current frame as well as the estimated per-part 6D poses from the last frame, our KPA-Tracker can effectively update the poses with learned 3D keypoints between the adjacent frames. Specifically, we first canonicalize the input point cloud and formulate the pose tracking as an inter-frame pose increment estimation task. To learn consistent and separate 3D keypoints for every rigid part, we build KPA-Gen that outputs the high-quality ordered 3D keypoints in an unsupervised manner. During pose tracking on the whole video, we further propose a keypoint-based articulation tracking algorithm that mines keyframes as reference for accurate pose updating. We provide extensive experiments on validating our KPA-Tracker on various datasets ranging from synthetic point cloud observation to real-world scenarios, which demonstrates the superior performance and robustness of the KPA-Tracker. We believe that our work has the potential to be applied in many fields including robotics, embodied intelligence and augmented reality. All the datasets and codes are available at https://github.com/hhhhhar/KPA-Tracker.

EAAI Journal 2024 Journal Article

Landslide displacement prediction with step-like curve based on convolutional neural network coupled with bi-directional gated recurrent unit optimized by attention mechanism

• Shaoqiang Meng
• Zhenming Shi
• Ming Peng
• Gang Li
• Hongchao Zheng
• Liu Liu
• Limin Zhang

This study aims to accurately predict landslide displacement characterized by a step-like displacement curve, resulting from complex interactions among multiple factors, including periodic and variable elements. Specifically, we propose a framework based on a Convolutional Neural Network (CNN) and optimized Bidirectional Gated Recurrent Unit (BiGRU) with an Attention mechanism, designed to forecast landslide displacement with a step-like curve. Initially, landslide displacements are decomposed into periodic and trend terms utilizing complementary ensemble empirical mode decomposition with adaptive noise (CEEMDAN) algorithm. Subsequently, a CNN layer is constructed to extract intricate high-dimensional features, while a BiGRU layer is established to capture temporal dependencies within historical sequences. Notably, an attention mechanism module is introduced to enhance the impact of key information within rainfall, water level, and historical displacement data. Finally, the nonlinear weighted Huber loss function (NLWHL) evaluation method is introduced to assess the accuracy of the model in predicting mutation states. The results demonstrate that the proposed framework exhibited superior accuracy in predicting landslide displacement in comparison to alternative intelligent algorithms. Regarding total displacement, the CNN-BiGRU-Attention model demonstrates superior predictive capabilities, reducing RMSE values by 12. 52%, 14. 15%, 15. 58%, 21. 68%, 21. 84%, 32. 81%, 58. 10%, and 53. 81% in the test set compared to CNN-BiGRU, CNN-BiLSTM, BiGRU-Attention, SMA-GRU, SMA-LSTM, SMA-SVM, GRU, and LSTM, respectively. Furthermore, the introduction of NLWHL underscores the remarkable accuracy of the proposed framework in forecasting both creep and mutation states. Assessing the displacement mutation state provides the necessary opportunity for early detection and intervention.

NeurIPS Conference 2024 Conference Paper

Rethinking 3D Convolution in $\ell_p$-norm Space

• Li Zhang
• Yan Zhong
• Jianan Wang
• Zhe Min
• Rujing Wang
• Liu Liu

Convolution is a fundamental operation in the 3D backbone. However, under certain conditions, the feature extraction ability of traditional convolution methods may be weakened. In this paper, we introduce a new convolution method based on $\ell_p$-norm. For theoretical support, we prove the universal approximation theorem for $\ell_p$-norm based convolution, and analyze the robustness and feasibility of $\ell_p$-norms in 3D point cloud tasks. Concretely, $\ell_{\infty}$-norm based convolution is prone to feature loss. $\ell_2$-norm based convolution is essentially a linear transformation of the traditional convolution. $\ell_1$-norm based convolution is an economical and effective feature extractor. We propose customized optimization strategies to accelerate the training process of $\ell_1$-norm based Nets and enhance the performance. Besides, a theoretical guarantee is given for the convergence by \textit{regret} argument. We apply our methods to classic networks and conduct related experiments. Experimental results indicate that our approach exhibits competitive performance with traditional CNNs, with lower energy consumption and instruction latency.

TMLR Journal 2023 Journal Article

Dynamics Adapted Imitation Learning

• Zixuan Liu
• Liu Liu
• Bingzhe Wu
• Lanqing Li
• Xueqian Wang
• Bo Yuan
• Peilin Zhao

We consider Imitation Learning with dynamics variation between the expert demonstration (source domain) and the environment (target domain). Based on the popular framework of Adversarial Imitation Learning, we propose a novel algorithm – Dynamics Adapted Imitation Learning (DYNAIL), which incorporates the dynamics variation into the state-action occupancy measure matching as a regularization term. The dynamics variation is modeled by a pair of classifiers to distinguish between source dynamics and target dynamics. Theoretically, we provide an upper bound on the divergence between the learned policy and expert demonstrations in the source domain. Our error bound only depends on the expectation of the discrepancy between the source and target dynamics for the optimal policy in the target domain. The experiment evaluation validates that our method achieves superior results on high dimensional continuous control tasks, compared to existing imitation learning methods

AAAI Conference 2023 Conference Paper

Reject Decoding via Language-Vision Models for Text-to-Image Synthesis

• Fuxiang Wu
• Liu Liu
• Fusheng Hao
• Fengxiang He
• Lei Wang
• Jun Cheng

Transformer-based text-to-image synthesis generates images from abstractive textual conditions and achieves prompt results. Since transformer-based models predict visual tokens step by step in testing, where the early error is hard to be corrected and would be propagated. To alleviate this issue, the common practice is drawing multi-paths from the transformer-based models and re-ranking the multi-images decoded from multi-paths to find the best one and filter out others. Therefore, the computing procedure of excluding images may be inefficient. To improve the effectiveness and efficiency of decoding, we exploit a reject decoding algorithm with tiny multi-modal models to enlarge the searching space and exclude the useless paths as early as possible. Specifically, we build tiny multi-modal models to evaluate the similarities between the partial paths and the caption at multi scales. Then, we propose a reject decoding algorithm to exclude some lowest quality partial paths at the inner steps. Thus, under the same computing load as the original decoding, we could search across more multi-paths to improve the decoding efficiency and synthesizing quality. The experiments conducted on the MS-COCO dataset and large-scale datasets show that the proposed reject decoding algorithm can exclude the useless paths and enlarge the searching paths to improve the synthesizing quality by consuming less time.

IROS Conference 2023 Conference Paper

Visual Localization Based on Multiple Maps

• Yukai Lin
• Liu Liu
• Xiao Liang
• Jiangwei Li

This paper proposes a multi-map based visual localization method for image sequences. Given multiple single-map based localization results, we combine them with SLAM to estimate robust and accurate camera poses under challenging conditions. Our method comprises three modules connected in a sequence. First, we reconstruct multiple reference maps using the Structure-from-Motion technique, one map for each reference sequence. A single-image-based localization pipeline is performed to estimate 6-DoF camera poses for each query image, one for each map. Second, a consensus set maximization module is proposed to select the best camera poses from multi-map poses, estimating one 6-DoF camera pose for each query image. Finally, a robust pose refinement module is proposed to optimize 6-DoF camera poses of query images, combining map-based localization and local SLAM information. Experiments show that the proposed pipeline achieves state-of-the-art performance on challenging map-based localization benchmarks. Demonstrating the broad applicability of our method, we obtained first place in the challenge of Map-Based Localization for Autonomous Driving at ECCV2022.

AAAI Conference 2022 Conference Paper

Channelized Axial Attention – considering Channel Relation within Spatial Attention for Semantic Segmentation

• Ye Huang
• Di Kang
• Wenjing Jia
• Liu Liu
• Xiangjian He

Spatial and channel attentions, modelling the semantic interdependencies in spatial and channel dimensions respectively, have recently been widely used for semantic segmentation. However, computing spatial and channel attentions separately sometimes causes errors, especially for those difficult cases. In this paper, we propose Channelized Axial Attention (CAA) to seamlessly integrate channel attention and spatial attention into a single operation with negligible computation overhead. Specifically, we break down the dot-product operation of the spatial attention into two parts and insert channel relation in between, allowing for independently optimized channel attention on each spatial location. We further develop grouped vectorization, which allows our model to run with very little memory consumption without slowing down the running speed. Comparative experiments conducted on multiple benchmark datasets, including Cityscapes, PAS- CAL Context, and COCO-Stuff, demonstrate that our CAA outperforms many state-of-the-art segmentation models (including dual attention) on all tested datasets.

AAAI Conference 2022 Conference Paper

Continual Learning through Retrieval and Imagination

• Zhen Wang
• Liu Liu
• Yiqun Duan
• Dacheng Tao

Continual learning is an intellectual ability of artificial agents to learn new streaming labels from sequential data. The main impediment to continual learning is catastrophic forgetting, a severe performance degradation on previously learned tasks. Although simply replaying all previous data or continuously adding the model parameters could alleviate the issue, it is impractical in real-world applications due to the limited available resources. Inspired by the mechanism of the human brain to deepen its past impression, we propose a novel framework, Deep Retrieval and Imagination (DRI), which consists of two components: 1) an embedding network that constructs a unified embedding space without adding model parameters on the arrival of new tasks; and 2) a generative model to produce additional (imaginary) data based on the limited memory. By retrieving the past experiences and corresponding imaginary data, DRI distills knowledge and rebalances the embedding space to further mitigate forgetting. Theoretical analysis demonstrates that DRI can reduce the loss approximation error and improve the robustness through retrieval and imagination, bringing better generalizability to the network. Extensive experiments show that DRI performs significantly better than the existing state-of-the-art continual learning methods and effectively alleviates catastrophic forgetting.

NeurIPS Conference 2022 Conference Paper

Escaping from the Barren Plateau via Gaussian Initializations in Deep Variational Quantum Circuits

• Kaining Zhang
• Liu Liu
• Min-Hsiu Hsieh
• Dacheng Tao

Variational quantum circuits have been widely employed in quantum simulation and quantum machine learning in recent years. However, quantum circuits with random structures have poor trainability due to the exponentially vanishing gradient with respect to the circuit depth and the qubit number. This result leads to a general standpoint that deep quantum circuits would not be feasible for practical tasks. In this work, we propose an initialization strategy with theoretical guarantees for the vanishing gradient problem in general deep quantum circuits. Specifically, we prove that under proper Gaussian initialized parameters, the norm of the gradient decays at most polynomially when the qubit number and the circuit depth increase. Our theoretical results hold for both the local and the global observable cases, where the latter was believed to have vanishing gradients even for very shallow circuits. Experimental results verify our theoretical findings in quantum simulation and quantum chemistry.

AAAI Conference 2022 Conference Paper

Resistance Training Using Prior Bias: Toward Unbiased Scene Graph Generation

• Chao Chen
• Yibing Zhan
• Baosheng Yu
• Liu Liu
• Yong Luo
• Bo Du

Scene Graph Generation (SGG) aims to build a structured representation of a scene using objects and pairwise relationships, which benefits downstream tasks. However, current SGG methods usually suffer from sub-optimal scene graph generation because of the long-tailed distribution of training data. To address this problem, we propose Resistance Training using Prior Bias (RTPB) for the scene graph generation. Specifically, RTPB uses a distributed-based prior bias to improve models’ detecting ability on less frequent relationships during training, thus improving the model generalizability on tail categories. In addition, to further explore the contextual information of objects and relationships, we design a contextual encoding backbone network, termed as Dual Transformer (DTrans). We perform extensive experiments on a very popular benchmark, VG150, to demonstrate the effectiveness of our method for the unbiased scene graph generation. In specific, our RTPB achieves an improvement of over 10% under the mean recall when applied to current SGG methods. Furthermore, DTrans with RTPB outperforms nearly all stateof-the-art methods with a large margin. Code is available at https: //github. com/ChCh1999/RTPB

AAAI Conference 2022 Conference Paper

Transcribing Natural Languages for the Deaf via Neural Editing Programs

• Dongxu Li
• Chenchen Xu
• Liu Liu
• Yiran Zhong
• Rong Wang
• Lars Petersson
• Hongdong Li

This work studies the task of glossification, of which the aim is to transcribe natural spoken language sentences for the Deaf (hard-of-hearing) community to ordered sign language glosses. Previous sequence-to-sequence language models trained with paired sentence-gloss data often fail to capture the rich connections between the two distinct languages, leading to unsatisfactory transcriptions. We observe that despite different grammars, glosses effectively simplify sentences for the ease of deaf communication, while sharing a large portion of vocabulary with sentences. This has motivated us to implement glossification by executing a collection of editing actions, e. g. word addition, deletion and copying, called editing programs, on their natural spoken language counterparts. Specifically, we design a new neural agent that learns to synthesize and execute editing programs, conditioned on sentence contexts and partial editing results. The agent is trained to imitate minimal editing programs, while exploring more widely the program space via policy gradients to optimize sequence-wise transcription quality. Results show that our approach outperforms previous glossification models by a large margin, improving the BLEU-4 score from 16. 45 to 18. 89 on RWTH-PHOENIX- WEATHER-2014T and from 18. 38 to 21. 30 on CSL-Daily.

NeurIPS Conference 2022 Conference Paper

UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup

• Zongbo Han
• Zhipeng Liang
• Fan Yang
• Liu Liu
• Lanqing Li
• Yatao Bian
• Peilin Zhao
• Bingzhe Wu

Subpopulation shift widely exists in many real-world machine learning applications, referring to the training and test distributions containing the same subpopulation groups but varying in subpopulation frequencies. Importance reweighting is a normal way to handle the subpopulation shift issue by imposing constant or adaptive sampling weights on each sample in the training dataset. However, some recent studies have recognized that most of these approaches fail to improve the performance over empirical risk minimization especially when applied to over-parameterized neural networks. In this work, we propose a simple yet practical framework, called uncertainty-aware mixup (UMIX), to mitigate the overfitting issue in over-parameterized models by reweighting the ''mixed'' samples according to the sample uncertainty. The training-trajectories-based uncertainty estimation is equipped in the proposed UMIX for each sample to flexibly characterize the subpopulation distribution. We also provide insightful theoretical analysis to verify that UMIX achieves better generalization bounds over prior works. Further, we conduct extensive empirical studies across a wide range of tasks to validate the effectiveness of our method both qualitatively and quantitatively. Code is available at https: //github. com/TencentAILabHealthcare/UMIX.

AAAI Conference 2021 Conference Paper

Activity Image-to-Video Retrieval by Disentangling Appearance and Motion

• Liu Liu
• Jiangtong Li
• Li Niu
• Ruicong Xu
• Liqing Zhang

With the rapid emergence of video data, image-to-video retrieval has attracted much attention. There are two types of image-to-video retrieval: instance-based and activity-based. The former task aims to retrieve videos containing the same main objects as the query image, while the latter focuses on finding the similar activity. Since dynamic information plays a significant role in the video, we pay attention to the latter task to explore the motion relation between images and videos. In this paper, we propose a Motion-assisted Activity Proposal-based Image-to-Video Retrieval (MAP-IVR) approach to disentangle the video features into motion features and appearance features and obtain appearance features from the images. Then, we perform image-to-video translation to improve the disentanglement quality. The retrieval is performed in both appearance and video feature spaces. Extensive experiments demonstrate that our MAP-IVR approach remarkably outperforms the state-of-the-art approaches on two benchmark activity-based video datasets.

NeurIPS Conference 2021 Conference Paper

Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels

• Sheng Wan
• Yibing Zhan
• Liu Liu
• Baosheng Yu
• Shirui Pan
• Chen Gong

Graph Neural Networks (GNNs) have achieved remarkable performance in the task of semi-supervised node classification. However, most existing GNN models require sufficient labeled data for effective network training. Their performance can be seriously degraded when labels are extremely limited. To address this issue, we propose a new framework termed Contrastive Graph Poisson Networks (CGPN) for node classification under extremely limited labeled data. Specifically, our CGPN derives from variational inference; integrates a newly designed Graph Poisson Network (GPN) to effectively propagate the limited labels to the entire graph and a normal GNN, such as Graph Attention Network, that flexibly guides the propagation of GPN; applies a contrastive objective to further exploit the supervision information from the learning process of GPN and GNN models. Essentially, our CGPN can enhance the learning performance of GNNs under extremely limited labels by contrastively propagating the limited labels to the entire graph. We conducted extensive experiments on different types of datasets to demonstrate the superiority of CGPN.

AAAI Conference 2021 Short Paper

Multi-label Few-shot Learning with Semantic Inference (Student Abstract)

• Zhen Wang
• Yiqun Duan
• Liu Liu
• Dacheng Tao

Few-shot learning can adapt the classification model to new labels with only a few labeled examples. Previous studies mainly focused on the scenario of a single category label per example but have not effectively solved the more challenging multi-label scenario, which has exponential-sized output space and low-data. In this paper, we propose a semanticaware meta-learning model for multi-label few-shot learning. Our approach can learn and infer the semantic correlation between unseen labels and historical labels to quickly adapt multi-label tasks based on only a few examples. Specifically, features can be mapped into the semantic space via label embeddings to exploit the label correlation, thus structuring the overwhelming output space. We design a novel semantic inference mechanism for leveraging prior knowledge learned from historical labels, which will produce good generalization performance on new labels to alleviate the overfitting caused by low-data. Finally, empirical results show that the proposed method significantly outperforms the existing stateof-the-art methods on the multi-label few-shot learning tasks.

NeurIPS Conference 2021 Conference Paper

Weak-shot Fine-grained Classification via Similarity Transfer

• Junjie Chen
• Li Niu
• Liu Liu
• Liqing Zhang

Recognizing fine-grained categories remains a challenging task, due to the subtle distinctions among different subordinate categories, which results in the need of abundant annotated samples. To alleviate the data-hungry problem, we consider the problem of learning novel categories from web data with the support of a clean set of base categories, which is referred to as weak-shot learning. In this setting, we propose a method called SimTrans to transfer pairwise semantic similarity from base categories to novel categories. Specifically, we firstly train a similarity net on clean data, and then leverage the transferred similarity to denoise web training data using two simple yet effective strategies. In addition, we apply adversarial loss on similarity net to enhance the transferability of similarity. Comprehensive experiments demonstrate the effectiveness of our weak-shot setting and our SimTrans method.

AAAI Conference 2020 Conference Paper

Optimal Feature Transport for Cross-View Image Geo-Localization

• Yujiao Shi
• Xin Yu
• Liu Liu
• Tong Zhang
• Hongdong Li

This paper addresses the problem of cross-view image geolocalization, where the geographic location of a ground-level street-view query image is estimated by matching it against a large scale aerial map (e. g. , a high-resolution satellite image). State-of-the-art deep-learning based methods tackle this problem as deep metric learning which aims to learn global feature representations of the scene seen by the two different views. Despite promising results are obtained by such deep metric learning methods, they, however, fail to exploit a crucial cue relevant for localization, namely, the spatial layout of local features. Moreover, little attention is paid to the obvious domain gap (between aerial view and ground view) in the context of cross-view localization. This paper proposes a novel Cross-View Feature Transport (CVFT) technique to explicitly establish cross-view domain transfer that facilitates feature alignment between ground and aerial images. Specifically, we implement the CVFT as network layers, which transports features from one domain to the other, leading to more meaningful feature similarity comparison. Our model is differentiable and can be learned end-to-end. Experiments on large-scale datasets have demonstrated that our method has remarkably boosted the state-of-the-art cross-view localization performance, e. g. , on the CVUSA dataset, with significant improvements for top-1 recall from 40. 79% to 61. 43%, and for top-10 from 76. 36% to 90. 49%. We expect the key insight of the paper (i. e. , explicitly handling domain difference via domain transport) will prove to be useful for other similar problems in computer vision as well.

NeurIPS Conference 2020 Conference Paper

Robust compressed sensing using generative models

• Ajil Jalal
• Liu Liu
• Alexandros G. Dimakis
• Constantine Caramanis

We consider estimating a high dimensional signal in $\R^n$ using a sublinear number of linear measurements. In analogy to classical compressed sensing, here we assume a generative model as a prior, that is, we assume the signal is represented by a deep generative model $G: \R^k \rightarrow \R^n$. Classical recovery approaches such as empirical risk minimization (ERM) are guaranteed to succeed when the measurement matrix is sub-Gaussian. However, when the measurement matrix and measurements are heavy tailed or have outliers, recovery may fail dramatically. In this paper we propose an algorithm inspired by the Median-of-Means (MOM). Our algorithm guarantees recovery for heavy tailed data, even in the presence of outliers. Theoretically, our results show our novel MOM-based algorithm enjoys the same sample complexity guarantees as ERM under sub-Gaussian assumptions. Our experiments validate both aspects of our claims: other algorithms are indeed fragile and fail under heavy tailed and/or corrupted data, while our approach exhibits the predicted robustness.

NeurIPS Conference 2019 Conference Paper

Spatial-Aware Feature Aggregation for Image based Cross-View Geo-Localization

• Yujiao Shi
• Liu Liu
• Xin Yu
• Hongdong Li

In this paper, we develop a new deep network to explicitly address these inherent differences between ground and aerial views. We observe there exist some approximate domain correspondences between ground and aerial images. Specifically, pixels lying on the same azimuth direction in an aerial image approximately correspond to a vertical image column in the ground view image. Thus, we propose a two-step approach to exploit this prior knowledge. The first step is to apply a regular polar transform to warp an aerial image such that its domain is closer to that of a ground-view panorama. Note that polar transform as a pure geometric transformation is agnostic to scene content, hence cannot bring the two domains into full alignment. Then, we add a subsequent spatial-attention mechanism which further brings corresponding deep features closer in the embedding space. To improve the robustness of feature representation, we introduce a feature aggregation strategy via learning multiple spatial embeddings. By the above two-step approach, we achieve more discriminative deep representations, facilitating cross-view Geo-localization more accurate. Our experiments on standard benchmark datasets show significant performance boosting, achieving more than doubled recall rate compared with the previous state of the art.

AAAI Conference 2018 Conference Paper

Statistical Inference Using SGD

• Tianyang Li
• Liu Liu
• Anastasios Kyrillidis
• Constantine Caramanis

We present a novel method for frequentist statistical inference in M-estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlenbeck process suggests that such averages are asymptotically normal. To show the merits of our scheme, we apply it to both synthetic and real data sets, and demonstrate that its accuracy is comparable to classical statistical methods, while requiring potentially far less computation.

IROS Conference 2014 Conference Paper

Pose estimation in physical human-machine interactions with application to bicycle riding

• Yizhai Zhang
• Kuo Chen
• Jingang Yi
• Liu Liu

Tracking whole-body human pose in physical human-machine interactions such as bicycling is challenging because of highly-dimensional human motions and lack of inexpensive, effective motion sensors in outdoor environment. In this paper, we present a computational scheme to estimate the whole-body pose in human-machine interaction with application to the rider-bicycle system. The estimation scheme is built on the fusions of gyroscopes, accelerometers and force sensors with six Extended Kalman filter designs. The use of physical human-machine interaction constraints further helps to eliminate the integration drifts of inertial sensors measurements and also to reduce the number of the inertial sensors for whole-body pose estimation. For each set of upper- and lower-limb, only one tri-axial gyroscope is needed to accurately obtain the pose information. The performance of the drift-free, reliable estimation scheme is demonstrated through both the indoor and outdoor bicycle riding experiments. The proposed approach can be further extended to other types of physical human-machine interactions.

NeurIPS Conference 2010 Conference Paper

Efficient Relational Learning with Hidden Variable Detection

• Ni Lao
• Jun Zhu
• Liu Liu
• Yandong Liu
• William Cohen

Markov networks (MNs) can incorporate arbitrarily complex features in modeling relational data. However, this flexibility comes at a sharp price of training an exponentially complex model. To address this challenge, we propose a novel relational learning approach, which consists of a restricted class of relational MNs (RMNs) called relation tree-based RMN (treeRMN), and an efficient Hidden Variable Detection algorithm called Contrastive Variable Induction (CVI). On one hand, the restricted treeRMN only considers simple (e. g. , unary and pairwise) features in relational data and thus achieves computational efficiency; and on the other hand, the CVI algorithm efficiently detects hidden variables which can capture long range dependencies. Therefore, the resultant approach is highly efficient yet does not sacrifice its expressive power. Empirical results on four real datasets show that the proposed relational learning method can achieve similar prediction quality as the state-of-the-art approaches, but is significantly more efficient in training; and the induced hidden variables are semantically meaningful and crucial to improve the training speed and prediction qualities of treeRMNs.