Qijun Chen

IROS Conference 2025 Conference Paper

LGPR: Local Feature Learning Brings More Generalizable Visual Place Recognition

• Shuai Su
• Jingwei Yang 0002
• Jiayuan Du
• Xianghui Pan
• Chengju Liu
• Qijun Chen

We propose a Visual Place Recognition (VPR) framework by sharing lightweight keypoint extraction modules for local features. Current research on the joint learning of local keypoint matching and VPR is relatively scarce, and the application deployment of real-time spatial computing on edge devices has a high learning cost. There is also a significant spatial structural difference between existing VPR methods and the scenarios in practical applications. To address these issues, we design a joint learning framework for local keypoint extraction and VPR, which shares local features and fuses irregularly distributed key features in space through self-attention and cross-attention mechanisms. Our framework achieves excellent results on several VPR datasets. In particular, we introduce a new VPR dataset, called TJPark, which has a significant spatial information difference from common street view data. Our method demonstrates that local features with strong generalization capabilities effectively help enhance the generalization of VPR. Our open source code and dataset are available at: https://github.com/ShuaiAlger/LGPR.

IROS Conference 2025 Conference Paper

LSW-Net: A Spatio-temporal Self-Supervised Framework for 2D LiDAR-Based Environment Perception

• Haojie Dai
• Yujie Cui
• Wenbo Shi
• Mazeyu Ji
• Chengju Liu
• Qijun Chen

In the deep learning era, 2D LiDAR perception is often overlooked as research prioritizes 3D point clouds. Yet, 2D LiDAR remains essential for low-cost robotic systems due to its affordability. Despite its simplicity, it faces major challenges-not only in perception but also in learning itself, as data sparsity and limited features hinder effective framework development. Additionally, fundamental structural differences prevent direct adaptation of 3D perception networks. To address these aforementioned concerns, we proposes LSW-Net (Laser Scan Weight-Net), a self-supervised framework for 2D LiDAR perception, enabling end-to-end learning from raw point clouds to adaptive weight extraction. This framework provides a scalable and lightweight solution for 2D LiDAR environmental perception, and its self-supervised nature reduces annotation costs. It includes: (i) a general 2D Laser Encoder (LS-Encoder) that integrates local convolutional perception with global attention perception to extract multi-scale spatio-temporal features; and (ii) an interpretable weight extraction module (Weight Extractor) that dynamically quantifies the contribution of each point in environmental perception tasks through contrastive learning and physical consistency constraints. Evaluated on diverse scenes for point cloud registration and SLAM tasks, LSW-Net outperforms traditional methods in feature discriminability and adaptability. Additionally, we performed ablation experiments to substantiate the rationality of our approach. Our code is available at https://github.com/cuiyujie0113/LSW-NET.

IROS Conference 2025 Conference Paper

Reinforcement Learning-based Optimization of Humanoid Joint Motion Control via Text-driven Human Motion Mapping

• Zihan Xu
• Mengxian Hu
• Kaiyan Xiao
• Qin Fang
• Chengju Liu
• Qijun Chen

Human motion retargeting for humanoid robots, transferring human motion data to robots for imitation, presents significant challenges but offers considerable potential for real-world applications. Traditionally, this process relies on human demonstrations captured through pose estimation or motion capture systems. In this paper, we explore a text-driven approach to obtain imitation motion data more flexibly and simply. To address the inherent discrepancies between the generated motion representations and the kinematic constraints of humanoid robots, we propose an angle signal network based on norm-position and rotation loss (NPR Loss). It generates joint angles, which serve as inputs to a reinforcement learning based whole-body motion control policy. The policy ensures tracking of the generated motions while maintaining the robot’s stability during execution. Our experimental results demonstrate the efficacy of this approach, successfully transferring text-driven human motion to a real humanoid robot NAO.

IROS Conference 2025 Conference Paper

Rotation-Equivariant Robot Vision: A Perspective via Correspondence-Matching and Pre-training

• Shuai Su
• Xianghui Pan
• Jiayuan Du
• Chengju Liu
• Qijun Chen

Correspondence matching is a fundamental and crucial task in robot vision. In recent years, deep learning-based keypoint matching techniques have shown outstanding performance in downstream tasks. Conventional learning-based correspondence matching methods rely on large datasets and a specific training procedure. Correspondence techniques based on pre-trained features have been preliminarily explored by researchers. Unfortunately, traditional convolutional neural networks only possess translation invariance but lack rotational invariance, hence, their performance suffers significantly under heavy rotations. Therefore, we propose a correspondence matching method based on pre-trained group-equivariant neural networks and compare the performance of various rotation-equivariant to rotation-invariant transformers. We conducted experiments on the Rotated-Hpatches and Rotated-MegaDepth datasets, and the results indicate that our proposed method is concise and effective, achieving state-of-the-art performance without the need for retraining in downstream tasks.

ICRA Conference 2024 Conference Paper

Dive Deeper into Rectifying Homography for Stereo Camera Online Self-Calibration

• Hongbo Zhao 0009
• Yikang Zhang 0001
• Qijun Chen
• Rui Fan 0001

Accurate estimation of stereo camera extrinsic parameters is crucial to guarantee the performance of stereo matching algorithms. In prior arts, the online self-calibration of stereo cameras has commonly been formulated as a specialized visual odometry problem, without taking into account the principles of stereo rectification. In this paper, we first delve deeply into the concept of rectifying homography, which serves as the cornerstone for the development of our novel stereo camera online self-calibration algorithm, for cases where only a single pair of images is available. Furthermore, we introduce a simple yet effective solution for global optimum extrinsic parameter estimation in the presence of stereo video sequences. Additionally, we emphasize the impracticality of using three Euler angles and three components in the translation vectors for performance quantification. Instead, we introduce four new evaluation metrics to quantify the robustness and accuracy of extrinsic parameter estimation, applicable to both single-pair and multi-pair cases. Extensive experiments conducted across indoor and outdoor environments using various experimental setups validate the effectiveness of our proposed algorithm. The comprehensive evaluation results demonstrate its superior performance in comparison to the baseline algorithm. Our source code, demo video, and supplement are publicly available at mias. group/StereoCalibrator.

IROS Conference 2024 Conference Paper

DVT: Decoupled Dual-Branch View Transformation for Monocular Bird's Eye View Semantic Segmentation

• Jiayuan Du
• Xianghui Pan
• Mengjiao Shen
• Shuai Su
• Jingwei Yang 0002
• Chengju Liu
• Qijun Chen

Monocular Bird’s Eye View (BEV) semantic segmentation is critical for autonomous driving for its inherent advantages in spatial representation and downstream tasks. However, it is challenging to simultaneously learn view transformation and pixel-wise classification. Previous works suffer from non-flat region distortion, distant depth ambiguity, and visual occlusion. To address these aforementioned concerns, we propose dual-branch view transformation (DVT), a novel framework for monocular BEV semantic segmentation. Our method consists of: (i) A dual-branch view transformation to decouple features into flat region and non-flat region and process them independently. (ii) A depth-aware weighting method to make the model pay more attention to the distant depth. (iii) An auxiliary task to introduce more inductive biases to alleviate the inaccuracy caused by visual occlusion. Furthermore, we design a class-aware weighting method to address the class and size imbalance of datasets. Experimental results on nuScenes and KITTI-360 datasets demonstrate that DVT outperforms previous state-of-the-art (SOTA). Our codes are available at https://github.com/MrPicklesGG/DVT.

IROS Conference 2024 Conference Paper

Enhanced Language-guided Robot Navigation with Panoramic Semantic Depth Perception and Cross-modal Fusion

• Liuyi Wang
• Jiagui Tang
• Zongtao He
• Ronghao Dang
• Chengju Liu
• Qijun Chen

Integrating visual observation with linguistic instruction holds significant promise for enhancing robot navigation across unstructured environments and enriches the human-robot interaction experience. However, while panoramic RGB views furnish robots with extensive environmental visuals, current methods significantly overlook crucial semantic and depth cues. This incomplete representation may lead to misinterpretation or inadequate execution of language instructions, thereby impeding navigation performance and adaptability. In this paper, we introduce SEAT, a semantic-depth aware cross-modal transformer model. Our approach incorporates an efficient panoramic multi-type visual encoder to capture comprehensive environmental details. To mitigate the rigidity of feature mapping stemming from the freezing of pre-training encoders, we propose a novel region query pre-training task. Additionally, we leverage an improved dual-scale cross-modal transformer to facilitate the integration of instructions, topological memory, and action prediction. Extensive experiments on three language-guided robot navigation datasets demonstrate the efficacy of our model, achieving competitive navigation success rates with fewer parameters and computational load. Furthermore, we validate SEAT’s effectiveness in real-world scenarios by deploying it on a mobile robot across various environments. The code is available at https://github.com/CrystalSixone/SEAT.

IROS Conference 2024 Conference Paper

GenerOcc: Self-supervised Framework of Real-time 3D Occupancy Prediction for Monocular Generic Cameras

• Xianghui Pan
• Jiayuan Du
• Shuai Su
• Wenhao Zong
• Xiao Wang
• Chengju Liu
• Qijun Chen

In the context of 3D scene perception tasks, the significance of 3D occupancy prediction has been progressively growing, aiming to forecast the occupancy state of voxels in a discrete 3D space. However, existing methods typically exhibit several limitations, such as restricted adaptability to non-pinhole cameras due to fixed camera parameters, heavy reliance on 3D annotations because of the inability to project 3D output back to the camera plane, and inferior real-time inference performance resulting from the conversion process from 2D to 3D features. To address these constrains, we introduce GenerOcc, a self-supervised framework of real-time 3D occupancy prediction for monocular generic cameras. We have collected the fisheye Dominant dataset to confirm the compatibility of our ray-based camera model with non-pinhole cameras. By transforming the occupancy prediction task into a depth estimation task in a self-supervised manner, we eliminate dependency on 3D annotations. Furthermore, we propose a parametric voxel probability distribution module that leverages 2D features to quickly predict 3D occupancy without 3D representations of the scene. Additionally, our GenerOcc has been extensively evaluated on public pinhole Occ3D-nuScenes dataset and our proprietary fisheye Dominant dataset, both yielding impressive performance.

ICLR Conference 2024 Conference Paper

InstructDET: Diversifying Referring Object Detection with Generalized Instructions

• Ronghao Dang
• Jiangyan Feng
• Haodong Zhang
• Chongjian Ge
• Lin Song 0002
• Lijun Gong
• Chengju Liu
• Qijun Chen

We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.

NeurIPS Conference 2024 Conference Paper

MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer

• Minghao Zhu
• Zhengpu Wang
• Mengxian Hu
• Ronghao Dang
• Xiao Lin
• Xun Zhou
• Chengju Liu
• Qijun Chen

Transferring visual-language knowledge from large-scale foundation models for video recognition has proved to be effective. To bridge the domain gap, additional parametric modules are added to capture the temporal information. However, zero-shot generalization diminishes with the increase in the number of specialized parameters, making existing works a trade-off between zero-shot and close-set performance. In this paper, we present MoTE, a novel framework that enables generalization and specialization to be balanced in one unified model. Our approach tunes a mixture of temporal experts to learn multiple task views with various degrees of data fitting. To maximally preserve the knowledge of each expert, we propose Weight Merging Regularization, which regularizes the merging process of experts in weight space. Additionally with temporal feature modulation to regularize the contribution of temporal feature during test. We achieve a sound balance between zero-shot and close-set video recognition tasks and obtain state-of-the-art or competitive results on various datasets, including Kinetics-400 & 600, UCF, and HMDB. Code is available at https: //github. com/ZMHH-H/MoTE.

IROS Conference 2024 Conference Paper

Multimodal Evolutionary Encoder for Continuous Vision-Language Navigation

• Zongtao He
• Liuyi Wang
• Lu Chen
• Shu Li 0005
• Qingqing Yan
• Chengju Liu
• Qijun Chen

Can multimodal encoder evolve when facing increasingly tough circumstances? Our work investigates this possibility in the context of continuous vision-language navigation (continuous VLN), which aims to navigate robots under linguistic supervision and visual feedback. We propose a multimodal evolutionary encoder (MEE) comprising a unified multimodal encoder architecture and an evolutionary pre-training strategy. The unified multimodal encoder unifies rich modalities, including depth and sub-instruction, to enhance the solid understanding of environments and tasks. It also effectively utilizes monocular observation, reducing the reliance on panoramic vision. The evolutionary pre-training strategy exposes the encoder to increasingly unfamiliar data domains and difficult objectives. The multi-stage adaption helps the encoder establish robust intra- and inter-modality connections and improve its generalization to unfamiliar environments. To achieve such evolution, we collect a large-scale multi-stage dataset with specialized objectives, addressing the absence of suitable continuous VLN pre-training. Evaluation on VLN-CE demonstrates the superiority of MEE over other direct action-predicting methods. Furthermore, we deploy MEE in real scenes using self-developed service robots, showcasing its effectiveness and potential for real-world applications. Our code and dataset are available at https://github.com/RavenKiller/MEE.

EAAI Journal 2024 Journal Article

PASTS: Progress-aware spatio-temporal transformer speaker for vision-and-language navigation

• Liuyi Wang
• Chengju Liu
• Zongtao He
• Shu Li
• Qingqing Yan
• Huiyi Chen
• Qijun Chen

Vision-and-language navigation (VLN) is a crucial but challenging cross-modal navigation task. One powerful technique to enhance the generalization performance in VLN is the use of an independent speaker model to provide pseudo instructions for data augmentation. However, current speaker models based on Long-Short Term Memory (LSTM) lack the ability to attend to features relevant at different locations and time steps. To address this, we propose a novel progress-aware spatio-temporal transformer speaker (PASTS) model that uses the transformer as the core of the network. PASTS uses a spatio-temporal encoder to fuse panoramic representations and encode intermediate connections through steps. Besides, to avoid the misalignment problem that could result in incorrect supervision, a speaker progress monitor (SPM) is proposed to enable the model to estimate the progress of instruction generation and facilitate more fine-grained caption results. Additionally, a multifeature dropout (MFD) strategy is introduced to alleviate overfitting. The proposed PASTS is flexible to be combined with existing VLN models. The experimental results demonstrate that PASTS outperforms previous speaker models and successfully improves the performance of previous VLN models, achieving state-of-the-art performance on the standard Room-to-Room (R2R) dataset.

AAAI Conference 2024 Conference Paper

PICNN: A Pathway towards Interpretable Convolutional Neural Networks

• Wengang Guo
• Jiayi Yang
• Huilin Yin
• Qijun Chen
• Wei Ye

Convolutional Neural Networks (CNNs) have exhibited great performance in discriminative feature learning for complex visual tasks. Besides discrimination power, interpretability is another important yet under-explored property for CNNs. One difficulty in the CNN interpretability is that filters and image classes are entangled. In this paper, we introduce a novel pathway to alleviate the entanglement between filters and image classes. The proposed pathway groups the filters in a late conv-layer of CNN into class-specific clusters. Clusters and classes are in a one-to-one relationship. Specifically, we use the Bernoulli sampling to generate the filter-cluster assignment matrix from a learnable filter-class correspondence matrix. To enable end-to-end optimization, we develop a novel reparameterization trick for handling the non-differentiable Bernoulli sampling. We evaluate the effectiveness of our method on ten widely used network architectures (including nine CNNs and a ViT) and five benchmark datasets. Experimental results have demonstrated that our method PICNN (the combination of standard CNNs with our proposed pathway) exhibits greater interpretability than standard CNNs while achieving higher or comparable discrimination power.

ICRA Conference 2024 Conference Paper

SG-RoadSeg: End-to-End Collision-Free Space Detection Sharing Encoder Representations Jointly Learned via Unsupervised Deep Stereo

• Zhiyuan Wu
• Jiaqi Li
• Yi Feng
• Chengju Liu
• Wei Ye 0001
• Qijun Chen
• Rui Fan 0001

Collision-free space detection is of utmost importance for autonomous robot perception and navigation. State-of-the-art (SoTA) approaches generally extract features from RGB images and an additional source or modality of 3-D information, such as depth or disparity images, using a pair of independent encoders. The extracted features are subsequently fused and decoded to yield semantic predictions of collision-free spaces. Such feature-fusion approaches become infeasible in scenarios, where the sensor for 3-D information acquisition is unavailable, or just when multi-sensor calibration falls short of the necessary precision. To overcome these limitations, this paper introduces a novel end-to-end collision-free space detection network, referred to as SG-RoadSeg, built upon our previous work SNE-RoadSeg. A key contribution of this paper is a strategy for sharing encoder representations that are co-learned through both semantic segmentation and unsupervised stereo matching tasks, enabling the features extracted from RGB images to contain both semantic and spatial geometric information. The unsupervised deep stereo serves as an auxiliary functionality, capable of generating accurate disparity maps that can be used by other perception tasks that require depth-related data. Comprehensive experimental results on the KITTI road and semantics datasets validate the effectiveness of our proposed architecture and encoder representation sharing strategy. SG-RoadSeg also demonstrates superior performance than other SoTA collision-free space detection approaches. Our source code, demo video, and supplement are publicly available at mias. group/SG-RoadSeg.

IROS Conference 2024 Conference Paper

SNF-Feat: Semantic-Guided Negative-Sample-Free Representation Learning for Local Feature Extraction

• Xun Zhou
• Qingqing Yan
• Minghao Zhu
• Mengxian Hu
• Chengju Liu
• Qijun Chen

Local feature extraction constitutes a foundational module crucial for numerous downstream tasks of computer vision. Its primary challenge lies in the generation of discriminative feature representations. Prior methodologies have employed contrastive learning within their pipelines, yet have encountered limitations stemming from inherent conflicts within their training data, including the ambiguity of negative samples and the distortion of positive samples. In this study, we propose a semantic-guided negative-sample-free method for local feature learning, denoted as SNF-Feat. Our framework entails dense patch-level representation learning without reliance on negative samples, aiming to ensure that descriptors derived from transformed views of the same local area exhibit predictive capability towards each other. To assess the impact of positive sample distortion, we harness high-level semantic information to derive point-wise loss weights. Furthermore, we establish a self-supervised feature learning paradigm that extends our utilization of datasets. Experimental results demonstrate the superior performance of our method across a range of typical datasets and tasks in comparison to state-of-the-art approaches.

IJCAI Conference 2023 Conference Paper

A Dual Semantic-Aware Recurrent Global-Adaptive Network for Vision-and-Language Navigation

• Liuyi Wang
• Zongtao He
• Jiagui Tang
• Ronghao Dang
• Naijia Wang
• Chengju Liu
• Qijun Chen

Vision-and-Language Navigation (VLN) is a realistic but challenging task that requires an agent to locate the target region using verbal and visual cues. While significant advancements have been achieved recently, there are still two broad limitations: (1) The explicit information mining for significant guiding semantics concealed in both vision and language is still under-explored; (2) The previously structured map method provides the average historical appearance of visited nodes, while it ignores distinctive contributions of various images and potent information retention in the reasoning process. This work proposes a dual semantic-aware recurrent global-adaptive network (DSRG) to address the above problems. First, DSRG proposes an instruction-guidance linguistic module (IGL) and an appearance-semantics visual module (ASV) for boosting vision and language semantic learning respectively. For the memory mechanism, a global adaptive aggregation module (GAA) is devised for explicit panoramic observation fusion, and a recurrent memory fusion module (RMF) is introduced to supply implicit temporal hidden states. Extensive experimental results on the R2R and REVERIE datasets demonstrate that our method achieves better performance than existing methods. Code is available at https: //github. com/CrystalSixone/DSRG.

ICRA Conference 2023 Conference Paper

D2NT: A High-Performing Depth-to-Normal Translator

• Yi Feng
• Bohuan Xue
• Ming Liu 0001
• Qijun Chen
• Rui Fan 0001

Surface normal holds significant importance in visual environmental perception, serving as a source of rich geometric information. However, the state-of-the-art (SoTA) surface normal estimators (SNEs) generally suffer from an unsatisfactory trade-off between efficiency and accuracy. To resolve this dilemma, this paper first presents a superfast depth-to-normal translator (D2NT), which can directly translate depth images into surface normal maps without calculating 3D coordinates. We then propose a discontinuity-aware gradient (DAG) filter, which adaptively generates gradient convolution kernels to improve depth gradient estimation. Finally, we propose a surface normal refinement module that can easily be integrated into any depth-to-normal SNEs, substantially improving the surface normal estimation accuracy. Our proposed algorithm demonstrates the best accuracy among all other existing real-time SNEs and achieves the SoTA trade-off between efficiency and accuracy.

ICRA Conference 2023 Conference Paper

FDLNet: Boosting Real-time Semantic Segmentation by Image-size Convolution via Frequency Domain Learning

• Qingqing Yan
• Shu Li 0005
• Chengju Liu
• Ming Liu 0001
• Qijun Chen

This paper proposes a novel real-time semantic segmentation network via frequency domain learning, called FDLNet, which revisits the segmentation task from two critical perspectives: spatial structure description and multilevel feature fusion. We first devise an image-size convolution (IS-Conv) as a global frequency-domain learning operator to capture long-range dependency in a single shot. To model spatial structure information, we construct the global structure representation path (GSRP) based on IS-Conv, which learns a unified edge-region representation with affordable complexity. For efficient and lightweight multi-level feature fusion, we propose the factorized stereoscopic attention (FSA) module, which alleviates semantic confusion and reduces feature redundancy by introducing level-wise attention before channel and spatial attention. Combining the above modules, we propose a concise semantic segmentation framework named FDLNet. We experimentally demonstrate the effectiveness and superiority of the proposed method. FDLNet achieves state-of-the-art performance on the Cityscapes, which reports 76. 32% mIoU at 150+ FPS and 79. 0% mIoU at 41+ FPS. The code is available at https://github.com/qyan0131/FDLNet.

ICML Conference 2023 Conference Paper

Multiple Thinking Achieving Meta-Ability Decoupling for Object Navigation

• Ronghao Dang
• Lu Chen
• Liuyi Wang
• Zongtao He
• Chengju Liu
• Qijun Chen

We propose a meta-ability decoupling (MAD) paradigm, which brings together various object navigation methods in an architecture system, allowing them to mutually enhance each other and evolve together. Based on the MAD paradigm, we design a multiple thinking (MT) model that leverages distinct thinking to abstract various meta-abilities. Our method decouples meta-abilities from three aspects: input, encoding, and reward while employing the multiple thinking collaboration (MTC) module to promote mutual cooperation between thinking. MAD introduces a novel qualitative and quantitative interpretability system for object navigation. Through extensive experiments on AI2-Thor and RoboTHOR, we demonstrate that our method outperforms state-of-the-art (SOTA) methods on both typical and zero-shot object navigation tasks.

ICRA Conference 2023 Conference Paper

Transparent Objects: A Corner Case in Stereo Matching

• Zhiyuan Wu
• Shuai Su
• Qijun Chen
• Rui Fan 0001

Stereo matching is a common technique used in 3D perception, but transparent objects such as reflective and penetrable glass pose a challenge as their disparities are often estimated inaccurately. In this paper, we propose transparency-aware stereo (TA-Stereo), an effective solution to tackle this issue. TA-Stereo first utilizes a semantic segmentation or salient object detection network to identify transparent objects, and then homogenizes them to enable stereo matching algorithms to handle them as non-transparent objects. To validate the effectiveness of our proposed TA-Stereo strategy, we collect 260 images containing transparent objects from the KITTI Stereo 2012 and 2015 datasets and manually label pixel-level ground truth. We evaluate our strategy with six deep stereo networks and two types of transparent object detection methods. Our experiments demonstrate that TA-Stereo significantly improves the disparity accuracy of transparent objects. Our project webpage can be accessed at mias. group/TA-Stereo.

ICRA Conference 2022 Conference Paper

HoloSeg: An Efficient Holographic Segmentation Network for Real-time Scene Parsing

• Shu Li 0005
• Qingqing Yan
• Chengju Liu
• Ming Liu 0001
• Qijun Chen

Real-time semantic segmentation is a crucial but challenging dense prediction task for scene parsing. However, the existing CNN-based methods commonly bias the model in favor of speed-boosting compromising spatial resolution due to business requirements and hardware constrains, which impedes the high-accuracy segmentation result. To address the dilemma, we provide a novel Holographic Segmentation Network (HoloSeg), which presents a strong ability of comprehensive information preservation and extraction, and achieves a better trade-off between speed and accuracy. We first design a Lossless Sample Pair (LSP) without any stride for early spatial preservation and later resolution recovery while modeling long-range context dependence. Then, we propose Distributed Pyramid Learning (DPL) to efficiently extract multiscale features and saves a lot of computation. Finally, we propose Resolution Fusion and Restoration (RFR) to fuse multi-level semantic representations across stages and generate output without decoder. Without bells and whistles, HoloSeg achieves state-of-the-art performance on the Cityscapes benchmark which reports 76. 24% mIoU at 231 FPS. Code is available online: https://github.com/LiShuTJ/HoloSeg.

IROS Conference 2019 Conference Paper

Improving Learning-based Ego-motion Estimation with Homomorphism-based Losses and Drift Correction

• Xiangwei Wang
• Daniel Maturana
• Shichao Yang
• Wenshan Wang
• Qijun Chen
• Sebastian A. Scherer

Visual odometry is an essential problem for mobile robots. Traditional methods for solving VO mostly utilize geometric optimization. While capable of achieving high accuracy, these methods require accurate sensor calibration and complicated parameter tuning to work well in practice. With the rise of deep learning, there has been increased interest in the end-to-end, learning-based methods for VO, which have the potential to improve robustness. However, learning-based methods for VO so far are less accurate than geometric methods. We argue that one of the main issues is that the current ego-motion estimation task is different from other problems where deep learning has been successful such as object detection. We define a novel cost function for learning-based VO considering the mathematical properties of the group homomorphism. In addition to the standard L2 loss, we incorporate losses based on the identity, inverse and closure properties of SE(3) rigid motion. Furthermore, we propose to reduce the VO drift by estimating the drivable regions using semantic segmentation and incorporate this information into a pose graph optimization. Experiments on KITTI datasets show that the novel cost function can improve ego-motion estimation compared to the state-of-the-art and the drivable region-based correction further reduces the VO drift.

ICRA Conference 2018 Conference Paper

Monocular Visual Odometry Scale Recovery Using Geometrical Constraint

• Xiangwei Wang
• Hui Zhang 0074
• Xiaochuan Yin
• Mingxiao Du
• Qijun Chen

Scale recovery is one of the essential problems for monocular visual odometry. The camera height is usually used as an absolute reference to recover the scale. In this case, the precision of scale recovery depends on the accuracy of the road region detection and road geometrical model calculation. In previous works, road detection and road geometrical model calculation are solved sequentially: the road geometrical model calculation is based on the road detection and the road region detection is based on the color information. However, the color information of a road is not stable enough. In the proposed method, the estimated road geometrical model is taken into consideration to detect the road region as a feedback. Therefore, the road region detection and road geometrical model estimation can benefit each other. Delaunay Triangulation method is used to segment an input image to many triangles with the matched feature points as vertices. Every triangle region is classified as a road region or not by comparing their geometrical model with that of the road and the road geometrical model is updated online. We evaluate our visual odometry scale recovery method on the KITTI dataset and the results show that our method is achieving the best performance among all existing monocular visual odometry scale recovery methods without additional sensors.

IROS Conference 2017 Conference Paper

Rebalance control for humanoid walking based on online foot position compensation

• Tong Zhang
• Chengju Liu
• Qijun Chen

This study proposes an online foot position compensator (FPC) for improving the robustness of biped walking. The proposed FPC learns foot placement parameters through sensory feedback and preview control theory to re-plan the trajectories of the center of mass (CoM) and gait pattern; the re-plan is based on the online generation of a walking pattern using the policy gradient reinforcement learning method (PGRL) to reject external perturbations and recover the walking position. A method of upper-body posture control is also presented to further improve the ability of humanoid robots to suppress external disturbances. The proposed posture controller stabilizes the robot by utilizing hip joints to modulate the upper-body posture in real time. The effectiveness of the proposed methods is validated by simulations on a full-body NAO humanoid robot.

ICRA Conference 2016 Conference Paper

Deep metric learning autoencoder for nonlinear temporal alignment of human motion

• Xiaochuan Yin
• Qijun Chen

Temporal alignment is an important preprocessing procedure for human action recognition. The challenge of temporal alignment problem is the temporal scale difference between human actions as well as the variability of each subject. Metric learning is the central problem of temporal alignment. This paper presents a nonlinear time alignment method with deep autoencoder. The spatio-temporal features obtained from the neural network contain the metric information for feature comparison. The effectiveness of our method is verified with k-nearest neighbor (k-NN) classifier on MSR-Action 3D and MSR-Daily Activity 3D datasets. Experimental results illustrate that the proposed method achieves superior performance to other metric based techniques.

ICRA Conference 2012 Conference Paper

Walking control strategy for biped robots based on central pattern generator

• Chengju Liu
• Qijun Chen

This paper deals with the walking control of biped robots inspired by biological concept of central pattern generator (CPG). A control architecture is proposed with a trajectory generator and a motion engine. The trajectory generator consists of a CoG (center of gravity) trajectory generator and a foot trajectory modulator. The CoG generator generates adaptive CoG trajectories online and the foot trajectories can be modulated based on the generated CoG trajectories. A biped platform NAO is used to validate the proposed locomotion control system. The experimental results confirm the effectiveness of the proposed control architecture.

ICRA Conference 2011 Conference Paper

Locomotion control of quadruped robots based on CPG-inspired workspace trajectory generation

• Chengju Liu
• Qijun Chen
• Danwei Wang

This paper presents a locomotion control strategy for quadruped robots based on central pattern generator (CPG). The proposed control architecture consists of a workspace trajectory generator and a motion engine. The CPG-inspired trajectory generator can generate workspace trajectories and the motion engine can calculate the accurate joint control signals. Moreover, entrainment with sensory feedback information from robot-environment interaction, the presented control system can generate adaptive joint control signals. A quadruped platform AIBO is used to validate the proposed control architecture and experimental results confirm the effectiveness of the control system.

ICRA Conference 2011 Conference Paper

Rebalance strategies for humanoids walking by foot positioning compensator based on adaptive heteroscedastic SpGPs

• Tao Xu
• Qijun Chen
• Zhiqiang Cai 0005

To solve the rebalance problem of a full-body humanoid walking, an adaptive foot positioning compensation approach is proposed. To obtain a more precise initial policy, a constrained dynamics model is used to generate the offline policy. A heteroscedastic sparse Gaussian process is applied for online calculation of the foot positioning policy. In order to make the generated policy to adapt with the full-body dynamics, a sample-efficient MAP-like updating method for the heteroscedastic sparse Gaussian process model is also proposed. Experiments on both simulation and a real full-body humanoid are developed to show the performance of the final foot positioning policy. With the help of proposed method, the full-body humanoid robot succeeded walking down an elastic deformable platform and several obvious compensation foot steps can be observed for the robot to retrieve its balance.

ICRA Conference 2010 Conference Paper

A bearing-only 2D/3D-homing method under a visual servoing framework

• Ming Liu 0001
• Cédric Pradalier
• Qijun Chen
• Roland Siegwart

Homing is one of the fundamental functions for both the mobile robot and the flying robot. Furthermore, homing can be introduced into a topological navigation system by cyclically setting Home positions at the keypoints/nodes in a topological map. In this work, we describe a bearing-only homing method based on only few matching keypoints to grant the mobile robot the homing ability. Our method considers the homing problem as a visual servoing problem in 2D plane and even in 3D space, using an omnidirectional camera as the visual sensor. It doesn't require the distance information to the reference feature points. The proof of the convergence for the algorithm is also given. The simulation results confirm the feasibility and robustness of our method.

ICRA Conference 2009 Conference Paper

Adaptive output feedback control of uncertain nonholonomic systems with strong nonlinear drifts

• Zhanping Yuan
• Zhuping Wang
• Qijun Chen

In this paper, an adaptive output feedback control strategy is presented to solve the stabilization problem of nonholonomic systems in chained form with strong nonlinear drifts and uncertain parameters using output signals only. The control law is developed using input-state scaling and backstepping techniques. The objective is to design adaptive nonlinear output feedback laws which can steer the closed-loop systems globally converge to the origin, while the estimated parameters remain bounded. An adaptive output feedback controller is proposed for a class of uncertain chained systems. Simulation results demonstrate the effectiveness of the proposed controllers.

IROS Conference 2009 Conference Paper

Scene recognition with omnidirectional vision for topological map using lightweight adaptive descriptors

• Ming Liu 0001
• Davide Scaramuzza 0001
• Cédric Pradalier
• Roland Siegwart
• Qijun Chen

Mobile robots rely on their ability of scene recognition to build a topological map of the environment and perform location-related tasks. In this paper, we describe a novel lightweight scene recognition method using an adaptive descriptor which is based on color features and geometric information for omnidirectional vision. Our method enables the robot to add nodes to a topological map automatically and solve the localization problem of mobile robot in realtime. The descriptor of a scene is extracted in the YUV color space and its dimension is adaptive depending on the segmentation result of the panoramic image. Furthermore, the descriptor is invariant to rotation and slight changes of illumination. The robustness of the scene matching and recognition is tested through real experiments in a dynamic indoor environment. The experiment is carried out on a mobile robot equipped with an omnidirectional camera. In our tests, the average processing time is 30 ms for each frame including feature extraction, matching, and the adding of new nodes.

ICML Conference 2003 Conference Paper

Eliminating Class Noise in Large Datasets

• Xingquan Zhu 0001
• Xindong Wu 0001
• Qijun Chen