Haibin Yan

IROS Conference 2025 Conference Paper

Anyview: General Indoor 3D Object Detection with Variable Frames

• Zhenyu Wu
• Xiuwei Xu
• Ziwei Wang 0001
• Chong Xia
• Linqing Zhao
• Jiwen Lu
• Haibin Yan

In this paper, we propose a novel network framework for indoor 3D object detection to handle variable input frame numbers in practical scenarios. Existing methods only consider fixed frames of input data for a single detector, such as monocular RGB-D images or point clouds reconstructed from dense multi-view RGB-D images. While in practical application scenes such as robot navigation and manipulation, the raw input to the 3D detectors is the RGB-D images with variable frame numbers instead of the reconstructed scene point cloud. However, the previous approaches can only handle fixed frame input data and have poor performance with variable frame input. In order to facilitate 3D object detection methods suitable for practical tasks, we present a novel 3D detection framework named AnyView for our practical applications, which generalizes well across different numbers of input frames with a single model. To be specific, we propose a geometric learner to mine the local geometric features of each input RGB-D image frame and implement local-global feature interaction through a designed spatial mixture module. Meanwhile, we further utilize a dynamic token strategy to adaptively adjust the number of extracted features for each frame, which ensures consistent global feature density and further enhances the generalization after fusion. Extensive experiments on the ScanNet dataset show our method achieves both great generalizability and high detection accuracy with a simple and clean architecture containing a similar amount of parameters with the baselines.

IROS Conference 2025 Conference Paper

Embodied Instruction Following in Unknown Environments

• Zhenyu Wu
• Ziwei Wang 0001
• Xiuwei Xu
• Hang Yin
• Yinan Liang
• Angyuan Ma
• Jiwen Lu
• Haibin Yan

Enabling embodied agents to complete complex human instructions from natural language is crucial to autonomous systems in household services. Conventional methods can only accomplish human instructions in the known environment where all interactive objects are provided to the embodied agent, and directly deploying the existing approaches for the unknown environment usually generates infeasible plans that manipulate non-existing objects. On the contrary, we propose an embodied instruction following (EIF) method for complex tasks in the unknown environment, where the agent efficiently explores the unknown environment to generate feasible plans with existing objects to accomplish abstract instructions. Specifically, we build a hierarchical embodied instruction following framework including the high-level task planner and the low-level exploration controller with multimodal large language models. We then construct a semantic representation map of the scene with dynamic region attention to demonstrate the known visual clues, where the goal of task planning and scene exploration is aligned for human instruction. For the task planner, we generate the feasible step-by-step plans for human goal accomplishment according to the task completion process and the known visual clues. For the exploration controller, the optimal navigation or object interaction policy is predicted based on the generated step-wise plans and the known visual clues. The experimental results demonstrate that our method can achieve 45. 09% success rate in 204 complex human instructions such as making breakfast and tidying rooms in large house-level scenes. Code and supplementary are available at https://gary3410.github.io/eif_unknown/.

IROS Conference 2025 Conference Paper

iGaussian: Real-Time Camera Pose Estimation via Feed-Forward 3D Gaussian Splatting Inversion

• Hao Wang
• Linqing Zhao
• Xiuwei Xu
• Jiwen Lu
• Haibin Yan

Recent trends in SLAM and visual navigation have embraced 3D Gaussians as the preferred scene representation, highlighting the importance of estimating camera poses from a single image using a pre-built Gaussian model. However, existing approaches typically rely on an iterative render-compare-refine loop, where candidate views are first rendered using NeRF or Gaussian Splatting, then compared against the target image, and finally, discrepancies are used to update the pose. This multi-round process incurs significant computational overhead, hindering real-time performance in robotics. In this paper, we propose iGaussian, a two-stage feed-forward framework that achieves real-time camera pose estimation through direct 3D Gaussian inversion. Our method first regresses a coarse 6DoF pose using a Gaussian Scene Prior-based Pose Regression Network with spatial uniform sampling and guided attention mechanisms, then refines it through feature matching and multi-model fusion. The key contribution lies in our cross-correlation module that aligns image embeddings with 3D Gaussian attributes without differentiable rendering, coupled with a Weighted Multiview Predictor that fuses features from Multiple strategically sampled viewpoints. Experimental results on the NeRF Synthetic, Mip-NeRF 360, and T&T+DB datasets demonstrate a significant performance improvement over previous methods, reducing median rotation errors to 0. 2° while achieving 2. 87 FPS tracking on mobile robots, which is an impressive 10× speedup compared to optimization-based approaches. Project page: https://github.com/pythongod-exe/iGaussian

IROS Conference 2025 Conference Paper

Pseudo Depth Meets Gaussian: A Feed-forward RGB SLAM Baseline

• Linqing Zhao
• Xiuwei Xu
• Yirui Wang
• Hao Wang
• Wenzhao Zheng
• Yansong Tang
• Haibin Yan
• Jiwen Lu

ICRA Conference 2023 Conference Paper

Category-level Shape Estimation for Densely Cluttered Objects

• Zhenyu Wu
• Ziwei Wang 0001
• Jiwen Lu
• Haibin Yan

Accurately estimating the shape of objects in dense clutters makes important contribution to robotic packing, because the optimal object arrangement requires the robot planner to acquire shape information of all existed objects. However, the objects for packing are usually piled in dense clutters with severe occlusion, and the object shape varies significantly across different instances for the same category. They respectively cause large object segmentation errors and inaccurate shape recovery on unseen instances, which both degrade the performance of shape estimation during deployment. In this paper, we propose a category-level shape estimation method for densely cluttered objects. Our framework partitions each object in the clutter via the multi-view visual information fusion to achieve high segmentation accuracy, and the instance shape is recovered by deforming the category templates with diverse geometric transformations to obtain strengthened generalization ability. Specifically, we first collect the multi-view RGB-D images of the object clutters for point cloud reconstruction. Then we fuse the feature maps representing the visual information of multi-view RGB images and the pixel affinity learned from the clutter point cloud, where the acquired instance segmentation masks of multi-view RGB images are projected to partition the clutter point cloud. Finally, the instance geometry information is obtained from the partially observed instance point cloud and the corresponding category template, and the deformation parameters regarding the template are predicted for shape estimation. Experiments in the simulated environment and real world show that our method achieves high shape estimation accuracy for densely cluttered everyday objects with various shapes.

IROS Conference 2022 Conference Paper

Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration

• Zhenyu Wu
• Ziwei Wang 0001
• Zibu Wei
• Yi Wei 0003
• Haibin Yan

Recognizing objects in dense clutter accurately plays an important role to a wide variety of robotic manipulation tasks including grasping, packing, rearranging and many others. However, conventional visual recognition models usually miss objects because of the significant occlusion among instances and causes incorrect prediction due to the visual ambiguity with the high object crowdedness. In this paper, we propose an interactive exploration framework called Smart Explorer for recognizing all objects in dense clutters. Our Smart Explorer physically interacts with the clutter to maximize the recognition performance while minimize the number of motions, where the false positives and negatives can be alleviated effectively with the optimal accuracy-efficiency trade-offs. Specifically, we first collect the multi-view RGB-D images of the clutter and reconstruct the corresponding point cloud. By aggregating the instance segmentation of RGB images across views, we acquire the instance-wise point cloud partition of the clutter through which the existed classes and the number of objects for each class are predicted. The pushing actions for effective physical interaction are generated to sizably reduce the recognition uncertainty that consists of the instance segmentation entropy and multi-view object disagreement. Therefore, the optimal accuracy-efficiency trade-off of object recognition in dense clutter is achieved via iterative instance prediction and physical interaction. Extensive experiments demonstrate that our Smart Explorer acquires promising recognition accuracy with only a few actions, which also outperforms the random pushing by a large margin.

ICRA Conference 2011 Conference Paper

Cross-dataset facial expression recognition

• Haibin Yan
• Marcelo H. Ang
• Aun Neow Poo

This paper investigates the problem of cross-dataset facial expression recognition. To the best of our knowledge, this problem has not been formally addressed in the literature. Conventional facial expression recognition methods assume expression images in the training and testing sets are collected under the same condition such that they are independent and identically distributed. In many real applications, this assumption may not hold as the testing data are usually collected online and generally more uncontrollable than the training data, and hence, they are likely different from the training data. This problem is referred to as cross-dataset facial expression recognition in this paper as the training and testing data are considered to be collected from different datasets due to different acquisition conditions. To address this, we propose a new transfer subspace learning approach to learn a feature space which transfers the knowledge gained from the training set to the target (testing) data to improve the recognition performance under cross-dataset scenarios. Experimental results for facial expression recognition tasks on different datasets are presented to demonstrate the efficacy of the proposed approach.

ICRA Conference 2011 Conference Paper

Weighted biased linear discriminant analysis for misalignment-robust facial expression recognition

• Haibin Yan
• Marcelo H. Ang
• Aun Neow Poo

We investigate in this paper the problem of misalignment-robust facial expression recognition. To the best of our knowledge, this problem has not been formally addressed in the literature. Most existing facial expression recognition methods, however, can only work well when face images are well-aligned. In many real world applications such as human robot interaction and visual surveillance, it is still very challenging to obtain well-aligned face images for expression recognition due to currently imperfect vision techniques, especially under uncontrolled conditions. Motivated by the fact that interclass facial images with small differences are more easily mis-classified than those with large differences, we propose a biased linear discriminant analysis (BLDA) method by imposing large penalties on interclass samples with small differences and small penalties on those samples with large differences simultaneously, such that more discriminative features can be extracted for recognition. Moreover, we generate more virtually misaligned facial expression samples and assign different weights to them according to their occurrence probabilities in the testing phase to learn a weighted BLDA (WBLDA) feature space to extract misalignment-robust discriminative features for recognition. Experimental results on two widely used face databases are presented to show the efficacy of the proposed method.