Author name cluster

Zhenshan Bing

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

38 papers

2 author rows

IROS Conference 2025 Conference Paper

CIT: Context-Based Biased Batch-Sampling for Almost-Surely Asymptotically Optimal Motion Planning

Liding Zhang
Yankun Wei
Kuanqi Cai
Zhenshan Bing
Yuan Meng
Fan Wu 0015
Sami Haddadin
Alois C. Knoll

This paper introduces Context Informed Trees (CIT*), a sampling-based motion planning algorithm that enhances exploration efficiency by biasing sampling based on uncertainty estimation from local samples and connectivity information obtained during the search process. CIT* is based on Flexible Informed Trees (FIT*) and incorporates three key components: region-based sampling, uncertainty-driven weighting, and connection-greedy prioritization (CGP). It generates regions from sampled states based on local obstacle proximity, assigning weights to these regions using probability uncertainty estimation via kernel density estimation (KDE) classification. To further refine the sampling focus, CGP prioritizes regions that exhibit strong connectivity in previous searches, ensuring that exploration is directed toward unknown and critical areas that have a higher likelihood of contributing to feasible and efficient paths. The sampling process is then guided by a mixture of Gaussian distributions centered on weighted regions, where the weighting biases sampling toward more critical regions, thereby improving search efficiency and accelerating convergence. Benchmark evaluations demonstrate that CIT* improves efficiency by reducing reliance on random sampling, which often leads to slower solution discovery and higher path costs. With biased sampling, CIT* maintains strong performance in solving complex motion planning problems in ${\mathbb{R}^4}$ to ${\mathbb{R}^{16}}$ and has been demonstrated on a real-world manipulation task. A video showcasing our method and experimental results is available at: https://youtu.be/SG2cy9WmjD0.

IROS Conference 2025 Conference Paper

ContactDexNet: Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-Object Contact Semantic Mapping

Lei Zhang 0035
Kaixin Bai
Guowen Huang
Zhenshan Bing
Zhaopeng Chen
Alois C. Knoll
Jianwei Zhang 0001

The deep learning models has significantly advanced dexterous manipulation techniques for multi-fingered hand grasping. However, the contact information-guided grasping in cluttered environments remains largely underexplored. To address this gap, we have developed ContactDexNet, a method for generating multi-fingered hand grasp samples in cluttered settings through contact semantic map. We introduce a contact semantic conditional variational autoencoder network (CoSe-CVAE) for creating comprehensive contact semantic map from object point cloud. We utilize grasp detection method to estimate hand grasp poses from the contact semantic map. Finally, an unified grasp evaluation model PointNetGPD++ is designed to assess grasp quality and collision probability, substantially improving the reliability of identifying optimal grasps in cluttered scenarios. Our grasp generation method has demonstrated remarkable success, outperforming state-of-the-art (SOTA) methods by at least 4. 7%, with 81. 0% average grasping success rate in real-world single-object grasping using a known hand, and by at least 9. 0% when using an unknown hand. Moreover, in cluttered scenes, our method attains a 76. 7% success rate, outperforming the SOTA method by 6. 3%. We also proposed the multi-modal multi-fingered grasping dataset generation method. Our multi-fingered hand grasping dataset outperforms previous datasets in scene diversity, modality diversity. More details and supplementary materials can be found at https://sites.google.com/view/contact-dexnet.

ICRA Conference 2025 Conference Paper

Direction Informed Trees (DIT*): Optimal Path Planning via Direction Filter and Direction Cost Heuristic

Liding Zhang
Kejia Chen 0005
Kuanqi Cai
Yu Zhang 0182
Yixuan Dang
Yansong Wu
Zhenshan Bing
Fan Wu 0015

Optimal path planning requires finding a series of feasible states from the starting point to the goal to optimize objectives. Popular path planning algorithms, such as Effort Informed Trees (EIT*), employ effort heuristics to guide the search. Effective heuristics are accurate and computationally efficient, but achieving both can be challenging due to their conflicting nature. This paper proposes Direction Informed Trees (DIT*), a sampling-based planner that focuses on optimizing the search direction for each edge, resulting in goal bias during exploration. We define edges as generalized vectors and integrate similarity indexes to establish a directional filter that selects the nearest neighbors and estimates direction costs. The estimated direction cost heuristics are utilized in edge evaluation. This strategy allows the exploration to share directional information efficiently. DIT* convergence faster than existing single-query, sampling-based planners on tested problems in $\mathbb{R}^{4}$ to $\mathbb{R}^{16}$ and has been demonstrated in real-world environments with various planning tasks. A video showcasing our experimental results is available at: https://youtu.be/2SX6QT2NOek.

ICRA Conference 2025 Conference Paper

Gassidy: Gaussian Splatting SLAM in Dynamic Environments

Long Wen 0003
Shixin Li
Yu Zhang 0182
Yuhong Huang
Jianjie Lin
Fengjunjie Pan
Zhenshan Bing
Alois C. Knoll

3D Gaussian Splatting (3DGS) allows flexible adjustments to scene representation, enabling continuous optimization of scene quality during dense visual simultaneous localization and mapping (SLAM) in static environments. However, 3DGS faces challenges in handling environmental disturbances from dynamic objects with irregular movement, leading to degradation in both camera tracking accuracy and map reconstruction quality. To address this challenge, we develop an RGB-D dense SLAM which is called Gaussian Splatting SLAM in Dynamic Environments (Gassidy). This approach calculates Gaussians to generate rendering loss flows for each environmental component based on a designed photometricgeometric loss function. To distinguish and filter environmental disturbances, we iteratively analyze rendering loss flows to detect features characterized by changes in loss values between dynamic objects and static components. This process ensures a clean environment for accurate scene reconstruction. Compared to state-of-the-art SLAM methods, experimental results on open datasets show that Gassidy improves camera tracking precision by up to 97. 9 % and enhances map quality by up to 6 %. Video of experiments is available here: https://www.wixsite.com.com/wen-Gassidy.

IROS Conference 2025 Conference Paper

Instantaneous Contact Localization on A Magnetically Transduced Tapered Whisker

Yixuan Dang
Yichen Zhang
Yuhong Huang
Long Wen 0003
Yu Zhang 0182
Zhenshan Bing
Florian Röhrbein
Alois C. Knoll

The whisker-inspired tactile sensor is advantageous for enhancing robotic perception in proximate range and darkness via non-intrusive contacts. However, localizing contact along the whisker shaft is challenging due to the non-injective mapping between tangential contacts and the resulting bending moments at the whisker base. Previous studies suggest that incorporating axial force measurements can resolve this ambiguity. In this work, we develop a magnetically transduced whisker sensor that integrates axial force sensing as an additional mechanical signal. The sensor features a tapered whisker with a custom slope and a 3-DoF suspension mechanism, enabling axial displacement at the base, which is proportional to the applied axial force. We construct a Penalized Gaussian Process model trained on synthetic data to estimate the whisker’s motion and refine it with real-data constraints. The design is compact, low-cost, and validated through simulations and real-world experiments to differentiate tangential contacts. Furthermore, we propose an optimization-based approach for estimating instantaneous contact locations. Experimental results demonstrate that the proposed method can effectively track contacts in millimeter-level accuracy with a mean error of 7. 17 mm, achieving a higher accuracy with only 4. 02 mm in large-deflection and close-to-base regions.

ICRA Conference 2025 Conference Paper

LEMMo-Plan: LLM-Enhanced Learning from Multi-Modal Demonstration for Planning Sequential Contact-Rich Manipulation Tasks

Kejia Chen 0005
Zheng Shen
Yue Zhang
Lingyun Chen
Fan Wu 0015
Zhenshan Bing
Sami Haddadin
Alois C. Knoll

Large Language Models (LLMs) have gained popularity in task planning for long-horizon manipulation tasks. To enhance the validity of LLM-generated plans, visual demonstrations and online videos have been widely employed to guide the planning process. However, for manipulation tasks involving subtle movements but rich contact interactions, visual perception alone may be insufficient for the LLM to fully interpret the demonstration. Additionally, visual data provides limited information on force-related parameters and conditions, which are crucial for effective execution on real robots. In this paper, we introduce LEMMo-Plan, an in-context learning framework that incorporates tactile and force-torque information from human demonstrations to enhance LLMs' ability to generate plans for new task scenarios. We propose a bootstrapped reasoning pipeline that sequentially integrates each modality into a comprehensive task plan. This task plan is then used as a reference for planning in new task configurations. Real-world experiments on two different sequential manipulation tasks demonstrate the effectiveness of our framework in improving LLMs' understanding of multi-modal demonstrations and enhancing the overall planning performance. More materials are available on our project website: lemmo-plan.github.io/LEMMo-Plan/.

IROS Conference 2025 Conference Paper

Multi-Robot Assembly of Deformable Linear Objects Using Multi-Modal Perception

Kejia Chen 0005
Celina Dettmering
Florian Pachler
Zhuo Liu
Yue Zhang
Tailai Cheng
Jonas Dirr
Zhenshan Bing

Industrial assembly of deformable linear objects (DLOs) such as cables offers great potential for many industries. However, DLOs pose several challenges for robot-based automation due to the inherent complexity of deformation and, consequentially, the difficulties in anticipating the behavior of DLOs in dynamic situations. Although existing studies have addressed isolated subproblems like shape tracking, grasping, and shape control, there has been limited exploration of integrated workflows that combine these individual processes. To address this gap, we propose an object-centric perception and planning framework to achieve a comprehensive DLO assembly process throughout the industrial value chain. The framework utilizes visual and tactile information to track the DLO’s shape as well as contact state across different stages, which facilitates effective planning of robot actions. Our approach encompasses robot-based bin picking of DLOs from cluttered environments, followed by a coordinated handover to two additional robots that mount the DLOs onto designated fixtures. Real-World experiments employing a setup with multiple robots demonstrate the effectiveness of the approach and its relevance to industrial scenarios.

IROS Conference 2025 Conference Paper

Multi-Sets Trees (MST*): Accelerated Asymptotically Optimal Motion Planning Optimization Informed by Multiple Domain Subsets

Liding Zhang
Sicheng Wang
Kuanqi Cai
Zhenshan Bing
Alois C. Knoll

Robotic motion planning faces formidable challenges in constrained environments, particularly in rapidly searching for feasible solutions and converging towards optimal. This study introduces Multi-Sets Tree (MST*), a sampling-based planner designed to accelerate path searching and solution optimization. MST* integrates estimated guided incremental local densification (GuILD) sets that are based on prior estimated solution costs before finding the initial solution. For path optimization, MST* integrates novel beacon selectors to define problem subsets, thereby guiding exploration and effectively exploiting high-potential areas. This multi-set strategy ensures balanced exploration and exploitation, enabling MST* to handle sparse free space. Moreover, MST* utilizes adaptive sampling techniques via Lebesgue’s measure of domain subsets for rapid search. MST* improves search efficiency and path optimality, particularly in constrained high-dimensional environments. It extends the informed sampling concept by refining the search region and batch sampling. Experimental results demonstrate that MST* outperforms single-query planners across ℝ 4 to ℝ 16 benchmarks and in real-world robotic navigation tasks. A video showcasing our experimental results is available at: https://youtu.be/obftvS0a41M.

ICRA Conference 2025 Conference Paper

Safety-Critical Control with Saliency Detection for Mobile Robots in Dynamic Multi-Obstacle Environments

Yu Zhang 0182
Long Wen 0003
Lin Hong
Liding Zhang
Qun Guo
Shixin Li
Zhenshan Bing
Alois C. Knoll

This paper proposes a novel dual-filter architecture utilizing RGB-D camera data and dynamic control barrier functions (D-CBFs) for real-time obstacle avoidance in unstructured environments. The proposed method efficiently handles static, suddenly appearing, and dynamic obstacles, maintaining consistent computational performance across diverse scenarios. To achieve this, two key challenges must be addressed. First, the substantial volume of pixel and depth map data requires robust, real-time processing for efficient D-CBF construction. Second, constructing D-CBFs for each obstacle in multi-obstacle scenarios increases optimization solver time. To address these challenges, we adapt the concept of salient object detection (SOD), proposing an enhanced FastSOD (E-FastSOD) method for rapid risk area identification. This approach rapidly filters out low-risk areas, while high-risk regions are mathematically represented utilizing the proposed enhanced minimal bounding circle (E-MBC) technique. We differentiate static and dynamic obstacles by comparing current and previous MBC states, employing Kalman filtering for obstacle state prediction. This setup enables efficient online D-CBF construction for each MBC, balancing computational speed with accurate obstacle representation. Subsequently, the second filter establishes buffer zones around established D-CBFs, activating only those corresponding to zones the robot actually enters, rather than all D-CBFs to increase real-time performance. We prove the system's safety and asymptotic stabilization under this architecture. Simulated and real-world experiments validate our method, demonstrating an equipped mobile robot's ability to accomplish tasks while ensuring safety across diverse, unknown scenarios.

ICRA Conference 2025 Conference Paper

TacDiffusion: Force-Domain Diffusion Policy for Precise Tactile Manipulation

Yansong Wu
Zongxie Chen
Fan Wu 0015
Lingyun Chen
Liding Zhang
Zhenshan Bing
Abdalla Swikir
Sami Haddadin

Assembly is a crucial skill for robots in both modern manufacturing and service robotics. However, mastering transferable insertion skills that can handle a variety of high-precision assembly tasks remains a significant challenge. This paper presents a novel framework that utilizes diffusion models to generate 6D wrench for high-precision tactile robotic insertion tasks. It learns from demonstrations performed on a single task and achieves a zero-shot transfer success rate of 95. 7% across various novel high-precision tasks. Our method effectively inherits the self-adaptability demonstrated by our previous work. In this framework, we address the frequency misalignment between the diffusion policy and the real-time control loop with a dynamic system-based filter, significantly improving the task success rate by 9. 15%. Furthermore, we provide a practical guideline regarding the trade-off between diffusion models' inference ability and speed.

ICRA Conference 2025 Conference Paper

Whisker-Based Active Tactile Perception for Contour Reconstruction

Yixuan Dang
Qinyang Xu
Yu Zhang 0182
Xiangtong Yao
Liding Zhang
Zhenshan Bing
Florian Röhrbein
Alois C. Knoll

Perception using whisker-inspired tactile sensors currently faces a major challenge: the lack of active control in robots based on direct contact information from the whisker. To accurately reconstruct object contours, it is crucial for the whisker sensor to continuously follow and maintain an appropriate relative touch pose on the surface. This is especially important for localization based on tip contact, which has a low tolerance for sharp surfaces and must avoid slipping into tangential contact. In this paper, we first construct a magnetically transduced whisker sensor featuring a compact and robust suspension system composed of three flexible spiral arms. We develop a method that leverages a characterized whisker deflection profile to directly extract the tip contact position using gradient descent, with a Bayesian filter applied to reduce fluctuations. We then propose an active motion control policy to maintain the optimal relative pose of the whisker sensor against the object surface. A B-Spline curve is employed to predict the local surface curvature and determine the sensor orientation. Results demonstrate that our algorithm can effectively track objects and reconstruct contours with sub-millimeter accuracy. Finally, we validate the method in simulations and real-world experiments where a robot arm drives the whisker sensor to follow the surfaces of three different objects.

ICRA Conference 2024 Conference Paper

1 kHz Behavior Tree for Self-adaptable Tactile Insertion

Yansong Wu
Fan Wu 0015
Lingyun Chen
Kejia Chen 0005
Samuel Schneider
Lars Johannsmeier
Zhenshan Bing
Fares J. Abu-Dakka

Insertion is an essential skill for robots in both modern manufacturing and services robotics. In our previous study, we proposed an insertion skill framework based on forcedomain wiggle motion. The main limitation of this method lies in the robot’s inability to adjust its behavior according to changing contact state during interaction. In this paper, we extend the skill formalism by incorporating a behavior tree-based primitive switching mechanism that leverages highfrequency tactile data for the estimation of contact state. The efficacy of our proposed framework is validated with a series of experiments that involve the execution of tightly constrained peg-in-hole tasks. The experiment results demonstrate a significant improvement in performance, characterized by reduced execution time, heightened robustness, and superior adaptability when confronted with unknown tasks. Moreover, in the context of transfer learning, our paper provides empirical evidence indicating that the proposed skill framework contributes to enhanced transferability across distinct operational contexts and tasks.

ICRA Conference 2024 Conference Paper

Contact Energy Based Hindsight Experience Prioritization

Erdi Sayar
Zhenshan Bing
Carlo D'Eramo
Ozgur S. Oguz
Alois C. Knoll

Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be utilized as a contribution to learning. However, HER uniformly chooses failed trajectories, without taking into account which ones might be the most valuable for learning. In this paper, we address this problem and propose a novel approach Contact Energy Based Prioritization (CEBP) to select the samples from the replay buffer based on rich information due to contact, leveraging the touch sensors in the gripper of the robot and object displacement. Our prioritization scheme favors sampling of contact-rich experiences, which are arguably the ones providing the largest amount of information. We evaluate our proposed approach on various sparse reward robotic tasks and compare it with the state-of-the-art methods. We show that our method surpasses or performs on par with those methods on robot manipulation tasks. Finally, we deploy the trained policy from our method to a real Franka robot for a pick-and-place task. We observe that the robot can solve the task successfully. The videos and code are publicly available at: https://erdiphd.github.io/HER_force/.

IROS Conference 2024 Conference Paper

Elliptical K-Nearest Neighbors - Path Optimization via Coulomb's Law and Invalid Vertices in C-space Obstacles

Liding Zhang
Zhenshan Bing
Yu Zhang 0182
Kuanqi Cai
Lingyun Chen
Fan Wu 0015
Sami Haddadin
Alois C. Knoll

Path planning has long been an important and active research area in robotics. To address challenges in high-dimensional motion planning, this study introduces the Force Direction Informed Trees (FDIT*), a sampling-based planner designed to enhance speed and cost-effectiveness in pathfinding. FDIT* builds upon the state-of-the-art informed sampling planner, the Effort Informed Trees (EIT*), by capitalizing on often-overlooked information in invalid vertices. It incorporates principles of physical force, particularly Coulomb’s law. This approach proposes the elliptical k-nearest neighbors search method, enabling fast convergence navigation and avoiding high solution cost or infeasible paths by exploring more problem-specific search-worthy areas. It demonstrates benefits in search efficiency and cost reduction, particularly in confined, high-dimensional environments. It can be viewed as an extension of nearest neighbors search techniques. Fusing invalid vertex data with physical dynamics facilitates force-direction-based search regions, resulting in an improved convergence rate to the optimum. FDIT* outperforms existing single-query, sampling-based planners on the tested problems in ℝ 4 to ℝ 16 and has been demonstrated on a real-world mobile manipulation task.

IROS Conference 2024 Conference Paper

Flexible Informed Trees (FIT*): Adaptive Batch-Size Approach in Informed Sampling-Based Path Planning

Liding Zhang
Zhenshan Bing
Kejia Chen 0005
Lingyun Chen
Kuanqi Cai
Yu Zhang 0182
Fan Wu 0015
Peter Krumbholz

In path planning, anytime almost-surely asymptotically optimal planners dominate the benchmark of sampling-based planners. A notable example is Batch Informed Trees (BIT*), where planners iteratively determine paths to batches of vertices within the exploration area. However, utilizing a consistent batch size is inefficient for initial pathfinding and optimal performance, it relies on effective task allocation. This paper introduces Flexible Informed Trees (FIT*), a sampling-based planner that integrates an adaptive batch-size method to enhance the initial path convergence rate. FIT* employs a flexible approach in adjusting batch sizes dynamically based on the inherent dimension of the configuration spaces and the hypervolume of the n-dimensional hyperellipsoid. By applying dense and sparse sampling strategy, FIT* improves convergence rate while finding successful solutions faster with lower initial solution cost. This method enhances the planner’s ability to handle confined, narrow spaces in the initial finding phase and increases batch vertices sampling frequency in the optimization phase. FIT* outperforms existing single-query, sampling-based planners on the tested problems in R 2 to R 8, and was demonstrated on a real-world mobile manipulation task.

IROS Conference 2024 Conference Paper

IMU Based Pose Reconstruction and Closed-loop Control for Soft Robotic Arms

Guanran Pei
Francesco Stella
Omar Meebed
Zhenshan Bing
Cosimo Della Santina
Josie Hughes

Soft continuum manipulators are celebrated for their versatility and physical robustness to external forces and perturbations. However, this feature comes at a cost. The many degrees of freedom and compliance pose challenges for accurate pose reconstruction, both in terms of distributed sensing and pose reconstruction algorithms. Moreover, soft arms are inherently susceptible to deformation from external forces or loads, meaning that closed-loop control is essential for robust task performance. In this article, we propose the integration of multiple Inertial Measurement Units (IMUs) of a soft robot arm, Helix, for reconstruction of pose under internal and external forces. Furthermore, we integrate this dynamic pose reconstruction for kinematic-based closed-loop control strategies. By serially integrating sensing in the body of the Helix soft manipulator, we provide the system with high-frequency pose reconstruction and demonstrate improvements in end effector position with comparison to open-loop performance.

IROS Conference 2024 Conference Paper

Online Efficient Safety-Critical Control for Mobile Robots in Unknown Dynamic Multi-Obstacle Environments

Yu Zhang 0182
Guangyao Tian
Long Wen 0003
Xiangtong Yao
Liding Zhang
Zhenshan Bing
Wei He 0001
Alois C. Knoll

This paper proposes a LiDAR-based goal-seeking and exploration framework, addressing the efficiency of online obstacle avoidance in unstructured environments populated with static and moving obstacles. This framework addresses two significant challenges associated with traditional dynamic control barrier functions (D-CBFs): their online construction and the diminished real-time performance caused by utilizing multiple D-CBFs. To tackle the first challenge, the framework’s perception component begins with clustering point clouds via the DBSCAN algorithm, followed by encapsulating these clusters with the minimum bounding ellipses (MBEs) algorithm to create elliptical representations. By comparing the current state of MBEs with those stored from previous moments, the differentiation between static and dynamic obstacles is realized, and the Kalman filter is utilized to predict the movements of the latter. Such analysis facilitates the D-CBF’s online construction for each MBE. To tackle the second challenge, we introduce buffer zones, generating Type-II D-CBFs online for each identified obstacle. Utilizing these buffer zones as activation areas substantially reduces the number of D-CBFs that need to be activated. Upon entering these buffer zones, the system prioritizes safety, autonomously navigating safe paths, and hence referred to as the exploration mode. Exiting these buffer zones triggers the system’s transition to goal-seeking mode. We demonstrate that the system’s states under this framework achieve safety and asymptotic stabilization. Experimental results in simulated and real-world environments have validated our framework’s capability, allowing a LiDAR-equipped mobile robot to efficiently and safely reach the desired location within dynamic environments containing multiple obstacles. Video and code are available: https://zyzhang4.wixsite.com/iros2024.

IROS Conference 2024 Conference Paper

Ontology Based AI Planning and Scheduling for Robotic Assembly

Jingyun Zhao
Birgit Vogel-Heuser
Jicong Ao
Yansong Wu
Liding Zhang
Fandi Hartl
Dominik Hujo
Zhenshan Bing

The rising demand for customized products necessitates the integration of multiple robotic systems, underscoring the need for advanced production planning and scheduling. This paper introduces an ontology-based, artificial intelligence-enhanced method for dynamic task planning and scheduling, aimed at improving the efficiency of production process, reducing machine downtime, and consequently increasing throughput in assembly operations. Designed to generate and execute feasible production plans dynamically, this method minimizes manual planning and scheduling efforts. We evaluate its effectiveness using two gear assembly use cases with various robot skills, highlighting its flexibility in planning and scheduling and its contributions to the evolution of smart manufacturing. The method’s adaptability suggests its applicability across diverse smart factory environments.

ICRA Conference 2024 Conference Paper

Optimizing Dynamic Balance in a Rat Robot via the Lateral Flexion of a Soft Actuated Spine

Yuhong Huang
Zhenshan Bing
Zitao Zhang
Genghang Zhuang
Kai Huang 0001
Alois C. Knoll

Balancing oneself using the spine is a physiological alignment of the body posture in the most efficient manner by the muscular forces for mammals. For this reason, we can see many disabled quadruped animals can still stand or walk even with three limbs. This paper investigates the optimization of dynamic balance during trot gait based on the spatial relationship between the center of mass (CoM) and support area influenced by spinal flexion. During trotting, the robot balance is significantly influenced by the distance of the CoM to the support area formed by diagonal footholds. In this context, lateral spinal flexion, which is able to modify the position of footholds, holds promise for optimizing balance during trotting. This paper explores this phenomenon using a rat robot equipped with a soft actuated spine. Based on the lateral flexion of the spine, we establish a kinematic model to quantify the impact of spinal flexion on robot balance during trot gait. Subsequently, we develop an optimized controller for spinal flexion, designed to enhance balance without altering the leg locomotion. The effectiveness of our proposed controller is evaluated through extensive simulations and physical experiments conducted on a rat robot. Compared to both a non-spine based trot gait controller and a trot gait controller with lateral spinal flexion, our proposed optimized controller effectively improves the dynamic balance of the robot and retains the desired locomotion during trotting.

ICRA Conference 2024 Conference Paper

Real-Time Adaptive Safety-Critical Control with Gaussian Processes in High-Order Uncertain Models

Yu Zhang 0182
Long Wen 0003
Xiangtong Yao
Zhenshan Bing
Linghuan Kong
Wei He 0001
Alois C. Knoll

This paper presents an adaptive online learning framework for systems with uncertain parameters to ensure safety-critical control in non-stationary environments. Our approach consists of two phases. The initial phase is centered on a novel sparse Gaussian process (GP) framework. We first integrate a forgetting factor to refine a variational sparse GP algorithm, thus enhancing its adaptability. Subsequently, the hyperparameters of the Gaussian model are trained with a specially compound kernel, and the Gaussian model’s online inferential capability and computational efficiency are strengthened by updating a solitary inducing point derived from newly samples, in conjunction with the learned hyperparameters. In the second phase, we propose a safety filter based on high order control barrier functions (HOCBFs), synergized with the previously trained learning model. By leveraging the compound kernel from the first phase, we effectively address the inherent limitations of GPs in handling high-dimensional problems for real-time applications. The derived controller ensures a rigorous lower bound on the probability of satisfying the safety specification. Finally, the efficacy of our proposed algorithm is demonstrated through real-time obstacle avoidance experiments executed using both simulation platform and a real-world 7-DOF robot.

ICRA Conference 2024 Conference Paper

Real-time Contact State Estimation in Shape Control of Deformable Linear Objects under Small Environmental Constraints

Kejia Chen 0005
Zhenshan Bing
Yansong Wu
Fan Wu 0015
Liding Zhang
Sami Haddadin
Alois C. Knoll

Controlling the shape of deformable linear objects using robots and constraints provided by environmental fixtures has diverse industrial applications. In order to establish robust contacts with these fixtures, accurate estimation of the contact state is essential for preventing and rectifying potential anomalies. However, this task is challenging due to the small sizes of fixtures, the requirement for real-time performances, and the infinite degrees of freedom of the deformable linear objects. In this paper, we propose a real-time approach for estimating both contact establishment and subsequent changes by leveraging the dependency between the applied and detected contact force on the deformable linear objects. We seamlessly integrate this method into the robot control loop and achieve an adaptive shape control framework which avoids, detects and corrects anomalies automatically. Real-world experiments validate the robustness and effectiveness of our contact estimation approach across various scenarios, significantly increasing the success rate of shape control processes.

IROS Conference 2023 Conference Paper

An Energy-Efficient Lane-Keeping System Using 3D LiDAR Based on Spiking Neural Network

Genghang Zhuang
Zhenshan Bing
Zhen Zhou
Xiangtong Yao
Yuhong Huang
Kai Huang 0001
Alois C. Knoll

Lane keeping, as a fundamental functionality of autonomous navigation, remains a challenging task for autonomous robots and vehicles. Recently, spiking neural networks (SNNs) have gained attention and research interest due to their biological plausibility and application potential on neuromorphic processors. SNNs have also been successfully deployed on robots to solve autonomous navigation problems. However, lane keeping with a LiDAR sensor is still an open problem for SNNs. In this work, we propose an end-to-end approach based on an SNN to solve the lane-keeping problem using a 3D LiDAR sensor. For the first time, we explore the capability of the proposed SNN controller to perceive the LiDAR input and exploit the features to perform reward-based feedback learning. To ensure the effectiveness of the controller, the proposed method is deployed and evaluated on two high-fidelity simulators. The experimental results demonstrate the high applicability and performance in different scenarios. Furthermore, experiments show that the SNN is capable of performing lane keeping in a simulated urban environment with only 18 control neurons and 32 synapse connections, producing on average only a 17cm deviation from lane center, which is 4. 3 % of the lane width.

IROS Conference 2023 Conference Paper

Contact-Aware Shaping and Maintenance of Deformable Linear Objects With Fixtures

Kejia Chen 0005
Zhenshan Bing
Fan Wu 0015
Yuan Meng
André Kraft
Sami Haddadin
Alois C. Knoll

Studying the manipulation of deformable linear objects has significant practical applications in industry, including car manufacturing, textile production, and electronics automation. However, deformable linear object manipulation poses a significant challenge in developing planning and control algorithms, due to the precise and continuous control required to effectively manipulate the deformable nature of these objects. In this paper, we propose a new framework to control and maintain the shape of deformable linear objects with two robot manipulators utilizing environmental contacts. The framework is composed of a shape planning algorithm which automatically generates appropriate positions to place fixtures, and an object-centered skill engine which includes task and motion planning to control the motion and force of both robots based on the object status. The status of the deformable linear object is estimated online utilizing visual as well as force information. The framework manages to handle a cable routing task in real-world experiments with two Panda robots and especially achieves contact-aware and flexible clip fixing with challenging fixtures.

IROS Conference 2023 Conference Paper

Learning from Symmetry: Meta-Reinforcement Learning with Symmetrical Behaviors and Language Instructions

Xiangtong Yao
Zhenshan Bing
Genghang Zhuang
Kejia Chen 0005
Hongkuan Zhou
Kai Huang 0001
Alois C. Knoll

Meta-reinforcement learning (meta-RL) is a promising approach that enables the agent to learn new tasks quickly. However, most meta-RL algorithms show poor generalization in multi-task scenarios due to the insufficient task information provided only by rewards. Language-conditioned meta-RL improves the generalization capability by matching language instructions with the agent's behaviors. While both behaviors and language instructions have symmetry, which can speed up human learning of new knowledge. Thus, combining symmetry and language instructions into meta-RL can help improve the algorithm's generalization and learning efficiency. We propose a dual-MDP meta-reinforcement learning method that enables learning new tasks efficiently with symmetrical behav-iors and language instructions. We evaluate our method in mul-tiple challenging manipulation tasks, and experimental results show that our method can greatly improve the generalization and learning efficiency of meta-reinforcement learning. Videos are available at https://tumi6robot.wixsite.com/symmetry/.

AAAI Conference 2023 Conference Paper

Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning

Mingyang Wang
Zhenshan Bing
Xiangtong Yao
Shuai Wang
Huang Kai
Hang Su
Chenguang Yang
Alois Knoll

Meta-reinforcement learning enables artificial agents to learn from related training tasks and adapt to new tasks efficiently with minimal interaction data. However, most existing research is still limited to narrow task distributions that are parametric and stationary, and does not consider out-of-distribution tasks during the evaluation, thus, restricting its application. In this paper, we propose MoSS, a context-based Meta-reinforcement learning algorithm based on Self-Supervised task representation learning to address this challenge. We extend meta-RL to broad non-parametric task distributions which have never been explored before, and also achieve state-of-the-art results in non-stationary and out-of-distribution tasks. Specifically, MoSS consists of a task inference module and a policy module. We utilize the Gaussian mixture model for task representation to imitate the parametric and non-parametric task variations. Additionally, our online adaptation strategy enables the agent to react at the first sight of a task change, thus being applicable in non-stationary tasks. MoSS also exhibits strong generalization robustness in out-of-distributions tasks which benefits from the reliable and robust task representation. The policy is built on top of an off-policy RL algorithm and the entire network is trained completely off-policy to ensure high sample efficiency. On MuJoCo and Meta-World benchmarks, MoSS outperforms prior works in terms of asymptotic performance, sample efficiency (3-50x faster), adaptation efficiency, and generalization robustness on broad and diverse task distributions.

PDF Details DOI

ICRA Conference 2023 Conference Paper

Meta-Reinforcement Learning via Language Instructions

Zhenshan Bing
Alexander W. Koch
Xiangtong Yao
Kai Huang 0001
Alois C. Knoll

Although deep reinforcement learning has recently been very successful at learning complex behaviors, it requires a tremendous amount of data to learn a task. One of the fundamental reasons causing this limitation lies in the nature of the trial-and-error learning paradigm of reinforcement learning, where the agent communicates with the environment and pro-gresses in the learning only relying on the reward signal. This is implicit and rather insufficient to learn a task well. On the con-trary, humans are usually taught new skills via natural language instructions. Utilizing language instructions for robotic motion control to improve the adaptability is a recently emerged topic and challenging. In this paper, we present a meta-RL algorithm that addresses the challenge of learning skills with language instructions in multiple manipulation tasks. On the one hand, our algorithm utilizes the language instructions to shape its in-terpretation of the task, on the other hand, it still learns to solve task in a trial-and-error process. We evaluate our algorithm on the robotic manipulation benchmark (Meta-World) and it significantly outperforms state-of-the-art methods in terms of training and testing task success rates. Codes are available at https://tumi6robot.wixsite.com/million.

ICLR Conference 2023 Conference Paper

Selective Frequency Network for Image Restoration

Yuning Cui 0001
Yi Tao
Zhenshan Bing
Wenqi Ren
Xinwei Gao
Xiaochun Cao
Kai Huang 0001
Alois C. Knoll

Image restoration aims to reconstruct the latent sharp image from its corrupted counterpart. Besides dealing with this long-standing task in the spatial domain, a few approaches seek solutions in the frequency domain in consideration of the large discrepancy between spectra of sharp/degraded image pairs. However, these works commonly utilize transformation tools, e.g., wavelet transform, to split features into several frequency parts, which is not flexible enough to select the most informative frequency component to recover. In this paper, we exploit a multi-branch and content-aware module to decompose features into separate frequency subbands dynamically and locally, and then accentuate the useful ones via channel-wise attention weights. In addition, to handle large-scale degradation blurs, we propose an extremely simple decoupling and modulation module to enlarge the receptive field via global and window-based average pooling. Integrating two developed modules into a U-Net backbone, the proposed Selective Frequency Network (SFNet) performs favorably against state-of-the-art algorithms on five image restoration tasks, including single-image defocus deblurring, image dehazing, image motion deblurring, image desnowing, and image deraining.

IROS Conference 2023 Conference Paper

Smooth Stride Length Change of Rat Robot with a Compliant Actuated Spine Based on CPG Controller

Yuhong Huang
Zhenshan Bing
Zitao Zhang
Kai Huang 0001
Fabrice O. Morin
Alois C. Knoll

The aim of this research is to investigate the relationship between spinal flexion and quadruped locomotion in a rat robot equipped with a compliant spine, controlled by a central pattern generator (CPG). The study reveals that spinal flexion can enhance limb stride length, but it may also cause significant and unexpected motion disturbances during stride length variations. To address this issue, this paper proposes a CPG model driven by spinal flexion and a novel oscillator that incorporates a circular limit cycle and accounts for the anticipated stride length transition process. This approach effectively matches the torque change with the dynamics of stride length changes, leading to lower energy consumption. Extensive simulations are conducted to evaluate the efficacy of the proposed oscillator and compare it with the original kinetic model and other CPG models. The results demonstrate that the designed CPG model with the proposed oscillator yields smoother gait transitions during stride length variations and reduces energy consumption.

IROS Conference 2022 Conference Paper

A Biologically-Inspired Simultaneous Localization and Mapping System Based on LiDAR Sensor

Genghang Zhuang
Zhenshan Bing
Yuhong Huang
Kai Huang 0001
Alois C. Knoll

Simultaneous localization and mapping (SLAM) is one of the essential techniques and functionalities used by robots to perform autonomous navigation tasks. Inspired by the rodent hippocampus, this paper presents a biologically inspired SLAM system based on a LiDAR sensor using a hippocampal model to build a cognitive map and estimate the robot pose in indoor environments. Based on the biologically inspired models mimicking boundary cells, place cells, and head direction cells, the SLAM system using LiDAR point cloud data is capable of leveraging the self-motion cues from the LiDAR odometry and the boundary cues from the LiDAR boundary cells to build a cognitive map and estimate the robot pose. Experiment results show that with the LiDAR boundary cells the proposed SLAM system greatly outperforms the camera-based brain-inspired method in both simulation and indoor environments, and is competitive with the conventional LiDAR-based SLAM methods.

IROS Conference 2022 Conference Paper

An Adaptive Approach to Whole-Body Balance Control of Wheel-Bipedal Robot Ollie

Jingfan Zhang
Shuai Wang 0007
Haitao Wang
Jie Lai
Zhenshan Bing
Yu Jiang
Yu Zheng 0001
Zhengyou Zhang

The wheel-bipedal robot has the advantages of both wheeled robots and legged robots, but as a cost, it is more challenging to perform flexible movements in various surroundings while keeping it balanced. The inaccurate dynamics of the robot makes the balance problem even more intractable. To solve this problem, the robot Ollie is used as a testbed. The whole-body control (WBC) framework is adopted to enhance the dexterity of the robot with multiple degrees of freedom in the task space. Moreover, a learning-based adaptive technique is applied to assist the WBC such that the balance controller can be designed in the absence of the accurate dynamics. Physical experiments demonstrate that the robot can manage various actions, with the help of the combination of the WBC and the learning-based adaptive technique.

IROS Conference 2022 Conference Paper

Enhanced Quadruped Locomotion of a Rat Robot Based on the Lateral Flexion of a Soft Actuated Spine

Yuhong Huang
Zhenshan Bing
Florian Walter
Alex Rohregger
Zitao Zhang
Kai Huang 0001
Fabrice O. Morin
Alois C. Knoll

In nature, the movement of quadrupeds is completed under the combined action of the spine and the legs. Inspired by this, this paper explores the effect of a lateral flexing spine on the locomotion of a rat robot. Benefiting from the regular lateral flexion of a soft actuated spine, the rat robot exhibits enhance step length of its hind legs and increased translational velocity by coordinating the opposite movements of the left and right sides. Furthermore, this paper introduces a mathematical model of the effect of the flexible spine on the robot velocity. Finally, extensive experiments are conducted in simulations and on the physical rat robot. Compared with the locomotion without a flexing spine, the simulation results show that the velocity of the robot can be increased up to 218. 29%, which is in line with the theoretical results from the proposed mathematical model. Limited by the gap between simulation and the real world, the experiment results of the physical rat robot show a slight performance than the theoretical results. But the physical rat robot can still enhance its translational velocity with the help of a lateral flexing spine.

IROS Conference 2020 Conference Paper

Target Tracking Control of a Wheel-less Snake Robot Based on a Supervised Multi-layered SNN

Zhuangyi Jiang
Richard Otto
Zhenshan Bing
Kai Huang 0001
Alois C. Knoll

The snake-like robot without wheels is a bio-inspired robot whose high degree of freedom results in a challenge in autonomous locomotion control. The use of a Spiking Neural Network (SNN) which is a biologically plausible artificial neural network can help to achieve the autonomous locomotion behavior of snake robots in an energy-efficient manner. Approaches that use an SNN without hidden layers have been applied in the single-target tracking task. However, due to the complexity of the 3D gaits on a wheel-less snake robot and the imprecision of the pose control while in motion, they have some fluctuation that adversely affects their performances. In this work, we design two multi-layered SNNs with different topology for a wheel-less snake robot to track a certain moving object. The visual signals obtained from a Dynamic Vision Sensor (DVS) are fed into the SNN to drive the locomotion controller. Furthermore, the Reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) learning rule is utilized to train the SNN end-to-end. Compared to the SNN without hidden layers, the proposed multi-layered SNN with a separated hidden layer shows its advantage in terms of robustness.

ICRA Conference 2019 Conference Paper

End to End Learning of a Multi-Layered Snn Based on R-Stdp for a Target Tracking Snake-Like Robot

Zhenshan Bing
Zhuangyi Jiang
Long Cheng 0007
Caixia Cai
Kai Huang 0001
Alois C. Knoll

This paper introduces an end-to-end learning approach based on Reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) for a multi-layered spiking neural network (SNN). As a case study, a snake-like robot is used as an agent to perform target tracking tasks on the basis of our proposed approach. Since the key of R-STDP is to use rewards to modulate synapse strengthens, we first propose a general way to propagate the reward back through a multi-layered SNN. Upon the proposed approach, we build up an SNN controller that drives a snake-like robot for performing target tracking tasks. We demonstrate the practicability and advantage of our approach in terms of lateral tracking accuracy by comparing it to other state-of-the-art learning algorithms for SNNs based on R-STDP.

IJCAI Conference 2019 Conference Paper

Energy-Efficient Slithering Gait Exploration for a Snake-Like Robot Based on Reinforcement Learning

Zhenshan Bing
Christian Lemke
Zhuangyi Jiang
Kai Huang
Alois Knoll

Similar to their counterparts in nature, the flexible bodies of snake-like robots enhance their movement capability and adaptability in diverse environments. However, this flexibility corresponds to a complex control task involving highly redundant degrees of freedom, where traditional model-based methods usually fail to propel the robots energy-efficiently. In this work, we present a novel approach for designing an energy-efficient slithering gait for a snake-like robot using a model-free reinforcement learning (RL) algorithm. Specifically, we present an RL-based controller for generating locomotion gaits at a wide range of velocities, which is trained using the proximal policy optimization (PPO) algorithm. Meanwhile, a traditional parameterized gait controller is presented and the parameter sets are optimized using the grid search and Bayesian optimization algorithms for the purposes of reasonable comparisons. Based on the analysis of the simulation results, we demonstrate that this RL-based controller exhibits very natural and adaptive movements, which are also substantially more energy-efficient than the gaits generated by the parameterized controller. Videos are shown at https: //videoviewsite. wixsite. com/rlsnake.

ICRA Conference 2019 Conference Paper

Mixed Frame-/Event-Driven Fast Pedestrian Detection

Zhuangyi Jiang
Pengfei Xia
Kai Huang 0001
Walter Stechele
Guang Chen 0001
Zhenshan Bing
Alois C. Knoll

Pedestrian detection has attracted enormous research attention in the field of Intelligent Transportation System (ITS) due to that pedestrians are the most vulnerable traffic participants. So far, almost all pedestrian detection solutions are based on the conventional frame-based camera. However, they cannot perform very well in scenarios with bad light condition and high-speed motion. In this work, a Dynamic and Active Pixel Sensor (DAVIS), whose two channels concurrently output conventional gray-scale frames and asynchronous low-latency temporal contrast events of light intensity, was first used to detect pedestrians in a traffic monitoring scenario. Data from two camera channels were fed into Convolutional Neural Networks (CNNs) including three YOLOv3 models and three YOLO-tiny models to gather bounding boxes of pedestrians with respective confidence map. Furthermore, a confidence map fusion method combining the CNN-based detection results from both DAVIS channels was proposed to obtain higher accuracy. The experiments were conducted on a custom dataset collected on TUM campus. Benefiting from the high speed, low latency and wide dynamic range of the event channel, our method achieved higher frame rate and lower latency than those only using a conventional camera. Additionally, it reached higher average precision by using the fusion approach.

ICRA Conference 2018 Conference Paper

End to End Learning of Spiking Neural Network Based on R-STDP for a Lane Keeping Vehicle

Zhenshan Bing
Claus Meschede
Kai Huang 0001
Guang Chen 0001
Florian Röhrbein
Mahmoud Akl
Alois C. Knoll

Learning-based methods have demonstrated clear advantages in controlling robot tasks, such as the information fusion abilities, strong robustness, and high accuracy. Meanwhile, the on-board systems of robots have limited computation and energy resources, which are contradictory with state-of-the-art learning approaches. They are either too lightweight to solve complex problems or too heavyweight to be used for mobile applications. On the other hand, training spiking neural networks (SNNs) with biological plausibility has great potentials of performing fast computation and energy efficiency. However, the lack of effective learning rules for SNNs impedes their wide usage in mobile robot applications. This paper addresses the problem by introducing an end to end learning approach of spiking neural networks for a lane keeping vehicle. We consider the reward-modulated spike-timing-dependent-plasticity (R-STDP) as a promising solution in training SNNs, since it combines the advantages of both reinforcement learning and the well-known STDP. We test our approach in three scenarios that a Pioneer robot is controlled to keep lanes based on an SNN. Specifically, the lane information is encoded by the event data from a neuromorphic vision sensor. The SNN is constructed using R-STDP synapses in an all-to-all fashion. We demonstrate the advantages of our approach in terms of the lateral localization accuracy by comparing with other state-of-the-art learning algorithms based on SNNs.

ICRA Conference 2017 Conference Paper

CPG-based control of smooth transition for body shape and locomotion speed of a snake-like robot

Zhenshan Bing
Long Cheng 0007
Kai Huang 0001
Mingchuan Zhou
Alois C. Knoll

In this paper, a lightweight central pattern generator(CPG) model is designed for a snake-like robot, to achieve smooth transition of body shape and locomotion speed. First, based on the convergence behavior of the gradient system, a lightweight CPG model with fast computing time is designed and compared with other widely adopted CPG models. Then, the body shape and locomotion speed transitions in rolling gait are simulated based on the proposed CPG model. Compared with the sinusoid-based method, a smooth transition process can be achieved, without generating undesired movement or abnormal torque. Finally, extensive prototype experiments are conducted to demonstrate that the CPG-based control can effectively ensure smooth transition process and avoid abnormal torque, when the body shape and locomotion speed are changed.

IROS Conference 2017 Conference Paper

Towards autonomous locomotion: Slithering gait design of a snake-like robot for target observation and tracking

Zhenshan Bing
Long Cheng 0007
Kai Huang 0001
Zhuangyi Jiang
Guang Chen 0001
Florian Röhrbein
Alois C. Knoll

In this paper, a biologically inspired 3D slithering gait for a snake-like robot is designed and implemented for the purpose of target tracking. First, by balancing the forward speed and the stability of the robot, a straight slithering gait is modelled, under which the robot can march straight, fast, and stably. Then, for the purpose of steering, the straight slithering gait is modified into a biased slithering gait. The relationship between turning radius and gait parameters is analyzed by the resistive force theory. With the head composition algorithm, we investigate the orientation problem of the snake robot's head module to obtain stable visual information during the locomotion process. Finally, with the guidance of the vision sensor mounted in the head module, target tracking simulations and prototype experiments are conducted to demonstrate the practicality and effectiveness of the slithering gait in autonomous locomotion scenarios.