Arrow Research search

Author name cluster

Martin Jägersand

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

56 papers
2 author rows

Possible papers

56

RLJ Journal 2025 Journal Article

Efficient Morphology-Aware Policy Transfer to New Embodiments

  • Michael Przystupa
  • Hongyao Tang
  • Glen Berseth
  • Mariano Phielipp
  • Santiago Miret
  • Martin Jägersand
  • Matthew E. Taylor

Morphology-aware policy learning is a means of enhancing policy sample efficiency by aggregating data from multiple agents. These types of policies have previously been shown to help generalize over dynamic, kinematic, and limb configuration variations between agent morphologies. Unfortunately, these policies still have sub-optimal zero-shot performance compared to end-to-end finetuning on morphologies at deployment. This limitation has ramifications in practical applications such as robotics because further data collection to perform end-to-end finetuning can be computationally expensive. In this work, we investigate combining morphology-aware pretraining with \textit{parameter efficient finetuning} (PEFT) techniques to help reduce the learnable parameters necessary to specialize a morphology-aware policy to a target embodiment. We compare directly tuning sub-sets of model weights, input learnable adapters, and prefix tuning techniques for online finetuning. Our analysis reveals that PEFT techniques in conjunction with policy pre-training generally help reduce the number of samples to necessary to improve a policy compared to training models end-to-end from scratch. We further find that tuning as few as less than 1\% of total parameters will improve policy performance compared the zero-shot performance of the base pretrained a policy.

PDF Details

IROS Conference 2025 Conference Paper

Interpreting Behaviors and Geometric Constraints as Knowledge Graphs for Robot Manipulation Control

  • Chen Jiang
  • Allie Wang
  • Martin Jägersand

In this paper, we investigate the feasibility of using knowledge graphs to interpret actions and behaviors for robot manipulation control. Equipped with an uncalibrated visual servoing controller, we propose to use robot knowledge graphs to unify behavior trees and geometric constraints, conceptualizing robot manipulation control as semantic events. The robot knowledge graphs not only preserve the advantages of behavior trees in scripting actions and behaviors, but also offer additional benefits of mapping natural interactions between concepts and events, which enable knowledgeable explanations of the manipulation contexts. Through real-world evaluations, we demonstrate the flexibility of the robot knowledge graphs to support explainable robot manipulation control.

Details

ICRA Conference 2025 Conference Paper

Point and Go: Intuitive Reference Frame Reallocation in Mode Switching for Assistive Robotics

  • Allie Wang
  • Chen Jiang
  • Michael Przystupa
  • Justin Valentine
  • Martin Jägersand

Operating high degree of freedom robots can be difficult for users of wheelchair mounted robotic manipulators. Mode switching in Cartesian space has several drawbacks such as unintuitive control reference frames, separate translation and orientation control, and limited movement capabilities that hinder performance. We propose Point and Go mode switching, which reallocates the Cartesian mode switching reference frames into a more intuitive action space comprised of new translation and rotation modes. We use a novel sweeping motion to point the gripper, which defines the new translation axis along the robot base frame's horizontal plane. This creates an intuitive ‘point and go’ translation mode that allows the user to easily perform complex, human-like movements without switching control modes. The system's rotation mode combines position control with a refined endeffector oriented frame that provides precise and consistent robot actions in various end-effector poses. We verified its effectiveness through initial experiments, followed by a three-task user study that compared our method to Cartesian mode switching and a state of the art learning method. Results show that Point and Go mode switching reduced completion times by 31%, pauses by 41%, and mode switches by 33%, while receiving significantly favorable responses in user surveys.

Details

ICRA Conference 2025 Conference Paper

Robot Manipulation in Salient Vision Through Referring Image Segmentation and Geometric Constraints

  • Chen Jiang
  • Allie Wang
  • Martin Jägersand

In this paper, we perform robot manipulation activities in real-world environments with language contexts by integrating a compact referring image segmentation model into the robot's perception module. First, we propose CLIPU 2 Net, a lightweight referring image segmentation model designed for fine-grain boundary and structure segmentation from language expressions. Then, we deploy the model in an eye-in-hand visual servoing system to enact robot control in the real world. The key to our system is the representation of salient visual information as geometric constraints, linking the robot's visual perception to actionable commands. Experimental results on 46 real-world robot manipulation tasks demonstrate that our method outperforms traditional visual servoing methods relying on labor-intensive feature annotations, excels in fine-grain referring image segmentation with a compact decoder size of 6. 6 MB, and supports robot control across diverse contexts.

Details

ICRA Conference 2024 Conference Paper

CLIPUNetr: Assisting Human-robot Interface for Uncalibrated Visual Servoing Control with CLIP-driven Referring Expression Segmentation

  • Chen Jiang
  • Yuchen Yang
  • Martin Jägersand

The classical human-robot interface in uncalibrated image-based visual servoing (UIBVS) relies on either human annotations or semantic segmentation with categorical labels. Both methods fail to match natural human communication and convey rich semantics in manipulation tasks as effectively as natural language expressions. In this paper, we tackle this problem by using referring expression segmentation, which is a prompt-based approach, to provide more in-depth information for robot perception. To generate high-quality segmentation predictions from referring expressions, we propose CLIPUNetr - a new CLIP-driven referring expression segmentation network. CLIPUNetr leverages CLIP’s strong vision-language representations to segment regions from referring expressions, while utilizing its "U-shaped" encoder-decoder architecture to generate predictions with sharper boundaries and finer structures. Furthermore, we propose a new pipeline to integrate CLIPUNetr into UIBVS and apply it to control robots in real-world environments. In experiments, our method improves boundary and structure measurements by an average of 120% and can successfully assist real-world UIBVS control in an unstructured manipulation environment.

Details

IROS Conference 2024 Conference Paper

Immersive Human-in-the-Loop Control: Real-Time 3D Surface Meshing and Physics Simulation

  • Sait Aktürk
  • Justin Valentine
  • Junaid Ahmad
  • Martin Jägersand

This paper introduces the TactiMesh Teleoperator Interface (TTI), a novel predictive visual and haptic system designed explicitly for human-in-the-loop robot control using a head-mounted display (HMD). By employing simultaneous localization and mapping (SLAM) in tandem with a space carving method (CARV), TTI creates a real-time 3D surface mesh of remote environments from an RGB camera mounted on a Barrett WAM arm. The generated mesh is integrated into a physics simulator, featuring a digital twin of the WAM robot arm to create a virtual environment. In this virtual environment, TTI provides haptic feedback directly in response to the operator’s movements, eliminating the problem with delayed response from the haptic follower robot. Furthermore, texturing the 3D mesh with keyframes from SLAM allows the operator to control the viewpoint of their Head Mounted Display (HMD) independently of the arm-mounted robot camera, giving a better visual immersion and improving manipulation speed. Incorporating predictive visual and haptic feedback significantly improves tele-operation in applications such as search and rescue, inspection, and remote maintenance.

Details

IROS Conference 2024 Conference Paper

Local Linearity is All You Need (in Data-Driven Teleoperation)

  • Michael Przystupa
  • Gauthier Gidel
  • Matthew E. Taylor
  • Martin Jägersand
  • Justus H. Piater
  • Samuele Tosatto

One of the critical aspects of assistive robotics is to provide a control system of a high-dimensional robot from a low-dimensional user input (i. e. a 2D joystick). Data-driven teleoperation seeks to provide an intuitive user interface called an action map to map the low dimensional input to robot velocities from human demonstrations. Action maps are machine learning models trained on robotic demonstration data to map user input directly to desired movements as opposed to aspects of robot pose ("move to cup or pour content" vs. "move along x- or y-axis"). Many works have investigated nonlinear action maps with multi-layer perceptrons, but recent work suggests that local-linear neural approximations provide better control of the system. However, local linear models assume actions exist on a linear subspace and may not capture nuanced motions in training data. In this work, we hypothesize that local-linear neural networks are effective because they make the action map odd w. r. t. the user input, enhancing the intuitiveness of the controller. Based on this assumption, we propose two nonlinear means of encoding odd behavior that do not constrain the action map to a local linear function. However, our analysis reveals that these models effectively behave like local linear models for relevant mappings between user joysticks and robot movements. We support this claim in simulation, and show on a realworld use case that there is no statistical benefit of using non-linear maps, according to the users experience. These negative results suggest that further investigation into model architectures beyond local linear models may offer diminishing returns for improving user experience in data-driven teleoperation systems.

Details

RLJ Journal 2024 Journal Article

Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning

  • Gautham Vasan
  • Yan Wang
  • Fahim Shahriar
  • James Bergstra
  • Martin Jägersand
  • A. Rupam Mahmood

Many real-world robot learning problems, such as pick-and-place or arriving at a destination, can be seen as a problem of reaching a goal state as soon as possible. These problems, when formulated as episodic reinforcement learning tasks, can easily be specified to align well with our intended goal: -1 reward every time step with termination upon reaching the goal state (termed $\textit{minimum-time}$ tasks). Despite this simplicity, such formulations are often overlooked in favor of dense rewards due to their perceived difficulty and lack of informativeness. Our studies contrast the two reward paradigms, revealing that the minimum-time task specification not only facilitates learning higher-quality policies but can also surpass dense-reward-based policies on their own performance metrics. Crucially, we also identify the goal-hit rate of the initial policy as a robust early indicator for learning success in such sparse feedback settings. Finally, using four distinct real-robotic platforms, we show that it is possible to learn pixel-based policies from scratch within two to three hours using constant negative rewards. Our video demo can be found here: https://youtu.be/a6zlVUuKzBc

PDF Details

IROS Conference 2023 Conference Paper

Deep Probabilistic Movement Primitives with a Bayesian Aggregator

  • Michael Przystupa
  • Faezeh Haghverd
  • Martin Jägersand
  • Samuele Tosatto

Movement primitives are trainable parametric models that reproduce robotic movements starting from a limited set of demonstrations. Previous works proposed simple linear models that exhibited high sample efficiency and generalization power by allowing temporal modulation of move-ments (reproducing movements faster or slower), blending (merging two movements into one), via-point conditioning (constraining a movement to meet some particular via-points) and context conditioning (generation of movements based on an observed variable, e. g. , position of an object). Previous works have proposed neural network-based motor primitive models, having demonstrated their capacity to perform tasks with some forms of input conditioning or time-modulation representations. However, there has not been a single unified deep movement primitive's model proposed that is capable of all previous operations, limiting neural movement primitive's potential applications. This paper proposes a deep movement primitive architecture that encodes all the operations above and uses a Bayesian context aggregator that allows a more sound context conditioning and blending. Our results demonstrate our approach can scale to reproduce complex motions on a larger variety of input choices compared to baselines while maintaining operations of linear movement primitives provide.

Details

IROS Conference 2023 Conference Paper

Dynamic Decision Frequency with Continuous Options

  • Amirmohammad Karimi
  • Jun Jin 0001
  • Jun Luo 0009
  • A. Rupam Mahmood
  • Martin Jägersand
  • Samuele Tosatto

In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The duration between decisions becomes a crucial hyperparameter, as setting it too short may increase the problem's difficulty by requiring the agent to make numerous decisions to achieve its goal while setting it too long can result in the agent losing control over the system. However, physical systems do not necessarily require a constant control frequency, and for learning agents, it is often preferable to operate with a low frequency when possible and a high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. These options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. We demonstrate the effectiveness of CTCO by comparing its performance to classical RL and temporal-abstraction RL methods on simulated continuous control tasks with various action-cycle times. We show that our algorithm's performance is not affected by the choice of environment interaction frequency. Furthermore, we demonstrate the efficacy of CTCO in facilitating exploration in a real-world visual reaching task for a 7 DOF robotic arm with sparse rewards.

Details

ICRA Conference 2023 Conference Paper

Learning State Conditioned Linear Mappings for Low-Dimensional Control of Robotic Manipulators

  • Michael Przystupa
  • Kerrick Johnstonbaugh
  • Zichen Zhang 0001
  • Laura Petrich
  • Masood Dehghan
  • Faezeh Haghverd
  • Martin Jägersand

Identifying an appropriate task space can simplify solving robotic manipulation problems. One solution is deploying control algorithms in a learned low-dimensional action space. Linear and nonlinear action mapping methods have trade-offs between simplicity and the ability to express motor commands outside of a single low-dimensional subspace. We propose that learning local linear action representations can achieve both of these benefits. Our state-conditioned linear maps ensure that for any given state, the high-dimensional robotic actuation is linear in the low-dimensional actions. As the robot state evolves, so do the action mappings, so that necessary motions can be performed during a task. These local linear representations guarantee desirable theoretical properties by design. We validate these findings empirically through two user studies. Results suggest state-conditioned linear maps outperform conditional autoencoder and PCA baselines on a pick-and-place task and perform comparably to mode switching in a more complex pouring task.

Details

ICRA Conference 2022 Conference Paper

A Quantitative Analysis of Activities of Daily Living: Insights into Improving Functional Independence with Assistive Robotics

  • Laura Petrich
  • Jun Jin 0001
  • Masood Dehghan
  • Martin Jägersand

Wheelchair-mounted robotic manipulators have the potential to help the elderly and individuals living with disabilities carry out their activities of daily living (ADLs) independently. Robotics researchers focus on assistive tasks from the perspective of various control schemes and motion types, whereas, health research focuses on clinical assessment and rehabilitation, arguably leaving important differences between the two domains. In particular, there have been many studies on which activities are relevant to functional independence, but little is known quantitatively about the frequencies of ADLs that are typically carried out in everyday life. Understanding what activities are frequently carried out during the day can help guide the development and prioritization of robotic technology for in-home assistive robotic deployment. Robotics and health care communities have differing terms and taxonomies for representing tasks and motions; we aim to ameliorate taxonomic differences by consolidating quantitative task data with prior results from subjective task priority surveys. This study targets lifelogging databases, where we compute (i) daily activity task frequency from long-term low sampling frequency video and Internet of Things sensor data, and (ii) short term arm and hand movement data from video data of domestic tasks. In this work, we aim to provide deeper insights and meaningful guidelines to focus research and future developments in the field of assistive robotic manipulation that support the needs and performance requirements of the target population.

Details

ICRA Conference 2022 Conference Paper

Generalizable task representation learning from human demonstration videos: a geometric approach

  • Jun Jin 0001
  • Martin Jägersand

We study the problem of generalizable task learning from human demonstration videos without extra training on the robot or pre-recorded robot motions. Given a set of human demonstration videos showing a task with different objects/tools (categorical objects), we aim to learn a representation of visual observation that generalizes to categorical objects and enables efficient controller design. We propose to introduce a geometric task structure to the representation learning problem that geometrically encodes the task specification from human demonstration videos, and that enables generalization by building task specification correspondence between categorical objects. Specifically, we propose CoVGS-IL, which uses a graph-structured task function to learn task representations under structural constraints. Our method enables task generalization by selecting geometric features from different objects whose inner connection relationships define the same task in geometric constraints. The learned task representation is then transferred to a robot controller using uncalibrated visual servoing (UVS); thus, the need for extra robot training or pre-recorded robot motions is removed.

Details

ICRA Conference 2022 Conference Paper

Offline Learning of Counterfactual Predictions for Real-World Robotic Reinforcement Learning

  • Jun Jin 0001
  • Daniel Graves
  • Cameron Haigh
  • Jun Luo 0009
  • Martin Jägersand

We consider real-world reinforcement learning (RL) of robotic manipulation tasks that involve both visuomotor skills and contact-rich skills. We aim to train a policy that maps multimodal sensory observations (vision and force) to a manipulator's joint velocities under practical considerations. We propose to use offline samples to learn a set of general value functions (GVFs) that make counterfactual predictions from the visual inputs. We show that combining the offline learned counterfactual predictions with force feedbacks in online policy learning allows efficient reinforcement learning given only a terminal (success/failure) reward. We argue that the learned counterfactual predictions form a compact and informative representation that enables sample efficiency and provides auxiliary reward signals that guide online explorations towards contact-rich states. Various experiments in simulation and real-world settings were performed for evaluation. Recordings of the real-world robot training can be found via https://sites.google.com/view/realrl.

Details

ICRA Conference 2021 Conference Paper

A Generative Model-Based Predictive Display for Robotic Teleoperation

  • Bowen Xie
  • Mingjie Han
  • Jun Jin 0001
  • Martin Barczyk
  • Martin Jägersand

We propose a new generative model-based predictive display for robotic teleoperation over high-latency communication links. Our method is capable of rendering photo-realistic images of the scene to the human operator in real time from RGB-D images acquired by the remote robot. A preliminary exploration stage is used to build a coarse 3D map of the remote environment and to train a generative model, both of which are then used to generate photo-realistic images for the human operator based on the commanded pose of the robot. Data captured by the remote robot is used to dynamically update the 3D map, enabling teleoperation in the presence of new and relocated objects. Various experiments validate our proposed method’s performance and benefits over alternative methods.

Details

ICRA Conference 2021 Conference Paper

Analyzing Neural Jacobian Methods in Applications of Visual Servoing and Kinematic Control

  • Michael Przystupa
  • Masood Dehghan
  • Martin Jägersand
  • A. Rupam Mahmood

Designing adaptable control laws that can transfer between different robots is a challenge because of kinematic and dynamic differences, as well as in scenarios where external sensors are used. In this work, we empirically investigate a neural networks ability to approximate the Jacobian matrix for an application in Cartesian control schemes. Specifically, we are interested in approximating the kinematic Jacobian, which arises from kinematic equations mapping a manipulator’s joint angles to the end-effector’s location. We propose two different approaches to learn the kinematic Jacobian. The first method arises from visual servoing where we learn the kinematic Jacobian as an approximate linear system of equations from the k-nearest neighbors for a desired joint configuration. The second, motivated by forward models in machine learning, learns the kinematic behavior directly and calculates the Jacobian by differentiating the learned neural kinematics model. Simulation experimental results show that both methods achieve better performance than alternative data-driven methods for control, provide closer approximations to the proper kinematics Jacobian matrix, and on average produce better-conditioned Jacobian matrices. Real-world experiments were conducted on a Kinova Gen-3 lightweight robotic manipulator, which includes an uncalibrated visual servoing experiment, a practical application of our methods, as well as a 7-DOF point-to-point task highlighting that our methods are applicable on real robotic manipulators.

Details

IROS Conference 2020 Conference Paper

A Geometric Perspective on Visual Imitation Learning

  • Jun Jin 0001
  • Laura Petrich
  • Masood Dehghan
  • Martin Jägersand

We consider the problem of visual imitation learning without human kinesthetic teaching or teleoperation, nor access to an interactive reinforcement learning training environment. We present a geometric perspective to this problem where geometric feature correspondences are learned from one training video and used to execute tasks via visual servoing. Specifically, we propose VGS-IL (Visual Geometric Skill Imitation Learning), an end-to-end geometry-parameterized task concept inference method, to infer globally consistent geometric feature association rules from human demonstration video frames. We show that, instead of learning actions from image pixels, learning a geometry-parameterized task concept provides an explainable and invariant representation across demonstrator to imitator under various environmental settings. Moreover, such a task concept representation provides a direct link with geometric vision based controllers (e. g. visual servoing), allowing for efficient mapping of high-level task concepts to low-level robot actions.

Details

ICRA Conference 2020 Conference Paper

Mapless Navigation among Dynamics with Social-safety-awareness: a reinforcement learning approach from 2D laser scans

  • Jun Jin 0001
  • Nhat M. Nguyen
  • Nazmus Sakib
  • Daniel Graves
  • Hengshuai Yao
  • Martin Jägersand

We consider the problem of mapless collision-avoidance navigation where humans are present using 2D laser scans. Our proposed method uses ego-safety to measure collision from the robot's perspective and social-safety to measure the impact of robot's actions on surrounding pedestrians. Specifically, the social-safety part predicts the intrusion impact of the robot's action into the interaction area with surrounding humans. We train the policy using reinforcement learning on a simple simulator and directly evaluate the learned policy in Gazebo and real robot tests. Experiments show the learned policy smoothly transferred to different scenarios without any fine tuning. We observe that our method demonstrates time-efficient path planning behavior with high success rate in the mapless navigation task. Furthermore, we test our method in a navigation task among dynamic crowds, considering both low and high volume traffic. Our learned policy demonstrates cooperative behavior that actively drives our robot into traffic flows while showing respect to nearby pedestrians. Evaluation videos are at https://sites.google.com/view/ssw-batman.

Details

IROS Conference 2020 Conference Paper

Understanding Contexts Inside Robot and Human Manipulation Tasks through Vision-Language Model and Ontology System in Video Streams

  • Chen Jiang
  • Masood Dehghan
  • Martin Jägersand

Manipulation tasks in daily life, such as pouring water, unfold through human intentions. Being able to process contextual knowledge from these Activities of Daily Living (ADLs) over time can help us understand manipulation intentions, which are essential for an intelligent robot to transition smoothly between various manipulation actions. In this paper, to model the intended concepts of manipulation, we present a vision dataset under a strictly constrained knowledge domain for both robot and human manipulations, where manipulation concepts and relations are stored by an ontology system in a taxonomic manner. Furthermore, we propose a scheme to generate a combination of visual attentions and an evolving knowledge graph filled with commonsense knowledge. Our scheme works with real-world camera streams and fuses an attention-based Vision-Language model with the ontology system. The experimental results demonstrate that the proposed scheme can successfully represent the evolution of an intended object manipulation procedure for both robots and humans. The proposed scheme allows the robot to mimic human-like intentional behaviors by watching real-time videos. We aim to develop this scheme further for real-world robot intelligence in Human-Robot Interaction.

Details

ICRA Conference 2020 Conference Paper

Visual Geometric Skill Inference by Watching Human Demonstration

  • Jun Jin 0001
  • Laura Petrich
  • Zichen Zhang 0001
  • Masood Dehghan
  • Martin Jägersand

We study the problem of learning manipulation skills from human demonstration video by inferring the association relationships between geometric features. Motivation for this work stems from the observation that humans perform eye-hand coordination tasks by using geometric primitives to define a task while a geometric control error drives the task through execution. We propose a graph based kernel regression method to directly infer the underlying association constraints from human demonstration video using Incremental Maximum Entropy Inverse Reinforcement Learning (InMaxEnt IRL). The learned skill inference provides human readable task definition and outputs control errors that can be directly plugged into traditional controllers. Our method removes the need for tedious feature selection and robust feature trackers required in traditional approaches (e. g. feature-based visual ser-voing). Experiments show our method infers correct geometric associations even with only one human demonstration video and can generalize well under variance.

Details

ICRA Conference 2019 Conference Paper

Online Object and Task Learning via Human Robot Interaction

  • Masood Dehghan
  • Zichen Zhang 0001
  • Mennatullah Siam
  • Jun Jin 0001
  • Laura Petrich
  • Martin Jägersand

This work describes the development of a robotic system that acquires knowledge incrementally through human interaction where new objects and motions are taught on the fly. The robotic system developed was one of the five finalists in the KUKA Innovation Award competition and demonstrated during the Hanover Messe 2018 in Germany. The main contributions of the system are i) a novel incremental object learning module - a deep learning based localization and recognition system - that allows a human to teach new objects to the robot, ii) an intuitive user interface for specifying 3D motion task associated with the new object, and iii) a hybrid force-vision control module for performing compliant motion on an unstructured surface. This paper describes the implementation and integration of the main modules of the system and summarizes the lessons learned from the competition.

Details

ICRA Conference 2019 Conference Paper

Robot eye-hand coordination learning by watching human demonstrations: a task function approximation approach

  • Jun Jin 0001
  • Laura Petrich
  • Masood Dehghan
  • Zichen Zhang 0001
  • Martin Jägersand

We present a robot eye-hand coordination learning method that can directly learn visual task specification by watching human demonstrations. Task specification is represented as a task function, which is learned using inverse reinforcement learning(IRL [1]) by inferring a reward model from state transitions. The learned reward model is then used as continuous feedbacks in an uncalibrated visual servoing(UVS [2]) controller designed for the execution phase. Our proposed method can directly learn from raw videos, which removes the need for hand-engineered task specification. Benefiting from the use of a traditional UVS controller, the training on real robot only happens at initial Jacobian estimation which takes an average of 4-7 seconds for a new task. Besides, the learned policy is independent from a particular robot, thus has the potential of fast adapting to other robot platforms. Various experiments were designed to show that, for a task with certain DOFs, our method can adapt to task/environment changes in target positions, backgrounds, illuminations, and occlusions.

Details

ICRA Conference 2019 Conference Paper

Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting

  • Mennatullah Siam
  • Chen Jiang
  • Steven Weikai Lu
  • Laura Petrich
  • Mahmoud Gamal
  • Mohamed Elhoseiny
  • Martin Jägersand

Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels. We propose a novel teacher-student learning paradigm to teach robots about their surrounding environment. A two-stream motion and appearance “teacher” network provides pseudo-labels to adapt an appearance “student” network. The student network is able to segment the newly learned objects in other scenes, whether they are static or in motion. We also introduce a carefully designed dataset that serves the proposed HRI setup, denoted as (I)nteractive (V)ideo (O)bject (S)egmentation. Our IVOS dataset contains teaching videos of different objects, and manipulation tasks. Our proposed adaptation method outperforms the state-of-theart on DAVIS and FBMS with 6. 8% and 1. 2% in F-measure respectively. It improves over the baseline on IVOS dataset with 46. 1% and 25. 9% in mIoU.

Details

IROS Conference 2018 Conference Paper

Real-Time Edge Template Tracking via Homography Estimation

  • Xuebin Qin
  • Shida He
  • Zichen Zhang 0001
  • Masood Dehghan
  • Jun Jin 0001
  • Martin Jägersand

In this paper, we propose a novel real-time method for tracking planar edge templates. This method tracks an edge template by estimating its homography transformations with respect to the sampled edge pixels detected from the incoming frames. Particularly, we define a cost function based on a new feature map of the to-be-tracked edge template and optimize it by a Lucas-Kanade-like algorithm. The feature map is defined as the fourth root of the distance transform. Our method operates on just edges so that it is good at tracking those low textured targets, such as hollow targets (mug rim), thin targets (cable, ring) and non-Lambertian objects (disc). We validate and compare our method with four other methods on five newly collected real-world video sequences. The results achieves the lowest overall average error (1. 58 pixels) and also outperforms others in terms of success rate. The per frame processing time of about 30 ms proves that our method is acceptable in realtime applications. The code and dataset are publicly available at: http://webdocs.cs.ualberta.ca/~xuebin/.

Details

IROS Conference 2018 Conference Paper

Real-Time Segmentation with Appearance, Motion and Geometry

  • Mennatullah Siam
  • Sara Elkerdawy
  • Mostafa Gamal
  • Moemen Abdel-Razek
  • Martin Jägersand
  • Hong Zhang 0013

Real-time Segmentation is of crucial importance to robotics related applications such as autonomous driving, driving assisted systems, and traffic monitoring from unmanned aerial vehicles imagery. We propose a novel two-stream convolutional network for motion segmentation, which exploits flow and geometric cues to balance the accuracy and computational efficiency trade-offs. The geometric cues take advantage of the domain knowledge of the application. In case of mostly planar scenes from high altitude unmanned aerial vehicles (UAVs), homography compensated flow is used. While in the case of urban scenes in autonomous driving, with GPS/IMU sensory data available, sparse projected depth estimates and odometry information are used. The network provides 4. 7× speedup over the state of the art networks in motion segmentation from 153ms to 36ms, at the expense of a reduction in the segmentation accuracy in terms of pixel boundaries. This enables the network to perform real-time on a Jetson T×2. In order to recuperate some of the accuracy loss, geometric priors is used while still achieving a much improved computational efficiency with respect to the state-of-the-art. The usage of geometric priors improved the segmentation in UAV imagery by 5. 2 % using the metric of IoU over the baseline network. While on KITTI-MoSeg the sparse depth estimates improved the segmentation by 12. 5 % over the baseline. Our proposed motion segmentation solution is verified on the popular KITTI and VIVID datasets, with additional labels we have produced. The code for our work is publicly available at 1.

Details

ICRA Conference 2017 Conference Paper

Flexible virtual fixture interface for path specification in tele-manipulation

  • Camilo Perez Quintero
  • Masood Dehghan
  • Oscar Ramirez
  • Marcelo H. Ang
  • Martin Jägersand

We present the design and implementation of a flexible force-vision-based interface; allowing local operators to visually specify a path constraint to a remote robot manipulator in an on-line fashion during the teleoperation. Using bilateral and unilateral configurations, we compare our system to direct teleoperation through user studies. Three performance metrics (smoothness, error and execution time) and a subjective evaluation (NASA TLX) were used to quantify user performance. The trials show that our system outperforms direct teleoperation and reduces cognitive load. Our findings show that the performance of a unilateral teleop configuration with visual-force constraints surpass a bilateral teleop configuration in terms of displacement error and variance, as well as allowing users to complete tasks faster and with a smoother trajectory.

Details

IROS Conference 2017 Conference Paper

Incremental learning for robot perception through HRI

  • Sepehr Valipour
  • Camilo Perez Quintero
  • Martin Jägersand

Visual scene understanding is a crucial skill for robots, yet difficult to achieve. Recently, Convolutional Neural Networks (CNN), have shown success in this task. However, there is still a gap between their performance on image datasets and real-world robotics scenarios. In particular, a-priori training is on a bounded set of object categories, while in many unstructured tasks new objects are encountered. We present a novel paradigm for incrementally improving a robot's visual perception through active human-robot interaction. In this paradigm, the user introduces novel objects to the robot by means of pointing and voice commands. Given this information, the robot visually explores the object and adds images from it to re-train the perception module. Our method leverages state of the art Convolutional Neutal Networks — CNNs from offline batch learning, human guidance, robot exploration and incremental on-line learning.

Details

IROS Conference 2017 Conference Paper

Modular tracking framework: A fast library for high precision tracking

  • Abhineet Singh
  • Martin Jägersand

This paper presents MTF - a modular, extensible and highly efficient open source framework for registration based tracking targeted at robotics applications. It is implemented entirely in C++ and is designed from the ground up to easily integrate with systems that support any of several major vision and robotics libraries including OpenCV, ROS, ViSP and Eigen. It is also faster and more precise than other existing tracking systems. In order to establish the theoretical basis for its design, a new way to conceptualize registration based trackers is also introduced that decomposes tracking into three sub modules - Search Method, Appearance Model and State Space Model. Along with being a practical solution for fast and high precision tracking, this framework can also serve as a useful research tool by allowing existing and new methods for any of the sub modules to be studied better. Through extensive use of generic programming, the system makes it easy to plug in a new method for any of the sub modules so that it can not only be tested comprehensively with existing methods but also become immediately available for deployment in any project that uses the framework.

Details

IROS Conference 2017 Conference Paper

Real-time salient closed boundary tracking via line segments perceptual grouping

  • Xuebin Qin
  • Shida He
  • Camilo Perez Quintero
  • Abhineet Singh
  • Masood Dehghan
  • Martin Jägersand

This paper presents a novel real-time method for tracking salient closed boundaries from video image sequences. This method operates on a set of straight line segments that are produced by line detection. The tracking scheme is coherently integrated into a perceptual grouping framework in which the visual tracking problem is tackled by identifying a subset of these line segments and connecting them sequentially to form a closed boundary with the largest saliency and a certain similarity to the previous one. Specifically, we define a new tracking criterion which combines a grouping cost and an area similarity constraint. The proposed criterion makes the resulting boundary tracking more robust to local minima. To achieve real-time tracking performance, we use Delaunay Triangulation to build a graph model with the detected line segments and then reduce the tracking problem to finding the optimal cycle in this graph. This is solved by our newly proposed closed boundary candidates searching algorithm called “Bidirectional Shortest Path (BDSP)”. The efficiency and robustness of the proposed method are tested on real video sequences as well as during a robot arm pouring experiment.

Details

ICRA Conference 2016 Conference Paper

ViTa: Visual task specification interface for manipulation with uncalibrated visual servoing

  • Mona Gridseth
  • Oscar Ramirez
  • Camilo Perez Quintero
  • Martin Jägersand

We present a human robot interface (HRI) for semi-autonomous human-in-the-loop control, that aims to tackle some of the challenges for robotics in unstructured environments. Our HRI lets the user specify desired object alignments in an image editor as geometric overlays on images. The HRI is based on the technique of visual task specification [1], which provides a well studied theoretical framework. Tasks are completed using uncalibrated image-based visual servoing (UVS). Our interface is shown to be effective for a versatile set of tasks that span both coarse and fine manipulation. We complete tasks such as inserting a marker in its cap, inserting a small cube in a shape sorter, grasping a circular lid, following a line, grasping a screw, cutting along a line, picking and placing a box and grasping a cylinder using a Barrett WAM arm and hand.

Details

ICRA Conference 2015 Conference Paper

On-line reconstruction based predictive display in unknown environment

  • Huan Hu
  • Camilo Perez Quintero
  • Hanxu Sun
  • Martin Jägersand

In tele-robotics, time delay is a significant problem. When video feedback is delayed, operators adopt inefficient move-wait strategies, so system performance decreases. Predictive display (PD) is an effective solution to compensate for delays by graphics rendering of predicted visual feedback. Using advanced computer vision technology, we implemented a PD system based-on online real-time 3D reconstruction from monocular video. This paper describes the client-server system architecture. Experimental results indicate it can capture 3D models and render the predicted image in realistic applications covering outdoor rover operation on earth, Canadian Space Agency's (CSA) Mars analogue environment, UAV operation.

Details

ICRA Conference 2015 Conference Paper

Tracking benchmark and evaluation for manipulation tasks

  • Ankush Roy
  • Xi Zhang
  • Nina Wolleb
  • Camilo Perez Quintero
  • Martin Jägersand

In this paper we present a public dataset to evaluate trackers used for human and robot manipulation tasks. For these tasks both high DOF motion and high accuracy is needed. We describe in detail, both the process of recording the sequences and how ground truth data was generated for the videos. The videos are tagged with challenges that a tracker would face while tracking the object. As an initial example, we evaluate the performance of six published trackers [5], [11], [12], [13], [15], [6] and analyse their result. We describe a new evaluation metric to test sensitivity of trackers to speed. A total of 100 annotated and tagged sequences are reported. All the videos, ground truth data, original implementation of trackers and evaluation scripts are made publicly available on the website so others can extend the results on their trackers and evaluation.

Details

ICRA Conference 2015 Conference Paper

VIBI: Assistive vision-based interface for robot manipulation

  • Camilo Perez Quintero
  • Oscar Ramirez
  • Martin Jägersand

Upper-body disabled people can benefit from the use of robot-arms to perform every day tasks. However, the adoption of this kind of technology has been limited by the complexity of robot manipulation tasks and the difficulty in controlling a multiple-DOF arm using a joystick or a similar device. Motivated by this need, we present an assistive vision-based interface for robot manipulation. Our proposal is to replace the direct joystick motor control interface present in a commercial wheelchair mounted assistive robotic manipulator with a human-robot interface based on visual selection. The scene in front of the robot is shown on a screen, and the user can then select an object with our novel grasping interface. We develop computer vision and motion control methods that drive the robot to that object. Our aim is not to replace user control, but instead augment user capabilities through our system with different levels of semi-autonomy, while leaving the user with a sense that he/she is in control of the task. Two disabled pilot users, were involved at different stages of our research. The first pilot user during the interface design along with rehab experts. The second performed user studies along with an 8 subject control group to evaluate our interface. Our system reduces robot instruction from a 6-DOF task in continuous space to either a 2-DOF pointing task or a discrete selection task among objects detected by computer vision.

Details

ICRA Conference 2013 Conference Paper

SEPO: Selecting by pointing as an intuitive human-robot command interface

  • Camilo Perez Quintero
  • Romeo Tatsambon Fomena
  • Azad Shademan
  • Nina Wolleb
  • Travis Dick
  • Martin Jägersand

Pointing to indicate direction or position is one of the intuitive communication mechanisms used by humans in all life stages. Our aim is to develop a natural human-robot command interface using pointing gestures for human-robot interaction (HRI). We propose an interface based on the Kinect sensor for selecting by pointing (SEPO) in a 3D real-world situation, where the user points to a target object or location and the interface returns the 3D position coordinates of the target. Through our interface we perform three experiments to study precision and accuracy of human pointing in typical household scenarios: pointing to a “wall”, pointing to a “table”, and pointing to a “floor”. Our results prove that the proposed SEPO interface enables users to point and select objects with an average 3D position accuracy of 9: 6 cm in household situations.

Details

IROS Conference 2012 Conference Paper

Robust sampling-based planning for uncalibrated visual servoing

  • Azad Shademan
  • Martin Jägersand

Planning for vision-based robot control is a challenging open problem, especially in unstructured environments where models are not known a priori and sensor measurements contain errors and outliers. In this paper, we propose a statistically-robust randomized planning algorithm for a model-free eye-in-hand manipulator. The planner is built on the success and efficiency of the sampling-based planners, while incorporating robustness to outliers. In particular, we generalize the Rapidly-Exploring Random Tree (RRT) planner to the visual-motor space, the space that encodes both visual measurements and motor readings. The proposed planner is used in conjunction with a closed-loop visual control law. While the control law is entirely image based, the planner helps avoid joint limits, field-of-view constraints, and more importantly, visual occlusion of the target by unmodeled obstacles. The algorithm is validated in simulations as well as experiments with a WAM robot arm.

Details

IROS Conference 2012 Conference Paper

Unimodal asymmetric interface for teleoperation of mobile manipulators: A user study

  • Alejandro Hernandez Herdocia
  • Azad Shademan
  • Martin Jägersand

There is demand to develop methods and interfaces for teleoperation of complex systems in mission-critical applications. In this paper, we study three different methods to command a one-arm mobile manipulator from a 6-DOF input device capable of haptic feedback. The linkage between the master and the slave devices is asymmetric, that is, the input haptic interface (master) is much smaller and has different kinematics and dynamics from the robot arm and the mobile base (slave). Three different master-slave motion coordination schemes are compared here (1) workspace clutching, (2) differential end-zone, and (3) position/rate switching. We study repetitive user performance for seven subjects in a static Tower of Hanoi manipulation task and present single case studies for two mobile manipulation tasks: door opening and large-displacement Towers of Hanoi. Our experimental platform consists of a 4-DOF WAM (Whole Arm Manipulator) on a Segway RMP (Robotic Mobility Platform) controlled by a Phantom Omni haptic device. Cameras are used to relay scene images to the remote operator. The human stays in the loop throughout the entire task. The results obtained from user studies provide insight on how to interface and command a mobile manipulator.

Details

ICRA Conference 2010 Conference Paper

Performance evaluation of monocular predictive display

  • Adam Rachmielowski
  • Neil Birkbeck
  • Martin Jägersand

In teleoperation systems, operator performance is negatively affected by time-delayed visual feedback. Predictive display (PD) compensates for delays by providing synthesized visual feedback. While most existing PD methods rely on a priori models (e. g. , from laser range finding or stereo vision), recent work on monocular SLAM and SFM makes it possible to acquire PD models in single camera applications. In this work, we evaluate operator performance of PD visual feedback based on a coarse 3D model. We report the experimental results of 12 human tele-operators each performing 96 visual alignment tasks with a 300ms delay. Four operating modes are considered: delayed video (no PD), video-based PD using a stabilizing plane (homography), 3D model-based PD, and no delay (ground truth). The results indicate that vision-based PD (both plane and 3D model-based) is significantly better than delayed video. It reduced task completion time 40% and is nearly as good as the no delay condition. PD based on a sparse a 3D model was somewhat better than the simpler plane based method.

Details

IROS Conference 2010 Conference Paper

Predictive display for mobile manipulators in unknown environments using online vision-based monocular modeling and localization

  • David Lovi
  • Neil Birkbeck
  • Alejandro Hernandez Herdocia
  • Adam Rachmielowski
  • Martin Jägersand
  • Dana Cobzas

To tele-operate a robot, visual feedback is critical. However, communication channel latency can delay feedback to the point where the operator is impeded in performing his task. This work presents a vision-based “predictive display” system that compensates for visual delay. The approach is online and relatively uncalibrated, thus it has the advantage of being useful in unknown environments and many applications. From monocular eye-in-hand video, we incrementally compute a 3D graphics model of the robot site in real time using our new technique. The method exploits free-space/occlusion constraints on the scene to produce a physically consistent mesh. Novel vantage points are immediately rendered in response to the operator's control commands, without waiting for delayed video. We implement a full prototype tele-operation system where the operator controls, via a PHANTOM Omni device, a Barrett WAM robot mounted on a mobile Segway. Experiments with this setup validate the efficacy of the proposed approach. We demonstrate significant improvement in task completion time with predictive display on a real robot, while our previous related results were established only in simulation.

Details

ICRA Conference 2010 Conference Paper

Robust Jacobian estimation for uncalibrated visual servoing

  • Azad Shademan
  • Amir Massoud Farahmand
  • Martin Jägersand

This paper addresses robust estimation of the uncalibrated visual-motor Jacobian for an image-based visual servoing (IBVS) system. The proposed method does not require knowledge of model or system parameters and is robust to outliers caused by various visual tracking errors, such as occlusion or mis-tracking. Previous uncalibrated methods are not robust to outliers and assume that the visual-motor data belong to the underlying model. In unstructured environments, this assumption may not hold. Outliers to the visual-motor model may deteriorate the Jacobian, which can make the system unstable or drive the arm in the wrong direction. We propose to apply a statistically robust M-estimator to reject the outliers. We compare the quality of the robust Jacobian estimation with the least squares-based estimation. The effect of outliers on the estimation quality is studied through MATLAB simulations and eye-in-hand visual servoing experiments using a WAM arm. Experimental results show that the Jacobian estimated by robust M-estimation is robust when up to 40% of the visual-motor data are outliers.

Details

IROS Conference 2010 Conference Paper

Three-view uncalibrated visual servoing

  • Azad Shademan
  • Martin Jägersand

A new uncalibrated visual servoing approach for motion control of 6-DOF manipulators is presented. Instead of image features, the elements of the scene-independent trifocal tensor of three views is used as features for visual servoing. These trifocal features depend only on the camera projection matrices and can be retrieved from point correspondences across the three views: initial, current, and desired. The Jacobian matrix that relates joint velocities and variations of the trifocal features is estimated online using least-squares. The visual servoing performance is evaluated for various easy and challenging motions, such as the translational motion along the view axis and the large rotation around the view axis. Simulations with a 6-DOF eye-in-hand manipulator show that the proposed uncalibrated approach rapidly converges in all cases.

Details

ICRA Conference 2009 Conference Paper

Model-based and model-free reinforcement learning for visual servoing

  • Amir Massoud Farahmand
  • Azad Shademan
  • Martin Jägersand
  • Csaba Szepesvári

To address the difficulty of designing a controller for complex visual-servoing tasks, two learning-based uncalibrated approaches are introduced. The first method starts by building an estimated model for the visual-motor forward kinematic of the vision-robot system by a locally linear regression method. Afterwards, it uses a reinforcement learning method named Regularized Fitted Q-Iteration to find a controller (i. e. policy) for the system (model-based RL). The second method directly uses samples coming from the robot without building any intermediate model (model-free RL). The simulation results show that both methods perform comparably well despite not having any a priori knowledge about the robot.

Details

IROS Conference 2007 Conference Paper

Global visual-motor estimation for uncalibrated visual servoing

  • Amir Massoud Farahmand
  • Azad Shademan
  • Martin Jägersand

In this paper, we present two methods for the estimation of a globally valid visual-motor model of a robotic manipulator. In conventional uncalibrated visual servoing, the visuo-motor function is approximated locally with a Jacobian. However, for optimal task planning, or nonlinear controller design with global stability guarantee, one needs to know a model that provides some information about the behavior of the system over the whole workspace. Our presented methods remedy this drawback in uncalibrated visual servoing by incrementally building a global estimator based on the movement history. We implement two such methods. The first method is a K-nearest neighborhood regressor over Jacobian that uses previously estimated local models. The second method stores previous movements and computes an estimate of the Jacobian by solving a local least squares problem. Experimental results show that both methods provide better global estimation quality compared to the conventional local estimation method with much lower estimation variance.

Details

ICRA Conference 2007 Conference Paper

On with the Visuomotor Function: A 6DOF Adaptive Approach for Modeling Image-Based Variations and Visual Servoing

  • Simon Léonard
  • Martin Jägersand

In this paper, we proposes a visual servoing method that approximates the relation between the variations of image points and the variations of a stereo rig in Euclidian space. As with most image-based visual servoing methods, commands are expressed in the space of image features. However, instead of relating instantaneous image-based variations to instantaneous variations in Euclidian space, the visuomotor function relates arbitrary image-based variations to Euclidian transformations. The visuomotor function is approximated in real-time by using online estimation techniques. The system improves its performance with experience and is able to adapt to different configurations of the cameras or environment. Given the disparities between two sets of corresponding image points, the visuomotor function provides the Euclidian transformation the robot must execute in order to align the image coordinates.

Details

ICRA Conference 2006 Conference Paper

Adaptive Control for Estimating Translations from Image-based Variations

  • Simon Léonard
  • Martin Jägersand

We investigate the problem of learning the mapping between arbitrary image-based variations and variations in Euclidean space for the application of visual-servoing. We derive a linear formulation of the visuomotor function, which captures this relationship for three dimensional translations. The resulting expression defines how the origin of the world coordinate frame shifts in stereo images as the robot translates. The parameters of the visuomotor function are estimated online by using incremental least squares and are generalized to other coordinate frames origins by using a function approximation method. It follows that the system is able to estimate the three dimensional translations between pairs of stereo points without performing 3D reconstruction or requiring a specific coordinate system and is fully adaptive

Details

ICRA Conference 2005 Conference Paper

Incremental Learning for Mapping Image Variations to Actions

  • Simon Léonard
  • Martin Jägersand

In this paper we introduce a method to learn the global relationship between camera motion and its effect on a feature space. Drawing from progress in vision based motion control, we derive an image-based method that learns an approximation of the visuomotor function while performing visual servoing tasks. The result is a reactive agent that measures its performance in the image feature space and controls a robot with position error commands. Thus combining image-based references with position-based commands.

Details

ICRA Conference 2005 Conference Paper

Tracking and Predictive Display for a Remote Operated Robot using Uncalibrated Video

  • Dana Cobzas
  • Martin Jägersand

Delays in the visual feedback can seriously impact operator performance in telerobotics. In predictive display synthesized visual feedback is rendered immediately in response to operator motions. In this paper we present a system using a geometric and appearance model that is captured using structure-from-motion by an uncalibrated camera. The geometric model is integrated into a registration based tracking algorithm that allows stable tracking of full 3D pose of the robot. Experimentally we show that predictive scene views can be rendered with both high visual fidelity and metric accuracy.

Details

IROS Conference 2005 Conference Paper

Uncalibrated visual servoing using a biased Newton method for on-line singularity detection and avoidance

  • Masoud Shahamiri
  • Martin Jägersand

While in calibrated settings trajectories can be planned so to avoid singular or poorly observable configurations, in uncalibrated visual servoing in general a priori information about singularities (visual or robotic) may be unavailable. Instead we propose a method where trajectories are corrected online to avoid singular and near singular regions. Mathematically this is achieved using a so called nullspace-biased Newton step in a visual servoing with a Broyden type Jacobian estimation. The bias is applied so to first hand use (any) robot redundancy and thus not compromise the visually specified aspects of the trajectory. The closeness to a singular region is measured online from the estimated visual motor Jacobian. We also illustrate how to apply the bias method for simple visual obstacle avoidance. To show the practical applicability of our method we have applied it to Barrett WAM and PUMA560 manipulators and tested both numerous real trajectories, as well as run exhaustive simulations around critical configurations using a simulation model to confirm empirically that both safe and efficient trajectories are chosen around singular regions.

Details

IROS Conference 2004 Conference Paper

Learning based visual servoing

  • Simon Léonard
  • Martin Jägersand

This paper proposes a method for learning a hand-eye calibration and its application for visual servoing. The goal is to develop a technique that combines the strengths of existing visual servoing methods. Particularly, as in image-based visual servoing, the error is measured in the visual space while the motor command is position-based. Hence this method approximates the visuomotor function that relates variations in the visual space to variations in the motor space at a global scale. The method used for approximating the visuomotor function is derived from the field of reinforcement learning, making our hand-eye calibration autonomous, continuous and adaptable. The visuomotor function is modeled by a linear combination of polynomials, each spanning a non-mutually exclusive subset of the visual space. Each polynomial represents the utility of motor commands for the servoing task. The goal of the calibration is to approximate the parameters of these polynomials while the system interacts with its environment. Preliminary results include centering a target in the image in which the system learns the motor commands that eliminates the errors in the visual space and generalizes the result to neighboring states in the visual space, depths and motor commands.

Details

ICRA Conference 2003 Conference Paper

Image-based localization with depth-enhanced image map

  • Dana Cobzas
  • Hong Zhang 0013
  • Martin Jägersand

In this paper, we present an image-based robot incremental localization algorithm which uses a panoramic image-based map enhanced with depth from a laser range finder. The image-based map (model) contains both intensity information as well as sparse 3D geometric features. By assuming motion continuity, a robot can use the depth information in the image-model to project the relevant 3D model features, specifically vertical lines, of the environment to its camera coordinate frame. To determine its location, the robot first acquires an intensity image and then matches the 2D geometric features in the image with the projected model features. The first contribution of this research is that we avoid the difficult problem of full 3D reconstruction from images by employing a range sensor registered with respect to the intensity image sensor; secondly, we provide an algorithm that performs incremental robot localization using only 2D images. Experimental results in indoor map building and localization demonstrate the feasibility of our approach and evaluate the performance of the algorithm.

Details

ICRA Conference 2003 Conference Paper

Predictive display models for tele-manipulation from uncalibrated camera-capture of scene geometry and appearance

  • Keith Yerex
  • Dana Cobzas
  • Martin Jägersand

In tele-robotics delayed visual feedback to the human operator can degrade task performance significantly. To improve this, predictive display, uses a scene model to estimate and render immediate visual feedback based on the operator's control commands. Traditional predictive display involves the calibration and overlay of an a-priori model with the delayed real video feedback. In this paper we present an image-based method where the scene geometry and appearance is captured using structure-from-motion by an uncalibrated eye-in-hand camera mounted on the remote robot. The model is then compressed and transmitted to the operator site, where it is used to generate immediate feedback in response to the operators movements. Calibration problems are avoided since the model is captured by the same scene camera as is being simulated in the predictive display. We show experiments where we capture the appearance of a robot hand and transmit it over the network to the operator site where the model renders scene appearance change in response to operator viewpoint motion.

Details

IROS Conference 2003 Conference Paper

Predictive display system for tele-manipulation using image-based modeling and rendering

  • Zhenyuan Deng
  • Martin Jägersand

Using a combination of techniques from visual tracking, image-based rendering, distributed programming, and robot motion control, we present a predictive display system to improve local operator's performance in tele-manipulations. More precisely, we construct a networked distributed system allowing the robot motion control and predictive display function to be implemented in a single PVM (Parallel Virtual Machine) program running on both the operator and remote scene computer. We also integrate our system with real-time pose tracking of the operator to allow 3D rendering in HMD.

Details

ICRA Conference 2002 Conference Paper

A Comparative Analysis of Geometric and Image-Based Volumetric and Intensity Data Registration Algorithms

  • Dana Cobzas
  • Hong Zhang 0013
  • Martin Jägersand

We present and contrast four methods for registering 3D range data to 2D images. Two are calibration techniques that recover the rigid transformation between the sensor poses, based on point or line correspondences. The two others recover a direct, image based mapping between the data sets. The accuracy of each method is experimentally evaluated on test patterns and objects. We found that the point based calibration method is the best approach to recover a global registration between the two sensors, while an image-based method performed best when registering local regions.

Details

IROS Conference 2000 Conference Paper

Uncalibrated hybrid force-vision manipulation

  • Andreas Pichler
  • Martin Jägersand

We present a method employing hybrid force and vision based control to effect a sequence of contact manipulations. Instead of requiring a-priori object and environment models, force sensing is used to simultaneously update a surface model while controlling the manipulator. This is incorporated into an uncalibrated visual servoing system, which also estimates the visual-motor coordinate transform. The result is a hybrid force-vision controller which does not need any a-priori robot, camera, object or environment models. The approach is validated experimentally using an IMI Zebra robot arm.

Details

IROS Conference 1999 Conference Paper

Hierarchical uncalibrated predictive display for a 16 DOF Utah/MIT hand

  • Martin Jägersand

The addition of immediate but estimated visual feedback, called predictive display, improves telemanipulation performance when the real video feedback is delayed. Current systems typically rely upon a previously-calibrated camera and manipulator. We present a method where the motor-visual calibration is estimated online from motor commands and returned video images only. Predicted visual feedback is presented in two forms. As soon as a basic model has been estimated a wire frame drawing of the predicted current pose is overlaid on the delayed video feedback. After some time when a rich model has been estimated predicted intensity images are synthesized and these replace the delayed real video. In an intermediate situation where blurry synthesized images can be computed, the wireframe is overlaid on the synthesized images to show precisely the pose of the object. Experiments with a Utah/MIT robot hand are shown.

Details

ICRA Conference 1999 Conference Paper

Image Based Predictive Display for Tele-Manipulation

  • Martin Jägersand

Presents an image based method for generating predicted immediate operator visual feedback in a robot tele-manipulation system where the real feedback is delayed. No a priori modeling or calibration is needed. The image model is generated from the real images with techniques similar to image compression. The visual-motor calibration is estimated online from the delayed real visual and robot controller feedback. Experiments with a Utah/MIT robot hand and a PUMA robot arm are shown.

Details

ICRA Conference 1997 Conference Paper

Experimental evaluation of uncalibrated visual servoing for precision manipulation

  • Martin Jägersand
  • Olac Fuentes
  • Randal C. Nelson

We present an experimental evaluation of adaptive and non-adaptive visual servoing in 3, 6 and 12 degrees of freedom (DOF), comparing it to traditional joint feedback control. While the purpose of experiments in most other work has been to show that the particular algorithm presented indeed also works in practice, we do not focus on the algorithm but rather on properties important to visual servoing in general. Our main results are: positioning of a 6 axis PUMA 762 arm is up to 5 times more precise under visual control than under joint control; positioning of a Utah/MIT dextrous hand is better under visual control than under joint control by a factor of 2; and a trust-region-based adaptive visual feedback controller is very robust. For m tracked visual features the algorithm can successfully estimate online the m/spl times/3 (m/spl ges/3) image Jacobian (J) without any prior information, while carrying out a 3 DOF manipulation task. For 6 and higher DOF manipulation, a rough initial estimate of J is beneficial. We also verified that redundant visual information is valuable. Errors due to imprecise tracking and goal specification were reduced as the number of visual features, m, was increased. Furthermore highly redundant systems allow us to detect outliers in the feature vector and deal with partial occlusion.

Details
v2026.09.13