Author name cluster

Simon Léonard

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

23 papers

1 author row

ICRA Conference 2023 Conference Paper

A Virtual Reality Planning Environment for High-Risk, High-Latency Teleoperation

Will Pryor
Liam J. Wang
Arko Chatterjee
Balázs P. Vágvölgyi
Anton Deguet
Simon Léonard
Louis L. Whitcomb
Peter Kazanzides

Teleoperation of robots in space is challenging due to high latency and limited workspace visibility. Previously, the Interactive Planning and Supervised Execution (IPSE) and Augmented Virtuality systems were developed to reduce failure risk. These tools were visualized on a 3D da Vinci surgical console and operated using the da Vinci manipulators or visualized on conventional monitors and operated with a keyboard and mouse. Experimental studies indicated operator preference for the latter. In this work, we develop a 3D virtual reality (VR) interface for IPSE, implemented on a Meta Quest 2 head-mounted display (HMD), and evaluate it against the prior 2D, keyboard-and-mouse-based interface. The results demonstrate improved operator load with the 3D VR interface, with no decrease in task performance, while also providing cost and portability benefits compared to the conventional 2D interface.

ICRA Conference 2021 Conference Paper

Automated Mosquito Salivary Gland Extractor for PfSPZ-based Malaria Vaccine Production

Wanze Li
Zhuohong He
Parth Vora
Yanzhou Wang
Balázs P. Vágvölgyi
Simon Léonard
Anna Goodridge
Iulian I. Iordachita

Malaria is a worldwide scourge, and the broad deployment of an effective vaccine would improve the lives of millions of people. A vaccine based on Plasmodium falciparum (PfSPZ) sporozoites extracted from the salivary glands of infected mosquitoes shows significant promise. However, the large-scale industrial production of PfSPZ-based vaccines will benefit from automation of the key step of extracting sporozoites from mosquito salivary glands that is currently performed by manual microdissection. In this work, we demonstrate a robotic system prototype for extracting salivary glands from mosquitoes to streamline vaccine production and reduce the need for operators. In the proposed system, mosquitoes are decapitated in an automated robotic pick-place-decapitate process, then a squeezer apparatus extracts mosquito salivary glands from the body. Mosquito detection and body part localization are performed by computer vision methods. The software allows system operation in simulation and on the robotic hardware, which facilitates subsystem development and integration. Experiments show encouraging results with success rates of 93% in robotic mosquito manipulation and 87. 1% in salivary gland extraction. The system has the potential to improve the efficiency of PfSPZ vaccine production with significant gains in throughput and reduction in training times for a highly deskilled initial manual step. Further, this system is expected to pave the way for a more mature future system.

IROS Conference 2020 Conference Paper

Interactive Planning and Supervised Execution for High-Risk, High-Latency Teleoperation

Will Pryor
Balázs P. Vágvölgyi
Anton Deguet
Simon Léonard
Louis L. Whitcomb
Peter Kazanzides

Ground-based teleoperation of robot manipulators for on-orbit servicing of spacecraft represents an example of high-payoff, high-risk operations that are challenging to perform due to high latency communications, with telemetry time delays of several seconds. In these scenarios, confidence of operating without failure is paramount. We report the development of an Interactive Planning and Supervised Execution (IPSE) system that takes advantage of accurate 3D reconstruction of the remote environment to enable operators to plan motions in the virtual world, evaluate and adjust the plan, and then supervise execution with the ability to pause and return to the planning environment at any time. We report the results of an experimental evaluation of a representative on-orbit telerobotic servicing task from NASA's upcoming OSAM-1 mission to refuel a satellite in low earth orbit; specifically, to change the robot tool to acquire the fuel supply line and then to insert it into the satellite fill/drain valve. Results of a pilot study show that the operators preferred, and were more successful with, the IPSE system when compared to a conventional teleoperation implementation.

IROS Conference 2020 Conference Paper

Visual Monitoring and Servoing of a Cutting Blade during Telerobotic Satellite Servicing

Amama Mahmood
Balázs P. Vágvölgyi
Will Pryor
Louis L. Whitcomb
Peter Kazanzides
Simon Léonard

We propose a system for visually monitoring and servoing the cutting of a multi-layer insulation (MLI) blanket that covers the envelope of satellites and spacecraft. The main contributions of this paper are: 1) to propose a model for relating visual features describing the engagement depth of the blade to the force exerted on the MLI blanket by the cutting tool, 2) a blade design and algorithm to reliably detect the engagement depth of the blade inside the MLI, and 3) a servoing mechanism to achieve the desired applied force by monitoring the engagement depth. We present results that validate these contributions by comparing forces estimated from visual feedback to measured forces at the blade. We also demonstrate the robustness of the blade design and vision processing under challenging conditions.

ICRA Conference 2019 Conference Paper

Autonomous Laparoscopic Robotic Suturing with a Novel Actuated Suturing Tool and 3D Endoscope

Hamed Saeidi
Hanh N. D. Le
Justin D. Opfermann
Simon Léonard
A. Kim
Michael H. Hsieh
Jin U. Kang
Axel Krieger

Compared to open surgical techniques, laparoscopic surgical methods aim to reduce the collateral tissue damage and hence decrease the patient recovery time. However, constraints imposed by the laparoscopic surgery, i. e. the operation of surgical tools in limited spaces, turn simple surgical tasks such as suturing into time-consuming and inconsistent tasks for surgeons. In this paper, we develop an autonomous laparoscopic robotic suturing system. More specific, we expand our smart tissue anastomosis robot (STAR) by developing i) a new 3D imaging endoscope, ii) a novel actuated laparoscopic suturing tool, and iii) a suture planning strategy for the autonomous suturing. We experimentally test the accuracy and consistency of our developed system and compare it to sutures performed manually by surgeons. Our test results on suture pads indicate that STAR can reach 2. 9 times better consistency in suture spacing compared to manual method and also eliminate suture repositioning and adjustments. Moreover, the consistency of suture bite sizes obtained by STAR matches with those obtained by manual suturing.

ICRA Conference 2019 Conference Paper

Experimental Evaluation of Teleoperation Interfaces for Cutting of Satellite Insulation

Will Pryor
Balázs P. Vágvölgyi
William J. Gallagher
Anton Deguet
Simon Léonard
Louis L. Whitcomb
Peter Kazanzides

On-orbit servicing of satellites is complicated by the fact that almost all existing satellites were not designed to be serviced. This creates a number of challenges, one of which is to cut and partially remove the protective thermal blanketing that encases a satellite prior to performing the servicing operation. A human operator on Earth can perform this task telerobotically, but must overcome difficulties presented by the multi-second round-trip telemetry delay between the satellite and the operator and the limited, or even obstructed, views from the available cameras. This paper reports the results of ground-based experiments with trained NASA robot teleoperators to compare our recently-reported augmented virtuality visualization to the conventional camera-based visualization. We also compare the master console of a da Vinci surgical robot to the conventional teleoperation interface. The results show that, for the cutting task, the augmented virtuality visualization can improve operator performance compared to the conventional visualization, but that operators are more proficient with the conventional control interface than with the da Vinci master console.

IROS Conference 2018 Conference Paper

A Confidence-Based Shared Control Strategy for the Smart Tissue Autonomous Robot (STAR)

Hamed Saeidi
Justin D. Opfermann
Michael Kam
Sudarshan Raghunathan
Simon Léonard
Axel Krieger

Autonomous robotic assisted surgery (RAS) systems aim to reduce human errors and improve patient outcomes leveraging robotic accuracy and repeatability during surgical procedures. However, full automation of RAS in complex surgical environments is still not feasible and collaboration with the surgeon is required for safe and effective use. In this work, we utilize our Smart Tissue Autonomous Robot (STAR) to develop and evaluate a shared control strategy for the collaboration of the robot with a human operator in surgical scenarios. We consider 2D pattern cutting tasks with partial blood occlusion of the cutting pattern using a robotic electrocautery tool. For this surgical task and RAS system, we i) develop a confidence-based shared control strategy, ii) assess the pattern tracking performances of manual and autonomous controls and identify the confidence models for human and robot as well as a confidence-based control allocation function, and iii) experimentally evaluate the accuracy of our proposed shared control strategy. In our experiments on porcine fat samples, by combining the best elements of autonomous robot controller with complementary skills of a human operator, our proposed control strategy improved the cutting accuracy by 6. 4%, while reducing the operator work time to 44% compared to a pure manual control.

ICRA Conference 2018 Conference Paper

Semi-Autonomous Laparoscopic Robotic Electro-Surgery with a Novel 3D Endoscope * Research reported in this paper was supported by National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award numbers 1R01EB020610 and R21EB024707. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health

Hanh N. D. Le
Justin D. Opfermann
Michael Kam
Sudarshan Raghunathan
Hamed Saeidi
Simon Léonard
Jin U. Kang
Axel Krieger

This paper reports a robotic laparoscopic surgery system performing electro-surgery on porcine cadaver kidney, and evaluates its accuracy in an open loop control scheme to conduct targeting and cutting tasks guided by a novel 3D endoscope. We describe the design and integration of the novel laparoscopic imaging system that is capable of reconstructing the surgical field using structured light. A targeting task is first performed to determine the average positioning error of the system as guided by the laparoscopic camera. The imaging system is then used to reconstruct the surface of a porcine cadaver kidney, and generate a cutting trajectory with consistent depth. The paper concludes by using the robotic system in open loop control to cut this trajectory using a multi degree of freedom electro-surgical tool. It is demonstrated that for a cutting depth of 3 mm, the robotic surgical system follows the trajectory with an average depth of 2. 44 mm and standard deviation of 0. 34 mm. The average positional accuracy of the system was 2. 74±0. 99 mm.

IROS Conference 2017 Conference Paper

Semi-autonomous electrosurgery for tumor resection using a multi-degree of freedom electrosurgical tool and visual servoing

Justin D. Opfermann
Simon Léonard
Ryan S. Decker
Nicholas A. Uebele
Christopher E. Bayne
Arjun S. Joshi
Axel Krieger

This paper specifies a surgical robot performing semi-autonomous electrosurgery for tumor resection and evaluates its accuracy using a visual servoing paradigm. We describe the design and integration of a novel, multi-degree of freedom electrosurgical tool for the smart tissue autonomous robot (STAR). Standardized line tests are executed to determine ideal cut parameters in three different types of porcine tissue. STAR is then programmed with the ideal cut setting for porcine tissue and compared against expert surgeons using open and laparoscopic techniques in a line cutting task. We conclude with a proof of concept demonstration using STAR to semi-autonomously resect pseudo-tumors in porcine tissue using visual servoing. When tasked to excise tumors with a consistent 4mm margin, STAR can semi-autonomously dissect tissue with an average margin of 3. 67 mm and a standard deviation of 0. 89mm.

ICRA Conference 2016 Conference Paper

Plenoptic cameras in surgical robotics: Calibration, registration, and evaluation

Azad Shademan
Ryan S. Decker
Justin D. Opfermann
Simon Léonard
Peter C. W. Kim
Axel Krieger

Three-dimensional sensing of changing surgical scenes would improve the function of surgical robots. This paper explores the requirements and utility of a new type of depth sensor, the plenoptic camera, for surgical robots. We present a metric calibration procedure for the plenoptic camera and the registration of its coordinate frame to the robot (hand-eye calibration). We also demonstrate the utility in robotic needle insertion and application of sutures in phantoms. The metric calibration accuracy is reported as 1. 14 ± 0. 80 mm for the plenoptic camera and 1. 57 ± 0. 90 mm for hand-eye calibration. The accuracy of needle insertion task is 1. 79 ± 0. 35 mm for the entire robotic system. Additionally, the accuracy of suture placement with the presented system is reported at 1. 80 ± 0. 43 mm. Finally, we report consistent suture spacing with only 0. 11 mm standard deviation between inter-suture distances. The measured accuracy of less than 2 mm with consistent suture spacing is a promising result to provide repeatable leak-free suturing with a robotic tool and a plenoptic depth imager.

ICRA Conference 2015 Conference Paper

Experimental evaluation of force control for virtual-fixture-assisted teleoperation for on-orbit manipulation of satellite thermal blanket insulation

Steve Vozar
Simon Léonard
Peter Kazanzides
Louis L. Whitcomb

The ability to refuel satellites on-orbit using teleoperated robots could extend the life of satellite missions, potentially saving time and resources for both industry and space agencies. One critical step in the refueling process is to gain access to the satellite's fueling port by cutting through a seam of tape that adheres two sections of multi-layer insulation (MLI) covering the satellite body. The deformable, delicate, specular nature of the metalized tape used to adhere sections of MLI to one another makes cutting the tape seams a difficult task to perform remotely, a difficulty that is compounded by unavoidable multi-second communications delays. Virtual-fixture-assisted teleoperation can help mitigate some of these issues by constraining the motion of the cutting blade based on a priori knowledge of the task. Adding force control may further assist teleoperators by automatically setting a desired normal cutting force. This paper presents the results of a 20-participant user study for a representative satellite tape-cutting task, showing that force control based on a virtual fixture mapped to the MLI blanket plane can decrease both tape bunching and operator workload.

ICRA Conference 2015 Conference Paper

Registration of planar virtual fixtures by using augmented reality with dynamic textures

Simon Léonard

This paper presents a method to align a virtual plane with a real plane in an augmented reality environment. The method addresses the challenge of configuring a planar virtual fixture during the teleoperation of a robot in a visually challenging environment. The method assists the operator to manually align the virtual plane with the real plane by providing visual cues by dynamically texturing the virtual plane. Following the manual initialization a robust evaluation of the optical flow is used to refine the alignment of the virtual plane. The system combines a stereo camera with a stereo display and generates visual distortions in the displayed images to indicate a misalignment of the virtual plane with the real one. The distortion is computed by displaying warped images from the left camera in the right display and vice versa. Results demonstrate that the method improves the accuracy of the alignment with a real plane that is covered with reflective multi-layer insulation.

ICRA Conference 2014 Conference Paper

Smart Tissue Anastomosis Robot (STAR): Accuracy evaluation for supervisory suturing using near-infrared fluorescent markers

Simon Léonard
Azad Shademan
Yonjae Kim
Axel Krieger
Peter C. W. Kim

This paper specifies and evaluates the accuracy of the Smart Tissue Anastomosis Robot (STAR). The STAR is a proof of concept vision-guided robotic system equipped with an actuated laparoscopic suturing tool and a multispectral vision system. The STAR supports image-based suturing commands and is capable of detecting near-infrared fluorescent (NIRF) markers that provide reliable visual segmentation and tracking. The paper reports the best case scenario accuracy specifications of the STAR as derived from its configuration and calibration parameters. We also evaluate experimentally the effects of overlaying NIRF markers on the accuracy of the STAR when these markers are used as the source of image-based commands and we compare these results to the accuracy of the STAR with image-based commands generated from plain color images. Our results demonstrate that the STAR is able to place sutures on a planar phantom with an average accuracy of 0. 5 mm with a standard deviation of 0. 2 mm and that NIRF markers have no statistically significant adverse effect on the accuracy.

ICRA Conference 2013 Conference Paper

Model-based telerobotic control with virtual fixtures for satellite servicing tasks

Tian Xia
Simon Léonard
Isha Kandaswamy
Amy A. Blank
Louis L. Whitcomb
Peter Kazanzides

Our goal is to develop new methods for telerobotic on-orbit servicing of spacecraft under ground-based supervisory control of human operators to perform tasks in the presence of uncertainty and telemetry time delay of several seconds. We propose a new delay tolerant control methodology, using virtual fixtures, hybrid position/force control, task frame formalism, and environment modeling, that is robust to modeling and registration errors. The task model is represented by graphical primitives and virtual fixtures on the teleoperation master and by a hybrid position/force controller on the slave robot. The virtual fixtures guide the operator through a model-based simulation of the task, and the goal of the slave controller is to reproduce this action (after a few seconds of delay) or, if measurements are not consistent with the models, to stop motion and alert the operator. This approach is suitable for tasks in unstructured environments, such as servicing of existing on-orbit spacecraft that were not designed for servicing. We introduce the overall control concept, its main components, and an example application in which the remote slave robot cuts the tape that secures a flap of multi-layer insulation over the access panel of a satellite mockup.

IROS Conference 2012 Conference Paper

Augmented reality environment with virtual fixtures for robotic telemanipulation in space

Tian Xia
Simon Léonard
Anton Deguet
Louis L. Whitcomb
Peter Kazanzides

This paper presents an augmented reality framework, implemented on the master console of a modified da Vinci® surgical robot, that enables the operator to design and implement assistive virtual fixtures during teleoperation. Our specific goal is to facilitate teleoperation with large time delays, such as the delay of several seconds that occurs with ground-based control of robotic systems in earth orbit. The virtual fixtures give immediate visual feedback and motion guidance to the operator, while the remote slave performs motions consistent with those constraints. This approach is suitable for tasks in unstructured environments, such as servicing of existing on-orbit spacecraft that were not designed for servicing. We conducted a pilot study by teleoperating a remote slave robot for a thermal barrier blanket cutting task using virtual fixtures with and without time delay. The results show that virtual fixtures reduce the time required to complete the task while also eliminating significant manipulation errors, such as tearing the blanket. The improvement in performance is especially dramatic when a simulated time delay (4 seconds) is introduced.

ICRA Conference 2012 Conference Paper

Sequential scene parsing using range and intensity information

Manuel Brucker
Simon Léonard
Tim Bodenmüller
Gregory D. Hager

This paper describes an extension of the sequential scene analysis system presented by Hager and Wegbreit [12]. In contrast to the original system, which was limited to scenes consisting of geometric primitives, such as spheres, cuboids, and cylinders computed from range data, the extended system is capable of dealing with arbitrarily shaped objects computed from range and intensity images. An object model composed of a triangulated geometry and intensity-based SURF features is introduced. The integration of prior object models into the sequential scene parsing framework is described. The extended system is evaluated with respect to pose estimation and its ability to handle complex scene sequences. It is shown that the new object models enable accurate pose estimation and reliable recognition even in highly cluttered scenes.

IROS Conference 2009 Conference Paper

Planning collision-free and occlusion-free paths for industrial manipulators with eye-to-hand configuration

Simon Léonard
Elizabeth A. Croft
James J. Little

This paper presents a motion planning algorithm for industrial manipulators with the simultaneous constraints of avoiding collisions and avoiding the occlusion of specified pixellated regions of an eye-to-hand camera. The system uses a probabilistic roadmap to satisfy the constraints imposed by the command interface of typical industrial manipulators and uses dynamic collision checking to ensure collision-free motion. In the context of a task monitored by a camera, we enhance a probabilistic roadmap with a dynamic occlusion checking algorithm that is able to determine which pixels of the camera are occluded by the robot during each motion segment. The occlusion algorithm is formulated as collision algorithm where the field of view of the camera is represented as a quadtree of frustums. The proposed algorithm is demonstrated in industrial bin picking simulations where the gripper must not occlude the targeted object throughout the task.

ICRA Conference 2008 Conference Paper

Dynamic visibility checking for vision-based motion planning

Simon Léonard
Elizabeth A. Croft
James J. Little

An important problem in position-based visual servoing (PBVS) is to guarantee that a target will remain within the field of view for the duration of the task. In this paper, we propose a dynamic visibility checking algorithm that, given a parametrized trajectory of the camera, determines if an arbitrary 3D target will remain within the field of view. We reformulate this problem as the problem of determining if the 3D coordinates of the target collide with the frustum formed by the camera field of view during the camera trajectory. To solve this problem, our algorithm computes and compares the shortest distance between the target and the frustum with the length of the trajectory described by the target in the camera's coordinate frame. Furthermore, we demonstrate that our algorithm can be combined with path planning algorithms and, in particular, probabilistic roadmaps (PRM). Results suggest that our algorithm is computationally efficient even when the target moves in the vicinity of image borders. In simulations, we use our dynamic visibility checking algorithm in conjunction with a PRM to plan collision free paths while providing the guarantee that a specific target will not leave the field of view.

IROS Conference 2008 Conference Paper

Occlusion-free path planning with a probabilistic roadmap

Matthew A. Baumann
Donna C. Dupuis
Simon Léonard
Elizabeth A. Croft
James J. Little

We present a novel algorithm for path planning that avoids occlusions of a visual target for an ldquoeye-in-handrdquo sensor on an articulated robot arm. We compute paths using a probabilistic roadmap to avoid collisions between the robot and obstacles, while penalizing trajectories that do not maintain line-of-sight. The system determines the space from which line-of-sight is unimpeded to the target (the visible region). We assign penalties to trajectories within the roadmap proportional to the distance the camera travels while outside the visible region. Using Dijkstrapsilas algorithm, we compute paths of minimal occlusion (maximal visibility) through the roadmap. In our experiments, we compare a shortest-distance path to the minimal-occlusion path and discuss the impact of the improved visibility.

ICRA Conference 2007 Conference Paper

On with the Visuomotor Function: A 6DOF Adaptive Approach for Modeling Image-Based Variations and Visual Servoing

Simon Léonard
Martin Jägersand

In this paper, we proposes a visual servoing method that approximates the relation between the variations of image points and the variations of a stereo rig in Euclidian space. As with most image-based visual servoing methods, commands are expressed in the space of image features. However, instead of relating instantaneous image-based variations to instantaneous variations in Euclidian space, the visuomotor function relates arbitrary image-based variations to Euclidian transformations. The visuomotor function is approximated in real-time by using online estimation techniques. The system improves its performance with experience and is able to adapt to different configurations of the cameras or environment. Given the disparities between two sets of corresponding image points, the visuomotor function provides the Euclidian transformation the robot must execute in order to align the image coordinates.

ICRA Conference 2006 Conference Paper

Adaptive Control for Estimating Translations from Image-based Variations

Simon Léonard
Martin Jägersand

We investigate the problem of learning the mapping between arbitrary image-based variations and variations in Euclidean space for the application of visual-servoing. We derive a linear formulation of the visuomotor function, which captures this relationship for three dimensional translations. The resulting expression defines how the origin of the world coordinate frame shifts in stereo images as the robot translates. The parameters of the visuomotor function are estimated online by using incremental least squares and are generalized to other coordinate frames origins by using a function approximation method. It follows that the system is able to estimate the three dimensional translations between pairs of stereo points without performing 3D reconstruction or requiring a specific coordinate system and is fully adaptive

ICRA Conference 2005 Conference Paper

Incremental Learning for Mapping Image Variations to Actions

Simon Léonard
Martin Jägersand

In this paper we introduce a method to learn the global relationship between camera motion and its effect on a feature space. Drawing from progress in vision based motion control, we derive an image-based method that learns an approximation of the visuomotor function while performing visual servoing tasks. The result is a reactive agent that measures its performance in the image feature space and controls a robot with position error commands. Thus combining image-based references with position-based commands.

IROS Conference 2004 Conference Paper

Learning based visual servoing

Simon Léonard
Martin Jägersand

This paper proposes a method for learning a hand-eye calibration and its application for visual servoing. The goal is to develop a technique that combines the strengths of existing visual servoing methods. Particularly, as in image-based visual servoing, the error is measured in the visual space while the motor command is position-based. Hence this method approximates the visuomotor function that relates variations in the visual space to variations in the motor space at a global scale. The method used for approximating the visuomotor function is derived from the field of reinforcement learning, making our hand-eye calibration autonomous, continuous and adaptable. The visuomotor function is modeled by a linear combination of polynomials, each spanning a non-mutually exclusive subset of the visual space. Each polynomial represents the utility of motor commands for the servoing task. The goal of the calibration is to approximate the parameters of these polynomials while the system interacts with its environment. Preliminary results include centering a target in the image in which the system learns the motor commands that eliminates the errors in the visual space and generalizes the result to neighboring states in the visual space, depths and motor commands.