Author name cluster

Bradley J. Nelson

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

95 papers

2 author rows

IROS Conference 2025 Conference Paper

A Physics-Informed Neural Network for the Calibration of Electromagnetic Navigation Systems

Pascal Ernst
Simone Gervasoni
Derick Sivakumaran
Enea Masina
David F. Sargent
Bradley J. Nelson
Quentin Boehler

Electromagnetic Navigation Systems enable remote actuation of untethered micro and nanorobots, as well as the precise control of magnetic surgical tools for minimally invasive medical procedures. Accurate modeling of the magnetic fields generated by the electromagnets composing these systems is essential for achieving reliable and precise navigation. Existing modeling approaches either neglect nonlinear effects such as electromagnet saturation or fail to ensure that the field predictions are physically consistent. These limitations can lead to significant prediction errors, particularly in the estimation of field gradients, which directly impacts force calculations. As a result, inaccurate gradient predictions degrade force control performance, limiting the precision of magnetic actuation. In this work, we investigate physics-informed and data-driven modeling techniques to improve the accuracy of magnetic field and gradient predictions. Additionally, we introduce an approach for solving the inverse problem, developing models capable of predicting the required electromagnet currents to generate a desired magnetic field and gradient based on this approach. By incorporating physical constraints into the models, we enhance the predictive accuracy and physical consistency of the field estimates. In the experimental section, we demonstrate the benefits of these methods to enable improved force control in open-loop for untethered robots using a small-scale Electromagnetic Navigation System.

JBHI Journal 2023 Journal Article

Real-Time Gait Phase Detection on Wearable Devices for Real-World Free-Living Gait

Jiaen Wu
Barna Becsek
Alessandro Schaer
Henrik Maurenbrecher
George Chatzipirpiridis
Olgac Ergeneman
Salvador Pané
Hamdi Torun

Detecting gait phases with wearables unobtrusively and reliably in real-time is important for clinical gait rehabilitation and early diagnosis of neurological diseases. Due to hardware limitations of microcontrollers in wearable devices (e. g. , memory and computation power), reliable real-time gait phase detection on the microcontrollers remains a challenge, especially for long-term real-world free-living gait. In this work, a novel algorithm based on a reduced support vector machine (RSVM) and a finite state machine (FSM) is developed to address this. The RSVM is developed by exploiting the cascaded K-means clustering to reduce the model size and computation time of a standard SVM by 88% and a factor of 36, with only minor degradation in gait phase prediction accuracy of around 4%. For each gait phase prediction from the RSVM, the FSM is designed to validate the prediction and correct misclassifications. The developed algorithm is implemented on a microcontroller of a wearable device and its real-time (on the fly) classification performance is evaluated by twenty healthy subjects walking along a predefined real-world route with uncontrolled free-living gait. It shows a promising real-time performance with an accuracy of 91. 51%, a sensitivity of 91. 70%, and a specificity of 95. 77%. The algorithm also demonstrates its robustness with varying walking conditions.

IROS Conference 2022 Conference Paper

Shape memory polymer variable stiffness magnetic catheters with hybrid stiffness control

Michael Mattmann
Quentin Boehler
Xiangzhong Chen
Salvador Pané
Bradley J. Nelson

Variable stiffness catheters typically rely on thermally induced stiffness transitions with a transition temperature above body temperature. This imposes considerable safety limitations for medical applications. In this work, we present a variable stiffness catheter using a hybrid control strategy capable of actively heating and actively cooling the catheter material. The proposed catheter is made of a single biocompatible shape memory polymer, which significantly increases its manufacturability and scalability compared to existing designs. Potentially increased safety is obtained by ensuring a lower-risk compliant state at body temperature while maintaining higher stiffness ranges in actively controlled states. Additionally, the combined use of variable stiffness and magnetic actuation increases the dexterity and steerability of the device compared to existing robotic tools.

ICRA Conference 2020 Conference Paper

Modeling Electromagnetic Navigation Systems for Medical Applications using Random Forests and Artificial Neural Networks

Ruoxi Yu
Samuel Charreyron
Quentin Boehler
Cameron Weibel
Christophe Chautems
Carmen C. Y. Poon
Bradley J. Nelson

Electromagnetic Navigation Systems (eMNS) can be used to control a variety of multiscale devices within the human body for remote surgery. Accurate modeling of the magnetic fields generated by the electromagnets of an eMNS is crucial for the precise control of these devices. Existing methods assume a linear behavior of these systems, leading to significant modeling errors within nonlinear regions exhibited at higher magnetic fields, preventing these systems from operating at full capacity. In this paper, we use a random forest (RF) and an artificial neural network (ANN) to model the nonlinear behavior of the magnetic fields generated by an eMNS. Both machine learning methods outperformed the state-of-the-art linear multipole electromagnet model (MPEM). The RF and the ANN model reduced the root mean squared error (RMSE) of the MPEM when predicting the field magnitude by approximately 40% and 87%, respectively, over the entire current range of the eMNS. At high current regions, especially between 30 and 35 A, the field-magnitude RMSE improvement of the ANN model over the MPEM was 37 mT, equivalent to 90% error reduction. This study demonstrates the feasibility of using machine learning to model an eMNS for medical applications, and its ability to account for complex nonlinear behavior at high currents. The use of machine learning thus shows promise in developing accurate field predicting models, and ultimately improving surgical procedures that use magnetic navigation.

ICRA Conference 2020 Conference Paper

Reconfigurable Magnetic Microswarm for Thrombolysis under Ultrasound Imaging

Qianqian Wang 0003
Ben Wang 0007
Jiangfan Yu
Kathrin Schweizer
Bradley J. Nelson
Li Zhang 0010

We propose thrombolysis using a magnetic nanoparticle microswarm with tissue plasminogen activator (tPA) under ultrasound imaging. The microswarm is generated in blood using an oscillating magnetic field and can be navigated with locomotion along both the long and short axis. By modulating the input field, the aspect ratio of the microswarm can be reversibly tuned, showing the ability to adapt to different confined environments. Simulation results indicate that both in-plane and out-of-plane fluid convection are induced around the microswarm, which can be further enhanced by tuning the aspect ratio of the microswarm. Under ultrasound imaging, the microswarm is navigated in a microchannel towards a blood clot and deformed to obtain optimal lysis. Experimental results show that the lysis rate reaches -0. 1725 ± 0. 0612 mm 3 /min in the 37°C blood environment under the influence of the microswarm-induced fluid convection and tPA. The lysis rate is enhanced 2. 5-fold compared to that without the microswarm (-0. 0681 ± 0. 0263 mm 3 /min). Our method provides a new strategy to increase the efficiency of thrombolysis by applying microswarm-induced fluid convection, indicating that swarming micro/nanorobots have the potential to act as effective tools towards targeted therapy.

IROS Conference 2018 Conference Paper

Fabrication and Locomotion of Flexible Nanoswimmers

Bumjin Jang
Amanda Aho
Bradley J. Nelson
Salvador Pané

Small-scale robots with soft joints and hinges have recently attracted interest because these components allow for more sophisticated locomotion mechanisms. Here, we investigate two different types of nanoscale swimmers as depicted in Figure 1. One consists of a rigid magnetic head linked to a semi-soft tail (1-link swimmer). Another consists of a rigid magnetic head and tail connected by a soft hinge (2-link swimmer). Both swimmers exhibit undulatory locomotion under an applied oscillating magnetic field. The speeds of the swimmers are assessed as a function of the oscillating magnetic field frequency and the sweeping angle. We find that a resonance-like frequency increases as the length decreases, and, in general, the speed increases as the sweeping angle increases. Last, we show that 2-link swimmers can also swim in a corkscrew-like pattern under rotating magnetic fields.

IROS Conference 2017 Conference Paper

A variable stiffness catheter controlled with an external magnetic field

Christophe Chautems
Alice Tonazzini
Dario Floreano
Bradley J. Nelson

Remote magnetic navigation of catheters is a technique used to perform radiofrequency ablation of heart tissue in order to treat cardiac arrhythmias. The flexible magnetic catheters used in this context are in some cases not sufficiently dexterous to navigate the complex and patient-specific anatomy of the heart. To overcome such limitations, this paper proposes a new approach that relies on the integration of variable stiffness segments into a magnetic catheter. The magnetic variable stiffness (VS) catheter presented here is based on silicone and a low melting point alloy (LMPA) that transforms from a solid to liquid phase upon joule heating. This dramatically changes the bending stiffness of the segment in which it is integrated, improving dexterity. Compared to standard catheters, a VS catheter can partially (just one segment) or completely lock its shape (shape fixity) in order to explore a larger 3D volume inside a magnetic navigation system, thus extending regions of the heart that can be reached for performing ablation procedures.

ICRA Conference 2017 Conference Paper

In vivo tracking and measurement of pollen tube vesicle motion

Chengzhi Hu
Qi Zhang
Tobias Meyer
Hannes Vogler
Jan T. Burri
Naveen Shamsudhin
Ueli Grossniklaus
Bradley J. Nelson

Particle tracking has emerged as a powerful tool for investigating the swarm control of microrobots and the dynamic biological processes in the life sciences. In seed plants, pollen tubes, a part of the male gametophyte, are excellent models for understanding plant growth and cellular behavior, because vesicle motion within pollen tubes reveals important information about vesicle function and interactions. Conventional vesicle tracking is based on spatiotemporal image analysis, which requires high-quality images and vesicles with constant velocity. For in vivo tracking, vesicles may disappear in some frames, and image sequences may have spatial and temporal distortions, which hamper vesicle tracking for broader applications. In this paper, we studied intracellular motion during pollen tube growth with an optical flow method. Streaming images from confocal and optical microscopes were recorded to study the intracellular motion of vesicles of different size. Local motion for each vesicle was detected using a local displacement vector field. The displacement from two adjacent frames was then calculated. The flow field shows information such as the dynamics of vesicle secretion, endocytosis, exocytosis, and cytoskeletal stability. Vesicles from different regions inside the tube were tracked simultaneously with a Kanade-Lucas-Tomasi (KLT) feature matching algorithm. The spatial and temporal characteristics of intracellular vesicles were evaluated. The proposed methods can be of great use for studying the dynamics of fluorescently tagged particles in biological systems.

ICRA Conference 2017 Conference Paper

Shared control of a magnetic microcatheter for vitreoretinal targeted drug delivery

Samuel Charreyron
Burak Zeydan
Bradley J. Nelson

Retinal diseases including age-related macular degeneration and diabetic retinopathy are leading causes of visual impairment and a growing medical crisis worldwide. As a result, new therapies and methods of safely delivering these new therapies can be of significant benefit to society. In this paper, we present a proof-of-concept for performing drug delivery using flexible magnetic microcatheters. The potential benefits of these catheters are that they are safer to use than existing rigid vitreoretinal tools, are precisely manipulable, and can be used to deliver lower dosages of therapeutic agents to precisely targeted locations on the retina with higher efficacy. The task of positioning the catheter tip is shared between a human operator and a control algorithm, which have complementary skills and can separately address different aspects of the overall task. The system was tested in an eye-phantom by an untrained operator with a 93% success rate in reaching 43 target points uniformly dispersed over the retina.

ICRA Conference 2017 Conference Paper

The tethered magnet: Force and 5-DOF pose control for cardiac ablation

Christophe Chautems
Bradley J. Nelson

Cardiac arrhythmias are commonly treated by minimally invasive catheter ablation. Because of challenges in precisely manipulating the distal end of the catheter, several magnetic manipulation systems (MMS) have been developed to aid in steering the tip inside the heart. Due to the relative stiffness of catheters, magnetic gradients that create forces acting on the catheter tip position are limited in their effectiveness and are typically ignored. This dramatically limits the tip orientations possible at any specified position in the workspace. Replacing the flexible catheter tip by a string-like tether and replacing multiple magnets by a single magnet allows the ablation tool to be positioned at any location with essentially any orientation. In addition to the five pose degrees of freedom, the tension on the tether can also be controlled. During an ablation procedure, the distal end of the tethered magnet is placed in contact with the heart wall. Extending the tether after tip contact transfers the tension on the tether to a force at the contact point. The contact force can then be precisely controlled and is no longer dependent on catheter bending radius and insertion length, as is the case with current catheter steering systems. The kinematics of a tethered magnet is modeled and tested inside a clinical MMS called the Aeon Phocus. This demonstrates the feasibility of magnetic field gradient control in a medically certified system currently in clinical use.

ICRA Conference 2016 Conference Paper

Magnetic microrobots with addressable shape control

Hen-Wei Huang
Mahmut Selman Sakar
Katharina Riederer
Naveen Shamsudhin
Andrew J. Petruska
Salvador Pané
Bradley J. Nelson

Shape shifting soft microrobots are generated from self-folding hydrogel bilayer structures. The folding conditions are analyzed to develop an optimal strategy for producing desired three-dimensional shapes. We present two different methods for programming magnetization in these microrobots that are variant and invariant to folding. The microrobots can be navigated through user-defined trajectories using rotating magnetic fields, and the morphing in response to temperature changes can be tuned for adaptive behavior. On-demand modulation of the mobility of individual microrobots is demonstrated by morphing their shape using selective near infrared light (NIR) exposure.

ICRA Conference 2016 Conference Paper

Magnetic needle guidance for neurosurgery: Initial design and proof of concept

Andrew J. Petruska
Fabio Ruetz
Ayoung Hong
Luca Regli
Oguzkan Surucu
Ajmal Zemmar
Bradley J. Nelson

A magnetic-tip steerable needle is presented with application to aiding deep brain stimulation electrode placement. The magnetic needle is 1. 3mm in diameter at the tip with a 0. 7mm diameter shaft, which is selected to match the size of a deep brain stimulation electrode. The tip orientation is controlled by applying torques to the embedded neodymium-iron-boron permanent magnets with a clinically-sized magnetic-manipulation system. The prototype design is capable of following trajectories under human-in-the-loop control with minimum bend radii of 100mm without inducing tissue damage and down to 30mm if some tissue damage is tolerable. The device can be retracted and redirected to reach multiple targets with a single insertion point.

ICRA Conference 2015 Conference Paper

Magnetically actuated and guided milli-gripper for medical applications

Franziska Mathis-Ullrich
Kanika S. Dheman
Simone Schürle
Bradley J. Nelson

This paper presents the design, kinematics, fabrication, and magnetic manipulation of a milli-gripper for medical applications. The design employs a permanent magnet for two purposes. It actuates the compliant gripper and allows for maneuverability of the milli-gripper in an externally applied magnetic field generated by an electromagnetic manipulation system. The modular milli-gripper can be manipulated directly or attached to the distal tip of a magnetically steered catheter. Experiments show successful actuation of the gripper and guidance of the device with the integrated gripper in both the tethered and untethered configuration.

IROS Conference 2015 Conference Paper

Navigation of a rolling microrobot in cluttered environments for automated crystal harvesting

Samuel Charreyron
Roel Pieters
Hsi-Wen Tung
Maurice Gonzenbach
Bradley J. Nelson

In this paper, we present a holistic system for automating the motion of a rolling microrobot for protein crystal harvesting. The RodBot, which was introduced in previous work, is able to perform noncontact manipulation of microscopic objects such as fragile crystals by trapping them in an induced vortex fluid flow. Here, we are concerned with navigating the RodBot autonomously in a liquid environment containing obstacles such as crystals. A literature review shows existing approaches to untethered microrobot control are limited and cluttered environments are often not considered. We demonstrate real-time tracking of the RodBot and surrounding obstacles, kinematic obstacle-free path planning, and nonholonomic path following. The system was evaluated in qualitative and quantitative experiments, shows satisfactory performance, and presents itself as a first step towards fully automated crystal harvesting.

ICRA Conference 2015 Conference Paper

RodBot: A rolling microrobot for micromanipulation

Roel Pieters
Hsi-Wen Tung
Samuel Charreyron
David F. Sargent
Bradley J. Nelson

We introduce the modelling and control of a rolling microrobot. The microrobot is capable of manipulating micro-objects through the use of a magnetic visual control system. This system consists of a rod-shaped microrobot, a magnetic actuation system and a visual control system. Motion of the rolling microrobot on a supporting surface is induced by a rotating magnetic field. As the robot is submerged in a liquid this motion creates a rising flow in front, a sinking flow behind, and a vortex above the robot, thus enabling non-contact transportation of micro-objects. Besides this fluid-vortex approach, the microrobot is also able to manipulate micro-objects via a pushing strategy. We present the design and modelling of the 50×60×300 μm micro-agent, the visual control system, and an experimental analysis of the micromanipulation and control methods.

ICRA Conference 2015 Conference Paper

Tracking a magnetically guided catheter with a single rotating C-Arm

Ayoung Hong
Andrew J. Petruska
Bradley J. Nelson

This paper presents a method to localize a magnetically guided catheter using a single rotating C-Arm. Because the three dimensional position of the catheter tip, when extracted from a single view, contains large uncertainties, we propose to combine multiple views by rotating the C-Arm and using a Kalman filter. The proposed approach yields good tracking performance for several catheter poses and reduces the maximum estimated standard deviation to 0. 3 cm after incorporating six views. This represents a 62 percent reduction in three dimensional position uncertainty.

ICRA Conference 2014 Conference Paper

Automated capsulorhexis based on a hybrid magnetic-mechanical actuation system

Franziska Mathis-Ullrich
Simone Schürle
Roel Pieters
Avraham Dishy
Stephan Michels
Bradley J. Nelson

This paper presents a hybrid magnetic-mechanical manipulation system for automated capsulorhexis utilizing a flexible catheter with a sharp edge magnetic tip. Vision based closed loop control is implemented to guide the tip on a circular path in the anterior eye segment. A continuous motion with high repeatability is achieved. The system shows the first catheter-based application of the electromagnetic manipulation system, OctoMag, for fast and safe ophthalmic surgery that potentially reduces the risk of complications and improves precision.

IROS Conference 2014 Conference Paper

High-throughput analysis of the morphology and mechanics of tip growing cells using a microrobotic platform

Dimitrios Felekis
Hannes Vogler
Geraldo Mecja
Simon Muntwyler
Mahmut Selman Sakar
Ueli Grossniklaus
Bradley J. Nelson

We present a microrobotic platform that combines MEMS-based capacitive force sensing technology, a dual-stage positioning system and a real-time control and acquisition architecture with computer vision automation to manipulate and mechanically characterize growing plant cells. The topography accuracy of the system, using a silicon wafer sample is measured to be 28 nm(1σ, 200Hz). With an SI-traceable stiffness reference we estimate the accuracy of the RT-CFM to be 3. 49%. The target locations are selected from an interactive image of the workspace, and the sensing tip is positioned at each location using visual servoing techniques. Topography and stiffness maps were successfully obtained on growing pollen tubes. With the proposed system, cells can be mechanically stimulated at high speeds and with high precision while the intracellular components are visualized using confocal imaging. The system offers a versatile solution for dexterous and high-throughput characterization of biological specimen.

ICRA Conference 2014 Conference Paper

Non-contact manipulation for automated protein crystal harvesting using a rolling microrobot

Hsi-Wen Tung
Roel Pieters
David F. Sargent
Bradley J. Nelson

In this work, a magnetic visual control system for automated protein crystal harvesting is proposed. The system consists of a rod-shaped microrobot, a magnetic actuation system and a visual control system. A rotating magnetic field induces the microrobot to roll on the supporting surface, thereby creating a vortex in a liquid environment. This vortex enables the robot to trap and transport even delicate objects in a non-contact manner to a pre-defined position. We present the micro-agent, the actuation system and the visual control system to achieve this automated procedure.

ICRA Conference 2014 Conference Paper

Self-folding mobile microrobots for biomedical applications

Stefano Fusco
Mahmut Selman Sakar
Stephen Kennedy
Christian Peters
Salvador Pané
David Mooney
Bradley J. Nelson

The presented microrobotic platform combines the advantages of self-folding NIR light sensitive polymer bilayers, magnetic alginate microbeads, and a 3D manipulation system and introduces a solution for targeted, on-demand drug and cell delivery. First feasibility studies are presented together with the potential of the full design.

ICRA Conference 2013 Conference Paper

Three-dimensional, automated magnetic biomanipulation with subcellular resolution

Simone Schürle
Mahmut Selman Sakar
Alessandro Meo
Jens Möller
Bradley Kratochvil
Christopher S. Chen
Viola Vogel
Bradley J. Nelson

In our previous work we introduced the inverted MiniMag, a magnetic manipulation system capable of 5 degree-of-freedom (5-DOF) wireless control of micro- and nano structures. In this work, we implement a depth tracking algorithm to provide feedback on the z-position of manipulated particles in the absence of a side camera. We demonstrate 3D closed-loop servoing of magnetic microbeads along predefined trajectories. We successfully position magnetic microbeads functionalized with antibodies in close proximity to macrophages with sub-cellular resolution and record the phagocytosis of the beads. We also show that the system can be used for 3D targeted microtissue manipulation inside microfabricated devices.

ICRA Conference 2012 Conference Paper

Holonomic 5-DOF magnetic control of 1D nanostructures

Simone Schürle
Kathrin Eva Peyer
Bradley Kratochvil
Bradley J. Nelson

This paper presents a manipulation system capable of five degree of freedom (5-DOF) control of a magnetic nanoagent (3-DOF position, 2-DOF orientation) implemented on an inverted microscope. Magnetic fields up to 50 mT and gradients up to 5 T/m at frequencies up to 6 kHz can be achieved. The independent generation of field and gradient vectors enables holonomic 5-DOF wireless magnetic manipulation at the nanoscale. Multiple types of motion were investigated for nickel nanowires of different lengths and analyzed using resistive force theory.

IROS Conference 2012 Conference Paper

Movement of artificial bacterial flagella in heterogeneous viscous environments at the microscale

Kathrin Eva Peyer
Famin Qiu
Li Zhang 0010
Bradley J. Nelson

Swimming microrobots have the potential to be used in medical applications such as targeted drug delivery. The challenges for navigating microrobots in the human body lie not only in the viscosity of body fluids but also in the existence of different types of fibers and cells such as blood cells or protein strands. This paper investigates artificial bacterial flagella (ABFs), which are helical microrobots actuated by an external magnetic field, in methyl cellulose solutions of different concentrations. It can be shown that the microrobots can be propelled in these gel-like heterogeneous solutions and successful swimming was demonstrated in solutions with a viscosity of more than 20 times that of water. Furthermore, results indicate that the existence of fibers can help ABFs swim more effectively, which agrees with previous experimental results reported for natural bacteria.

ICRA Conference 2012 Conference Paper

Polymer-based Wireless Resonant Magnetic microrobots

Hsi-Wen Tung
Dominic R. Frutiger
Salvador Pané
Bradley J. Nelson

We present a class of Wireless Resonant Magnetic Microactuator (WRMMA) that integrates a polymer spring/body structure with electroplated ferromagnetic masses. The new devices, which we call PolyMites as they are derived from our previous MagMites, are simpler, faster and cheaper to fabricate than the MagMite. Like their predecessor, they are capable of moving on planar surfaces in dry and wet environments. Their improved biocompatibility also extends their potential for biological applications. PolyMites are 500 μm in diameter and 55 μm in height. In air they have attained a speed of 13 mm/s, approximately 26 body lengths per second. PolyMites are capable of micromanipulation on a surface, which is demonstrated by pushing and releasing micro-objects such as polystyrene beads in water.

ICRA Conference 2012 Conference Paper

Robust ℋ∞ control for electromagnetic steering of microrobots

Hamal Marino
Christos Bergeles
Bradley J. Nelson

Electromagnetic systems for in vivo microrobot steering have the potential to enable new types of localized and minimally invasive interventions. Accurate control of microrobots in natural fluids requires precise, high-bandwidth localization and accurate knowledge of the steering system's parameters. However, current in vivo imaging methodologies, such as fluoroscopy, must be used at low update rates to minimize radiation exposure. Low frame rates introduce localization uncertainties. Additionally, the parameters of the electromagnetic steering system are estimated with inaccuracies. These uncertainties can be addressed with robust H ∞ control, which is investigated in this paper. The controller is based on a linear uncertain dynamical model of the steering system and microrobot. Simulations show that the proposed control scheme accounts for modeling uncertainties, and that the controller can be used for servoing in low viscosity fluids using low frame rates. Experiments in a prototype electromagnetic steering system support the simulations.

ICRA Conference 2011 Conference Paper

Adaptive backstepping and MEMS force sensor for an MRI-guided microrobot in the vasculature

Laurent Arcese
Matthieu Fruchard
Felix Beyeler
Antoine Ferreira
Bradley J. Nelson

-A microrobot consisting of a polymer binded aggregate of ferromagnetic particles is controlled using a Magnetic Resonance Imaging (MRI) device in order to achieve targeted therapy. The primary contribution of this paper is the design of an adaptive backstepping controller coupled with a high gain observer based on a nonlinear model of a microrobot in a blood vessel. This work is motivated by the difficulty in accurately determining many biological parameters, which can result in parametric uncertainties to which model-based approaches are highly sensitive. We show that the most sensitive parameter, magnetization of the microrobot, can be measured using a Micro-Electro-Mechanical Systems (MEMS) force sensor, while the second one, the dielectric constant of blood, can be estimated on line. The efficacy of this approach is illustrated by simulation results.

ICRA Conference 2011 Conference Paper

Model-based localization of intraocular microrobots for wireless electromagnetic control

Christos Bergeles
Bradley Kratochvil
Bradley J. Nelson

The automated or semiautomated control of microrobots for targeted drug delivery, retinal surgeries, and other ophthalmic procedures has the potential to significantly augment the capabilities of human surgeons. For accurate magnetic control, the position of the microrobots is required. In this paper, we extract the intraocular projection mapping and use it in the established CAD-model-based pose-estimation framework. Our algorithm requires no focus information. This is the first work that treats a complicated refractive imaging system, such as the eye, in a model-based localization framework. We design a model-eye chamber with human-like optical elements in which we control a microrobot and estimate its position using monocular vision.

IROS Conference 2011 Conference Paper

MRI magnetic signature imaging, tracking and navigation for targeted micro/nano-capsule therapeutics

David Folio
Christian Dahmen
Tim Wortmann
M. Arif Zeeshan
Kaiyu Shou
Salvador Pané
Bradley J. Nelson
Antoine Ferreira

The propulsion of nano-ferromagnetic objects by means of MRI gradients is a promising approach to enable new forms of therapy. In this work, necessary techniques are presented to make this approach work. This includes path planning algorithms working on MRI data, ferromagnetic artifact imaging and a tracking algorithm which delivers position feedback for the microdevice and a propulsion sequence to enable interleaved magnetic propulsion and imaging. Using a dedicated software environment integrating path-planning methods and real-time tracking, a clinical MRI system is adapted to provide this new functionality for potential controlled interventional targeted therapeutic applications. Through MRI-based sensing analysis, this paper aims to propose a framework to plan a robust pathway to enhance the navigation ability to reach deep locations in human body. The proposed approaches are validated with different experiments.

IROS Conference 2011 Conference Paper

The Cellular Force Microscope (CFM): A microrobotic system for quantitating the growth mechanics of living, growing plant cells in situ

Dimitrios Felekis
Simon Muntwyler
Felix Beyeler
Bradley J. Nelson

As the field of biology becomes a more quantitative and predictive natural science, an increasing need for investigation and quantification of the mechanics of growth at individual cellular levels arises. This paper describes a microrobotic force-feedback based system and its application to the mechanical characterization of living, growing plant cells. The Cellular Force Microscope (CFM) is capable of performing the automated mechanical characterization of living plant cells in situ as these cells proliferate and grow. The microrobotic measurement system employs a single-axis capacitive MEMS microforce sensor capable of resolving forces down to 20 nN (1¿, at 10Hz). A multi-axis positioning system with 5 nm resolution position feedback is integrated into a complete system with a high-resolution optical microscope and a custom user interface for guiding an automated force-based measurement process. The CFM has been applied to characterize the mechanical properties of 20¿m wide Lilium pollen tubes while they grow at a rate of about 10 ¿m/min in growth medium. For the mechanical characterization of pollen tubes, loads up to 400 nN are applied that cause indentations up to 300 nm. The force-deformation data acquired show an increase in the observed stiffness from the tip to the apex demonstrating that CFM is a promising tool for better understanding the changing mechanics of living plant cell growth.

ICRA Conference 2010 Conference Paper

MagMites - Microrobots for wireless microhandling in dry and wet environments

Dominic R. Frutiger
Bradley Kratochvil
Bradley J. Nelson

Central to the challenge of building sub-mm robots, or microrobots, is the development of effective power storage and locomotion mechanisms. In 2007 we introduced the Wireless Resonant Magnetic Micro-actuator (WRMMA) and its application in a successful microrobotic platform, the MagMite.

ICRA Conference 2010 Conference Paper

Modeling and analysis of wireless resonant magnetic microactuators

Zoltán Nagy 0002
Dominic R. Frutiger
Remco I. Leine
Christoph Glocker
Bradley J. Nelson

We present a dynamic model of the wireless resonant magnetic microactuator (WRMMA), which is a key component of the MagMite family of microrobots. We analyze the interbody force and integrate the nonsmooth and nonlinear equations of motion using a time-stepping integration scheme. We investigate the influence of system parameters, such as friction, the frequency of the applied force, the magnitude of the applied field, the effect of a clamping force, and the effect on velocity when phase shifting the clamping signal with respect to the magnetic signal. Our results are qualitatively consistent with experimental observations, and explain several nonintuitive phenomena. We show that the robots are highly sensitive to the phase of the clamping force, that the velocity can switch directions with changing frequency, and that both erratic and controlled motion occur under specific conditions.

ICRA Conference 2010 Conference Paper

Non-ideal swimming of artificial bacterial flagella near a surface

Kathrin Eva Peyer
Li Zhang 0010
Bradley Kratochvil
Bradley J. Nelson

The artificial bacterial flagellum (ABF), a helical swimming microrobot, has the potential to be used for biomedical applications such as cellular and intracellular manipulation. The velocity and the propulsive force of the ABF can be controlled by the input frequency of the rotating magnetic field. In this paper the swimming behavior of the ABF near a solid surface is reported. Three regions have been observed for the frequency-dependent swimming behavior of the ABF, i. e. the step-out, the linear and the drift-dominated region. At low frequencies it has been found that the desired screw-type motion is replaced by a wobbling swimming movement with a frequency-dependent precession angle. Moreover, the experimental results show that the wobbling motion of the ABF enhances the undesired sidewise drift due to wall effects. Additionally, the cause of the precession motion has been investigated by a hydrodynamic model. Our results imply that the linear range of the input magnetic frequency and the output ABF velocity is not only limited by the applicable torque at high frequencies but also by the wobbling of helical swimming at low frequencies.

ICRA Conference 2010 Conference Paper

OctoMag: An electromagnetic system for 5-DOF wireless micromanipulation

Bradley Kratochvil
Michael P. Kummer
Jake J. Abbott
Ruedi Borer
Olgaç Ergeneman
Bradley J. Nelson

We demonstrate five-degree-of-freedom (5-DOF) wireless magnetic control of a fully untethered microrobot with a magnetic steering system we call OctoMag. Although only occupying a single hemisphere, this system is capable of isotropically applying forces on the order of 1–40 µN with unrestricted control of the 2 orienting DOF. These capabilities are enabled through the use of soft-magnetic-cores which provide an increase of approximately 20× that of air cores in magnetic-field strength, but comes at the cost of more complicated interactions between coils. We propose a modeling mechanism that assumes the field contributions of the individual currents superimpose linearly when using cores with large linear regions and negligible hysteresis. When designing the system, the locations and quantity of electromagnets were optimized with regards to the force generation in the worst-case direction predicted by the model. The resultant system is capable of both open and closed-loop operation over a workspace of 4 cm 3. OctoMag was primarily designed for the control of intraocular microrobots for delicate retinal procedures, but also has potential uses in other medical applications or micromanipulation under an optical microscope.

ICRA Conference 2010 Conference Paper

OctoMag: An electromagnetic system for 5-DOF wireless micromanipulation

Michael P. Kummer
Jake J. Abbott
Bradley Kratochvil
Ruedi Borer
Ali Sengul
Bradley J. Nelson

We demonstrate five-degree-of-freedom (5-DOF) wireless magnetic control of a fully untethered microrobot (3-DOF position, 2-DOF pointing orientation). The microrobot can move through a large workspace, and is completely unrestrained in the rotation degrees of freedom. We accomplish this level of wireless control with an electromagnetic system that we call OctoMag. OctoMag's unique abilities are due to its utilization of complex nonuniform magnetic fields, which capitalizes on a linear representation of the coupled field contributions of multiple soft-magnetic-core electromagnets acting in concert. OctoMag was primarily designed for the control of intraocular microrobots for delicate retinal procedures, but it also has potential uses in other medical applications or micromanipulation under an optical microscope.

ICRA Conference 2010 Conference Paper

Rotary nanomotors based on head-to-head nanotube shuttles

Mustapha Hamdi
Arunkumar Subramanian
Lixin Dong
Antoine Ferreira
Bradley J. Nelson

A novel rotary nanomotor is described using two axially aligned, opposing chirality nanotube shuttles. Based on inter-shell screw-like motion of nanotubes, rotary motion is generated by electrostatically pulling the two cores together. Simulations using molecular dynamics show the generation of rotation from armchair nanotube pairs and their actuation properties. The simulation results, together with recently reported progress in realizing batch-fabricated ultra-high density nanotube shuttles, point towards the use of these motors as building blocks in nanoelectromechanical systems (NEMS) and nanorobotic systems for sensing, actuation, and computation applications.

ICRA Conference 2010 Conference Paper

Three-axis micro-force sensor with tunable force range and sub-micronewton measurement uncertainty

Simon Muntwyler
Felix Beyeler
Bradley J. Nelson

This paper presents the design, fabrication and characterization of the first microfabricated three-axis tunable force sensor with sub-micronewton measurement uncertainty. The sensing range can be electrically tuned from +/-20 μN to +/-200 μN while taking measurements, ensuring optimal sensor characteristics for a large variety of applications. Since the sensor has been pre-calibrated for the entire tuning range the exact sensor gain and its uncertainty is known. Real-time, three degree of freedom force feedback makes this sensor a valuable tool for micromanipulation.

ICRA Conference 2009 Conference Paper

A wireless acoustic emitter for passive localization in liquids

Zoltán Nagy 0002
Michael Flückiger
Olgaç Ergeneman
Salvador Pané
Martin Probst
Bradley J. Nelson

For the localization of minimally invasive medical devices, such as capsule endoscopes in the human body, ultrasound combines good resolution, minimal adverse health effects, high speed, adequate frame rates, and low cost. In the case of capsule endoscopes, small onboard ultrasonic emitters with minimal power requirements have the potential to provide significantly enhanced localization. We demonstrate for the first time acoustic emission in the kHz range using a wireless emitter based on the actuation principle of the wireless resonant magnetic microactuator developed recently in our institute. Our experiments show good agreement with the theoretical model, and simulations show the potential for high resolution localization.

ICRA Conference 2009 Conference Paper

Design and calibration of a microfabricated 6-axis force-torque sensor for microrobotic applications

Felix Beyeler
Simon Muntwyler
Bradley J. Nelson

This work describes the design of a capacitive multi-axis force-torque sensor for the monitoring of forces in the sub-milli-Newton and sub-micro-Newtonmeter range. This force range makes it a valuable tool for microrobotic applications. The sensor is experimentally investigated and calibrated. This is the first microfabricated 6-axis force-torque sensor that has been successfully developed.

IROS Conference 2009 Conference Paper

Metal-filled carbon nanotubes for nanofluidic systems: Modes of melting and evaporation

Lixin Dong
Xinyong Tao
Li Zhang 0010
Xiaobin Zhang
Bradley J. Nelson

Modes of melting and evaporation of metal at attogram level from individual carbon nanotubes (CNTs) are investigated experimentally using nanorobotic manipulation inside a transmission electron microscope. We compared the melting and evaporation induced by electric current, Joule heating, charge, and ionization. Experiments show that the most effective method is by positively ionizing the encapsulated metal, therefore, an electrostatic field can be used to guide the flow. Applications and potential applications of mass transport and deposition in nanofluidic systems have been presented including self-welding, actuation, and storage.

IROS Conference 2009 Conference Paper

Micromanipulation using artificial bacterial flagella

Li Zhang 0010
Jake J. Abbott
Lixin Dong
Bradley Kratochvil
Haixin Zhang
Kathrin Eva Peyer
Bradley J. Nelson

Artificial bacterial flagella (ABF) are swimming microrobots that mimic the swimming motion of bacteria. The helical swimmer consists of an InGaAs/GaAs/Cr helical nanobelt tail fabricated by a self-scrolling technique with dimensions similar to a natural flagellum, and a thin soft-magnetic metal ¿head¿ consisting of a Cr/Ni/Au multi-layer. The swimming locomotion of ABF is precisely controlled in 3-D by external rotating magnetic fields. Microsphere manipulation is performed by ABF, and experimental results show that both the position and the orientation of microspheres can be precisely controlled. The propelling force of ABF is in the pico-Newton range. We also describe a swarm-like behavior in which three ABF swim in a pack, indicating the potential to handle several micro objects in parallel. Self-propelled devices such as these are candidates for wireless 6-DOF micro and nanomanipulation tools for handling cellular and sub-cellular objects.

IROS Conference 2009 Conference Paper

Morphology detection for magnetically self-assembled modular robots

Zoltán Nagy 0002
Shuhei Miyashita
Simon Muntwyler
Ashish Cherukuri
Jake J. Abbott
Rolf Pfeifer
Bradley J. Nelson

Self-assembly is a process in which individual components form an organized structure as a consequence of local interactions. When using magnetics to create interaction forces, the magnetic flux distribution of a self-assembling system changes as its assembly state varies. Since Hall effect sensors are a convenient and effective means to detect changes in the magnetic field, we explore their applicability to monitoring the morphology of such magnetically self-assembling systems. We find that optimal positions for the sensor can be found where the flux changes maximally. Our analysis is applied to two different systems by deriving the flux changes for all possible states, and theoretical flux changes are verified with experiments. In addition, we show that a small number of sensors is sufficient for robust state determination. In addition to state detection, experiments show the potential for angle measurement for compliant cylindrical magnet joints using a single Hall sensor.

ICRA Conference 2009 Conference Paper

Tracking intraocular microdevices based on colorspace evaluation and statistical color/shape information

Christos Bergeles
Georgios Fagogenis
Jake J. Abbott
Bradley J. Nelson

Successful ophthalmic surgeries using intraocular untethered microrobots or tethered robotic microtools require methods to robustly track the microdevices in the posterior of the human eye. The dimensions and specularities of the microdevices are major obstacles for accurate tracking. In addition, the optical structure of the human eye makes it challenging to keep the objects of interest constantly in focus, resulting in blurred images. In this paper, the advantages of using different colorspaces for intraocular tracking are examined. After selection of the appropriate colorspace, thresholds that ensure maximum separation of the device from the background are calculated. Based on trained color histograms, level sets are used to track in real time, and the use of statistical shape information is incorporated in the existing tracking framework. The efficacy of the algorithm is demonstrated by tracking a microrobot in a model eye, using a custom made ophthalmoscope and off-the-shelf ophthalmoscopy lenses. With the appropriate colorspace and threshold selection, tracking errors are minimized and are further diminished using shape information.

ICRA Conference 2009 Conference Paper

Visual servoing and characterization of resonant magnetic actuators for decoupled locomotion of multiple untethered mobile microrobots

Bradley Kratochvil
Dominic R. Frutiger
Karl Vollmers
Bradley J. Nelson

Wireless resonant magnetic micro-actuators have been previously described as highly effective propulsion mechanisms for untethered mobile microrobots. The discussion thus far has been primarily relegated to a characterization of stationary devices and the de facto observation of their mobility. Before applications of microrobots can be more fully explored, devices are required that can operate reliably and repeatably in a host of operating environments. In this paper, we analyze the in situ performance of resonant magnetic actuators for microrobotic locomotion to better understand their durability, substrate requirements, and driving characteristics.

IROS Conference 2009 Conference Paper

Wide-angle localization of intraocular devices from focus

Christos Bergeles
Kamran Shamaei
Jake J. Abbott
Bradley J. Nelson

Future retinal therapies will be partially automated in order to increase the surgeons' ability to operate near the sensitive structure of the human eye retina. Untethered robotic devices that achieve the desired precision have been proposed, but require localization information for their control. Since the interior of the human eye is externally observable, vision can be used for localization. Previously, a focus-based paraxial localization algorithm using a mechatronic vitreoretinal ophthalmoscope(MVO) was proposed and evaluated by the authors. In this paper, the first algorithm for wide-angle intraocular localization is presented. The effectiveness of this new localization approach is demonstrated by experiments using a model eye and a customized MVO, and there is clear improvement over previously reported results.

IROS Conference 2008 Conference Paper

Experimental investigation of magnetic self-assembly for swallowable modular robots

Zoltán Nagy 0002
Raymond Oung
Jake J. Abbott
Bradley J. Nelson

There is a clear trend toward the miniaturization of medical devices for minimally invasive medical procedures, ranging from diagnosis and targeted drug delivery to complex surgical interventions. Current research focuses on increasing the functionality of commercially successful capsule endoscope technology by developing active locomotion and telemetry. The size of such a capsule must not be larger than what a person can swallow without difficulty. One approach to increase functionality while still working within this size constraint is to build a modular robotic system in which the modules are swallowed one at a time, and the final assembly is performed inside the gastrointestinal (GI) tract. This paper addresses a fundamental challenge that must be be met for the success of such swallowable modular robots-their self-assembly. We propose to use magnets in a specific configuration on the mating faces of the modules. Our results show that high success rates can be achieved and snake-type robots can be self-assembled with compliant magnetic joints allowing them to adapt to highly irregular paths, such as the small intestine.

ICRA Conference 2008 Conference Paper

Magmites - wireless resonant magnetic microrobots

Dominic R. Frutiger
Bradley Kratochvil
Karl Vollmers
Bradley J. Nelson

Primary challenges in the building of untethered sub-millimeter sized robots include power supply, propulsion methods, and control. We present a novel type of microrobot termed Magmite that utilizes a new class of wireless magnetic actuator which accomplishes all three tasks. The device harvests magnetic energy from the environment and effectively transforms it into mechanical propulsion while being fully controllable. This microrobotic agent with dimensions less than 300 μm × 300 μm × 70 μm is capable of maneuvering with 3 degrees of freedom. A specially prepared substrate allows for adjustable speeds exceeding 12. 5 mm/s or 42 times the robot’s body length per second (see accompanying video). It is powered by oscillating fields in the kHz range and strengths as low as 2 mT - roughly 50 times the average earth magnetic field.

ICRA Conference 2008 Conference Paper

Modeling assembled-MEMS microrobots for wireless magnetic control

Zoltán Nagy 0002
Olgaç Ergeneman
Jake J. Abbott
Marco Hutter 0001
Ann M. Hirt
Bradley J. Nelson

Capitalizing on advances in CMOS and MEMS technologies, microrobots have the potential to dramatically change many aspects of medicine by navigating bodily fluids to perform targeted diagnosis and therapy. Onboard energy storage and actuation is very difficult at the microscale, but externally applied magnetic fields provide an unparalleled means of wireless power and control. Recent results have provided a model for accurate real-time control of soft-magnetic bodies with axially symmetric geometries. In this paper, we extend the model to consider the real-time control of assembled-MEMS devices that may have significantly more complex geometries. We validate the model through FEM and experiments. The model captures the characteristics of complex 3-D structures and allows us, for the first time, to consider full 6-DOF control of untethered devices, which can act as in vivo microrobots or as end-effectors of micromanipulation systems.

IROS Conference 2008 Conference Paper

Nanohelices as motion converters

Bradley Kratochvil
Lixin Dong
Li Zhang 0010
Jake J. Abbott
Bradley J. Nelson

Few rotational actuators currently exist with the ability to transmit motion at different speeds, torques, and directions at the nanometer scale. We present work regarding the application of helical nanobelts as linear-to-rotary and rotary-to-linear motion converters. We discuss their ability to rectify device rotation to linear motion for untethered microrobotic applications as well as their application as rotary sample stages for nanoscale imaging.

IROS Conference 2008 Conference Paper

NEMS-on-a-tip: Force sensors based on electromechanical coupling of individual multi-walled carbon nanotubes

Kaiyu Shou
Lixin Dong
Bradley J. Nelson

This paper introduces the concept of nanoelectromechanical systems on a tip (NEMS-on-a-tip). As an example, force sensors based on the interlayer electromechanical coupling of individual multi-walled carbon nanotubes (MWNT) are designed with various configurations. The devices are fabricated and characterized using nanorobotic manipulation inside a scanning electron microscope (SEM). Experimental results verify that the resistance of MWNTs is highly sensitive to applied mechanical load. A further performance analysis also indicates that a force sensor configuration in which two electrodes connect to the inner and outermost shells, respectively, provides the highest sensitivity and resolution (∼1nN) in the presence of lateral forces.

IROS Conference 2008 Conference Paper

Shaping electrodes for ultrahigh precision dielectrophoretic manipulation of carbon nanotubes

Didi Xu
Arunkumar Subramanian
Lixin Dong
Bradley J. Nelson

To achieve ultrahigh precision dielectrophoretic (DEP) manipulation of carbon nanotubes (CNTs), a novel technique is investigated both theoretically and experimentally by shaping the local geometries of nanoelectrodes to control the electrohydrodynamic behavior of CNTs. Motion trajectories and positions of CNTs assembled on electrodes are predicted based on calculated DEP forces and torques. Both simulation and experimental results show that the geometries of two opposing electrodes significantly affect the precision and robustness with which CNTs can be deposited. Investigation of an electrode array verifies our model and demonstrates that the spacing between neighboring electrode pairs should be larger than twice the width of electrodes to avoid overlapping of electric fields and imbalance of DEP forces; otherwise unequally distributed electric fields and DEP forces induce a significant number of assembly failures in the array.

IROS Conference 2008 Conference Paper

The Cyborg Fly: A biorobotic platform to investigate dynamic coupling effects between a fruit fly and a robot

Chauncey F. Graetzel
Vasco Medici
Nicola Rohrseitz
Bradley J. Nelson
Steven N. Fry

The robust and efficient flight control of insects provide a powerful model system for autonomous microrobots. Conversely, robots offer a robust experimental platform on which to test biological hypotheses. This interaction of biology and robotics is an exciting but challenging task, because the vast disparities between both can lead to inaccurate or even misleading conclusions. In this paper, we present a biorobotic platform that can arbitrarily define the dynamic couplings between a fruit fly and a robot. The platform is used to explore the stability and emergent properties of the biorobotic couple. The fruit fly’s wing kinematics are measured in real time and used to drive an autonomous robot. In turn, the robot’s sensory information is transformed back into visual feedback to the fly. Using different case studies, we explore how the choice of feedback influences the success of the biorobotic device. We discuss the meaning of such feedback in view of biomimetic implementations.

IROS Conference 2008 Conference Paper

Toward targeted retinal drug delivery with wireless magnetic microrobots

Gorkem Dogangil
Olgaç Ergeneman
Jake J. Abbott
Salvador Pané
Heike Hall
Simon Muntwyler
Bradley J. Nelson

Retinal vein occlusion is an obstruction of blood flow due to clot formation in the retinal vasculature, and is among the most common causes of vision loss. Currently, the most promising therapy involves injection of t-PA directly into small and delicate retinal vessels. This procedure requires surgical skills at the limits of human performance. In this paper, targeted retinal drug delivery with wireless magnetic microrobots is proposed. We focus on four fundamental issues involved in the development of such a system: biocompatible coating of magnetic microrobots, diffusion-based drug delivery, characterization of forces needed to puncture retinal veins, and wireless magnetic force generation. We conclude that targeted drug delivery with magnetic microrobots is feasible from an engineering perspective, and the idea should now be explored for clinical efficacy.

IROS Conference 2007 Conference Paper

A multi-axis MEMS force-torque sensor for measuring the load on a microrobot actuated by magnetic fields

Felix Beyeler
Simon Muntwyler
Zoltán Nagy 0002
Mathias Franz Karl Moser
Bradley J. Nelson

This paper presents the design of a multi-axis micro force-torque sensor. The sensor is able to measure forces along two axes and a torque perpendicular to these forces. The load is measured by capacitive comb drives which provide a high sensitivity. The microfabrication process, the sensor readout electronics as well as the calibration procedure are presented. The sensor was used to measure the force and torque on a magnetically actuated microrobot. This microrobot is assembled from microfabricated nickel parts for directed drug delivery inside the human body. Precise knowledge load on the microrobot is required for accurate positioning and control of the robot. The three-axis micro sensor is used to simultaneously measure the forces and torques acting on the microrobot in a magnetic field and thus provides valuable data for magnetic control methods of microrobots.

ICRA Conference 2007 Conference Paper

Flagella-like Propulsion for Microrobots Using a Nanocoil and a Rotating Electromagnetic Field

Dominik J. Bell
Stefan Leutenegger
K. Magnus Hammar
Lixin Dong
Bradley J. Nelson

A propulsion system similar in size and motion to the helical bacterial flagella motor is presented. The system consists of a magnetic nanocoil as a propeller (27 nm thick ribbon, 3 μm in diameter, 30-40 μm long) driven by an arrangement of macro coils. The macro coils generate a rotating field that induces rotational motion in the nanocoil. Viscous forces during rotation result in a net axial propulsion force on the nanocoil. Modeling of fluid mechanics and magnetics was used to estimate the requirements for such a system. The fabrication of the magnetic nanocoils and the system setup are explained. Experimental results from electromagnetic actuation of nanocoils as well as from their propulsion in both paraffin oil and water are presented. This is the first time a propulsion system of this size and motion-type has been fabricated and experimentally verified.

ICRA Conference 2007 Conference Paper

Measuring the Magnetic and Hydrodynamic Properties of Assembled-MEMS Microrobots

Michael P. Kummer
Jake J. Abbott
Karl Vollmers
Bradley J. Nelson

Microrobots experience physical phenomena that are difficult to model analytically and that are not completely captured with macro-scale prototypes. In this paper we present a reconfigurable robotic measurement system to characterize the magnetic and hydrodynamic properties of assembled-MEMS microrobots. The system consists of a powerful permanent magnet that is position controlled with a linear stage. The magnetic field is accurately characterized. Precision sensors are used to measure magnetic force as a function of applied field. The system is first used to validate an existing model for the magnetic force on a soft-magnetic ellipsoid. Next, the magnetic force on a soft-magnetic assembled-MEMS microrobot as a function of the applied field is measured experimentally. Finally, a vision tracking system is integrated with the setup to measure the hydrodynamic properties of the microrobot. The coefficient of viscous friction for the microrobot is obtained experimentally.

ICRA Conference 2007 Conference Paper

Nanorobotic Spot Welding by Attogram Precision Metal Deposition from Copper-filled Carbon Nanotubes

Lixin Dong
Xinyong Tao
Li Zhang 0010
Xiaobin Zhang
Bradley J. Nelson

Nanorobotic spot welding using single-crystalline-copper-filled carbon nanotubes (CNTs) is investigated experimentally inside a transmission electron microscope (TEM). Controlled melting and flowing of copper inside nanotube shells are realized by applying bias voltages between 1. 5 V and 2. 5 V. The average mass flow rate of the copper was found to be 120 ag/s according to TEM video imaging (measured visually at approximately 11. 6 nm/s through the CNT). Successful soldering of a copper-filled CNT onto another CNT using a nanorobotic manipulator shows promise for nano spot welding, which can play a role similar to its macro counterpart for the interconnection of nano building blocks for the assembly of nanoelectronic circuits and nanoelectromechanical systems (NEMS).

ICRA Conference 2006 Conference Paper

Calibration of Multi-axis MEMS Force Sensors using the Shape from Motion Method

Yu Sun 0001
Keekyoung Kim
Richard M. Voyles
Bradley J. Nelson

This paper presents a new design of a two-axis MEMS (microelectromechanical systems) capacitive force sensor with strict linearity and a new sensor calibration method for micro-sensors. Precise calibration of multi-axis micro force sensors is difficult for several reasons, including the need to apply many known force vectors at precise orientations at the micro force scale, and the risk of damaging the small, fragile MEMS device. In this paper the shape from motion method is introduced for micro force sensors resulting in a rapid and effective calibration technique. Structural-electrostatic coupled field simulations are conducted in order to optimize the sensor design, which is calibrated with the shape from motion method as well as the least squares method for comparison purposes. Calibration results demonstrate that the shape from motion method is an effective, practical, and accurate method for calibrating multiaxis micro force sensors

IROS Conference 2006 Conference Paper

Design of a Micro-Gripper and an Ultrasonic Manipulator for Handling Micron Sized Objects

Felix Beyeler
Dominik J. Bell
Bradley J. Nelson
Yu Sun 0001
Adrian Neild
S. Oberti
Jurg Dual

This work reports on a system consisting of a MEMS (microelectromechanical system) gripper and an ultrasonic manipulator. The gripper is electrostatically actuated and includes an integrated force sensor measuring the gripping force. The device is monolithically fabricated using a silicon-on-insulator (SOI) fabrication process. The resolution of the force sensor is in the sub-micronewton range and, therefore, provides feedback of the forces that dominate the micromanipulation processes. A MEMS ultrasonic device is described which aligns small objects such as biological cells prior to manipulation with the gripper. The concept is demonstrated with polymer spheres, glass spheres and Hela cancer cells, thus providing a useful tool in micro-robotics and biological research

ICRA Conference 2006 Conference Paper

Hybrid Nanorobotic Approaches for Fabricating NEMS from 3D Helical Nanostructures

Lixin Dong
Li Zhang 0010
Dominik J. Bell
Bradley J. Nelson
Detlev Grützmacher

Robotic manipulation at the nanometer scale is a promising technology for structuring, characterizing and assembling nano building blocks into nanoelectromechanical systems (NEMS). Combined with recently developed nanofabrication processes, a hybrid approach to building NEMS from SiGe/Si/Cr nanocoils and Si/Cr nanospirals is presented. Nanosensors and nanoactuators are investigated from experimental, theoretical, and design perspectives

ICRA Conference 2005 Conference Paper

A Deformable Object Tracking Algorithm Robust to Occlusions and Spurious Edges

Michael A. Greminger
Bradley J. Nelson

Deformable object tracking is used in many robotics applications including biomanipulation, vision-based force sensing, and the control of deformable structures. A tracking algorithm that is robust to occlusions and to spurious edges is essential since these situations can arise unexpectedly in the unstructured environments in which robots must operate. This paper presents a deformable object tracking algorithm that is robust to occlusion and to spurious edges. Robust statistical methods are used to handle occlusion and a modification of the Canny edge detector is presented to handle spurious edges. The modification of the Canny edge operator makes use of information about the object being tracked in order to eliminate spurious edges. The deformable object tracking algorithm's performance is evaluated visually and quantitively by tracking a four degree-of-freedom compliant gripper.

IROS Conference 2005 Conference Paper

A four degree of freedom MEMS microgripper with novel bi-directional thermal actuators

Michael A. Greminger
A. Serdar Sezen
Bradley J. Nelson

A four degree of freedom thermally actuated MEMS microgripper is presented in this paper. Each jaw of the microgripper has independent x and y degrees of freedom. These extra degrees of freedom give the gripper added dexterity for manipulating microparts. The motion of each gripper jaw is provided by a two degree of freedom compliant mechanism which is based on a five bar rigid link mechanism. This gripper also introduces a novel thermal microactuator design that is capable of bi-directional actuation giving it a greater range of motion than previous thermal actuator designs. The actuator provides a total range of motion of 12. 7 /spl mu/m and a maximum force of 1. 9 mN. Also, since this microgripper is based on a compliant mechanism, deformable object tracking can be used to provide force as well as position feedback for the gripper. This combination of an increased number of degrees of freedom and increased sensory feedback provides a level of dexterity that has not been previously available in microassembly.

IROS Conference 2005 Conference Paper

Autofocusing algorithm selection in computer microscopy

Yu Sun 0001
Stefan Duthaler
Bradley J. Nelson

Autofocusing is a fundamental technology for automated biological and biomedical analyses and is indispensable for routine use of microscopes on a large scale. This paper presents a comprehensive comparison study of 18 focus algorithms in which a total of 139, 000 microscope images are analyzed. Six samples were used with three observation methods (bright field, phase contrast, an d differential interference contrast (DIC)) under two magnifications (100/spl times/ and 400/spl times/). A ranking methodology is proposed, based on which the 18 focus algorithms are ranked. Image pre-processing is also conducted to extensively reveal the performance and robustness of the focus algorithms. The presented guidelines allow for the selection of the optimal focus algorithm for different microscopy applications.

IROS Conference 2005 Conference Paper

Single nanotube array based nano encoders

Lixin Dong
Arunkumar Subramanian
Bradley J. Nelson
Yu Sun 0001

Linear encoders for nanoscale position sensing based on single carbon nanotube (CNT) arrays are presented. Vertically aligned single multi-walled carbon nanotubes (MWNTs) are realized using a combination of e-beam lithography and plasma-enhanced chemical vapour deposition (PECVD) growth. Electron beam lithography is used to define 50-150 nm nickel catalyst dots at precise locations on a silicon chip. Precise control of the position, density and alignment of the tubes has been achieved. Aligned nanotube arrays with spacing varying from 250 nm to 25 /spl mu/m are realized. Field emission properties of the array are investigated inside a scanning electron microscope (SEM) equipped with a 3-DOF nanorobotic manipulator with nanometer resolution functioning as a scanning anode. With this scanning anode and the single MWNT array, a nano encoder is investigated experimentally. Vertical position is detected by the change in emission current, whereas the horizontal position of the scanning anode is sensed from the emission distribution. A resolution of 98. 3 nm in the vertical direction and 38. 0 nm (best: 12. 9 nm) in the lateral direction has been achieved.

ICRA Conference 2005 Conference Paper

Towards Linear Nano Servomotors with Integrated Position Sensing

Lixin Dong
Bradley J. Nelson
Toshio Fukuda
Fumihito Arai
Masahiro Nakajima

Nanoscale linear servomotors with integrated position sensing are investigated from experimental, theoretical, and design perspectives. Prismatic motion is realized using the interlayer motion of telescoping multi-walled carbon nanotubes (MWNTs). Position sensing can be achieved by monitoring field emission or by measuring resistance change between a MWNT and a gold substrate during sliding movement. Experimental results demonstrate resolution in the nanometer range. Actuation experiments demonstrate the feasibility of a linear nano servomotor with integrated position sensing based on field emission. A local “kink”-like fluctuation of emission current is observed, which is caused by the change of the protruding length of the nanotube core, thus demonstrating the potential of using emission as a “linear encoder”. Complete extension of the inner core is observed and the electrostatic force is calibrated to be tens of nano-Newtons for individual nanotubes— 13. 3 to 23. 3 nN for voltages from 20 to 30V. These results demonstrate the possibility of fabricating linear servomotors at the nanometer scale with integrated position sensing.

ICRA Conference 2004 Conference Paper

Analysis and Design of Wireless Magnetically Guided Microrobots in Body Fluids

Kemal Berk Yesin
Karl Vollmers
Bradley J. Nelson

The active guidance of magnetic particles inside biological organisms for drug delivery, cell separation, and protein manipulation has been actively pursued in biomedical research for many years. Recent advances in the integration of magnetic materials in MEMS are enabling the convergence of these two technologies towards the realization of wireless magnetically guided biomedical microrobots. This paper discusses some of the fundamental design issues for a biomedical microrobot that is actively steered in body fluids using magnetic fields. The effects of miniaturization on magnetic, fluid drag, and gravity/buoyancy forces and on control stability are analyzed. The advantages of using hard magnetic materials and the state of the art in the integration of hard magnetic materials into MEMS devices are discussed. The analysis indicates that untethered microrobots capable of being navigated under external control within biological organisms including the human body is a realistic goal.

ICRA Conference 2004 Conference Paper

Boundary Element Deformable Object Tracking with Equilibrium Constraints

Michael A. Greminger
Yu Sun 0001
Bradley J. Nelson

This paper presents a deformable object tracking algorithm based on the boundary element method (BEM). BEM differs from the finite element method (FEM) in that only the boundary of the object needs to be meshed for BEM. FEM requires that the interior of the object is meshed in addition to its boundary. This feature of BEM makes it attractive for computer vision problems. We present a deformable template that uses BEM to model deformations. This deformable template is registered to an image using an energy minimization approach. The BEM tracking algorithm presented in this paper constraints the tracking results to satisfy the condition of static equilibrium. This increases the robustness of the tracking results and enhances the usefulness of the forces obtained from the tracking procedure. We demonstrate the tracking performance of this algorithm for objects with linear and non-linear elastic properties. In addition, the results of tracking the deformations of a cell are presented.

IROS Conference 2004 Conference Paper

Guidance of magnetic intraocular microrobots by active defocused tracking

Kemal Berk Yesin
Karl Vollmers
Bradley J. Nelson

Current laparoscopic techniques for intraocular surgery require that the vitreous humor is removed and at least three cannulas are inserted through the sidewalls of the eye. This paper investigates an alternate intraocular surgical technique based on the use of wireless microrobots guided by external magnetic fields. Issues investigated include the effects of magnetic and viscous drag forces faced by magnetic microrobots in the vitreous humor and the 3D visual servoing of these microrobots using a single microscope view. A new active defocused tracking method is proposed for visually servoing the microrobot along the optical axis of the microscope. This method uses a purposely defocused view of the microrobot to unambiguously resolve depth while servoing. Experimental results demonstrating the method with a microrobot visually servoed in 3D at 60 Hz using a single microscope view are presented.

ICRA Conference 2004 Conference Paper

Investigating Protein Structure with a Microrobotic System

Yu Sun 0001
Michael A. Greminger
Bradley J. Nelson

This paper presents a microrobotic system integrating microscope vision and microforce feedback for characterizing biomembrane mechanical properties. Robust visual tracking of deformable biomembrane contour using physics-based models is described. A multi-axis MEMS-based force sensor is used to determine applied forces on biomembranes and develop a novel biomembrane mechanical model. By visually extracting geometry changes on a biomembrane during loading, geometry changes can be used to estimate applied forces using the biomembrane mechanical model and the determined elastic modulus. Forces on a biomembrane can be visually observed and controlled, thus creating a framework for vision and force assimilated cell manipulation. The experimental results quantitatively describe mouse zona pellucida (ZP) stiffness increase during ZP hardening and provide an understanding of ZP protein structure development, i. e. , an increase in the number of cross links of protein ZP1 between ZP2-ZP3 units that is conjectured to be responsible for ZP stiffness increase. Furthermore, the system, technique, and model presented in this paper can be applied to investigating mechanical properties of other biomembranes and other cell types, which has the potential to facilitate many biological studies, such as cell injury and recovery where biomembrane mechanical property changes need to be monitored.

ICRA Conference 2004 Conference Paper

Robust CAD Model Based Visual Tracking for 3D Microassembly Using Image Space Potentials

Kemal Berk Yesin
Bradley J. Nelson

A CAD model based visual tracking system for the flexible automation of 3D microassembly processes is presented. The system achieves six degrees-of-freedom tracking of MEMS components in real-time (30 Hz). The experimental results indicate a 1/spl sigma/ precision of 1. 0 /spl mu/m and a positioning accuracy of 2%. A major source of errors in model based tracking is due to ambiguous object configurations with closely spaced edges. We also present a real-time image space potential method that utilizes the readily available model information to reduce these errors and improve tracking robustness.

ICRA Conference 2003 Conference Paper

Actively Servoed Multi-Axis Microforce Sensors

Yu Sun 0001
David P. Potasek
Damrongrit Piyabongkarn
Rajesh Rajamani
Bradley J. Nelson

This paper presents design, fabrication, and calibration results of MEMS-based two-axis capacitive force sensors capable of resolving forces up to 490/spl mu/N with a resolution of 0. 01 /spl mu/N in x, and up to 900 /spl mu/N with a resolution of 0. 24 /spl mu/N in y in the passive mode. Electrostatic microactuators are integrated to enable the force sensors to operate in an actively servoed mode, in which system stiffness is modulated using force compensation, greatly increasing force measurement dynamic ranges. When the microforce sensor is actively servoed, an externally applied force is balanced by the electrostatic forces generated by the electrostatic microactuators within the sensor. The movable parts of the sensor are maintained in the equilibrium position, making the system a regulator system. The force measurement is obtained by interpreting the actuation voltages. Probes of different shapes are integrated with the sensors for micromanipulation. Other types of end-effectors, such as microgrippers and microneedles for different micromanipulation tasks can be integrated by modifying the fabrication sequence. The current application of the force sensors is for providing real-time force feedback during microrobotic cell manipulation.

ICRA Conference 2003 Conference Paper

Micromanipulation contact transition control by selective focusing and microforce control

Ge Yang 0002
Bradley J. Nelson

A fundamental requirement of micromanipulation is to control the impact force and subsequently the contact force in the transition of the micromanipulator end-effector from noncontact to contact state. This is especially important in protecting fragile microstructures and preventing undesirable motion. This paper proposes a method of using the integration of selective focusing and microforce control to achieve fast transition control while minimizing impact force. The method is applied to contact transition in microassembly pick-and-place operations. The initial long-range approach motion of the end-effector towards its target is controlled based on focus measures computed from images captured through a microscope. When the end-effector comes into focus near the target, the system switches to microforce control to minimize impact force and to regulate the contact force. An optics model for microscope focusing is proposed to characterize the dynamic behavior of the end-effector images during the approach motion. The connection between this model and the scale-space theory of computer vision is emphasized. Three different focus measures are tested and compared in performance. The proposed method has been experimentally verified to be able to achieve fast transition control with minimal impact force.

IROS Conference 2003 Conference Paper

Modeling elastic objects with neural networks for vision-based force measurement

Michael A. Greminger
Bradley J. Nelson

This paper presents a method to model the deformation of an elastic object with an artificial neural network. The neural network is trained directly from images of the elastic object deforming under known loads. Using this process, models can be created for objects such as biological tissues that cannot be modeled by existing techniques. The neural network elastic model is used in conjunction with a deformable template matching algorithm to perform vision-based force measurement (VBFM). We demonstrate this learning method on objects with both linear and nonlinear elastic properties.

IROS Conference 2003 Conference Paper

Wavelet-based autofocusing and unsupervised segmentation of microscopic images

Ge Yang 0002
Bradley J. Nelson

This paper reports on the construction of two new focus measure operators M/sub WT//sup 1/ an M/sub WT//sup 2/ defined in the wavelet transform domain. M/sub WT//sup 2/ provides significantly better focus performance in depth resolution than previously reported spatial domain operators. M/sub WT//sup 1/ provides performance equivalent to that of the best spatial domain operator but has lower computational cost than M/sub WT//sup 2/. Both operators can be used with a wide variety of wavelet bases optimized for different applications. Selection of wavelet bases is studied based on their number of vanishing moments, size of support and symmetry. The depth resolution of these operators makes them an important cue in the segmentation of low depth-of-field microscopic images. An unsupervised segmentation technique based on graph partition is then introduced. It uses M/sub WT//sup 2/ together with proximity and image intensity as segmentation features. This segmentation method does not depend on the connection of local image features and remains robust under defocusing. Experimental results confirm the effectiveness of the proposed focus measures and the segmentation algorithm. These techniques are especially suitable for high resolution microscopic computer vision tasks in high precision micromanipulation and microassembly applications.

IROS Conference 2002 Conference Paper

A novel dual-axis electrostatic microactuation system for micromanipulation

Yu Sun 0001
Damrongrit Piyabongkarn
A. Serdar Sezen
Bradley J. Nelson
Rajesh Rajamani
Reto Schoch
David P. Potasek

This paper presents the design, fabrication, modeling, and control of a dual-axis electrostatic microactuation system. To form the 3D structure only three masks are used on silicon-on-insulator wafers using deep reactive ion etching. The bulk micromachined high aspect ratio structure produces large force output, achieving the full motion range with 10. 7 V in x and 70. 1 V in y. To provide position feedback for high precision manipulation, a capacitive position sensing mechanism, capable of resolving position changes up to 5 /spl mu/m with a resolution of 0. 01 /spl mu/m in both x and y is integrated. A nonlinear model inversion technique is proposed for nonlinear electrostatic microactuation system identification and improving system linearity and response. The effectiveness of the technique was verified in experiments. Applications of the system include micromanipulation and microassembly.

IROS Conference 2002 Conference Paper

Communication and mobility enhancements to the Scout robot

Andrew Drenner
Ian T. Burt
Bradley Kratochvil
Bradley J. Nelson
Nikolaos P. Papanikolopoulos
K. B. Yesom

Small scale distributed robotic systems are ideal for tasks that larger, more expensive robots may not be able to undertake such as surveillance or inspection in enclosed areas. Small scale robots also have the advantage of being easily portable to an area of interest. However, the advantages related to the small size and portability cause an increase in the complexity of development. In addition, the more pronounced effects of interacting with the environment in terms of mobility and communications lead to robotic systems of limited functionality. In order to address such deficiencies, several novel developments and improvements have been made to the Scout robot, which will be discussed. These improvements include the development of a communication relay system facilitating longer distance operation, an improved actuating wheel system increasing the Scout's mobility and a grappling hook system to elevate the Scout. By enhancing the Scout robotic team in this way, the functional use of the team is expanded, allowing more practical use.

ICRA Conference 2002 Conference Paper

Mobility Enhancements to the Scout Robot Platform

Andrew Drenner
Ian T. Burt
Tom Dahlin
Bradley Kratochvil
Colin McMillen
Bradley J. Nelson
Nikolaos P. Papanikolopoulos
Paul E. Rybski

When a distributed robotic system is assigned to perform reconnaissance or surveillance, restrictions inherent to the design of an individual robot limit the system's performance in certain environments. Finding an ideal portable robotic platform capable of deploying and returning information in spatially restrictive areas is not a simple task. The Scout robot, developed at the University of Minnesota, is a viable robotic platform for these types of missions. The small form factor of the Scout allows for deployment, placement, and concealment of a team of robots equipped with a variety of sensory packages. However, the design of the Scout requires a compromise in power, sensor types, locomotion, and size; together these factors prevent an individual Scout from operating ideally in some environments. Several attempts to address these deficiencies have been implemented and are discussed. Among the prototype solutions are actuating wheels, allowing the Scout to increase ground clearance in varying terrains, a grappling hook enabling the Scout to obtain a position of elevated observation, and infrared emitters to facilitate low light operation.

ICRA Conference 2002 Conference Paper

Sensing Nanonewton Level Forces by Visually Tracking Structural Deformations

Michael A. Greminger
Ge Yang 0002
Bradley J. Nelson

When assembling MEMS devices or manipulating biological cells it is often beneficial to have information about the force that is being applied to these objects. This force information is difficult to measure at these scales. We demonstrate a method to reliably measure nanonewton scale forces applied to a micro scale cantilever beam using a computer vision approach. A template matching algorithm is used to estimate the beam deflection to sub-pixel resolution in order to determine the force applied to the beam. The template, in addition to containing information about the geometry of the beam, contains information about the elastic properties of the beam. Minimizing the error between this elastic template and the actual image by means of numerical optimization techniques, we are able to measure forces to within /spl plusmn/3 nN. In addition, we also discuss how this method can be generalized to measure forces in elastic configurations other than a simple cantilever beam using a micro-tweezer as an example. This provides the opportunity for this method to be used with specially designed micromanipulators to provide force as well as vision feedback for micromanipulation tasks.

ICRA Conference 2001 Conference Paper

A Flexible Experimental Workcell for Efficient and Reliable Wafer-Level 3D Microassembly

Ge Yang 0002
James A. Gaines
Bradley J. Nelson

This paper reports on an experimental micro-assembly workcell developed for efficient and reliable 3D assembly of large numbers of micro-machined thin metal parts into micromachined holes in 4 inch silicon wafers. The major objective is to integrate techniques of micro-gripper design, microscopic imaging and high precision motion control to build a prototype system for industrial applications. The workcell consists of a multiple-view imaging system, a 4-DOF micromanipulator with high resolution rotation control, a large working space 4-DOF precision positioning system, a flexible micro-gripper, and a control software system. A piezoelectric force sensing unit is developed to be integrated with the manipulator system to enhance pickup reliability. Operations are partially guided by a human operator through a graphical user interface. This system provides a highly flexible testbed for wafer-level 3D microassembly.

ICRA Conference 2001 Conference Paper

Microrobotic Cell Injection

Yu Sun 0001
Bradley J. Nelson

Advances in microbiology demonstrate the need for manipulating individual biological cells, such as for cell injection which includes pronuclei injection and intracytoplasmic injection. Conventionally, cell injection has been conducted manually. In this paper, we present a microrobotic system capable of performing automatic embryo pronuclei DNA injection autonomously and semi-autonomously through a hybrid visual servoing control scheme. After injection, the DNA injected embryos were transferred into a pseudopregnant foster female mouse to reproduce transgenic mice for cancer studies. Experimental results show that the injection success rate was 100%. The system setup, hybrid control scheme and other important issues in this application, such as automatic focusing, are discussed.

IROS Conference 2001 Conference Paper

Using orthogonal visual servoing errors for classifying terrain

Richard M. Voyles
Amy C. Larson
Kemal Berk Yesin
Bradley J. Nelson

A novel, centimeter-scale crawling robot has been developed to address applications in surveillance, search-and-rescue, and planetary exploration. This places constraints on size and durability that minimizes the mechanism. As a result, a dual-use design employing two arms for both manipulation and locomotion was conceived. In a complementary fashion, this paper investigates the dual-use of visual servoing error. Visual servoing can be used by a mobile robot for homing and tracking. But because ground-based mobile robots are inherently planar, the control methodology (steering) is one-dimensional. The two-dimensional nature of image-based servoing leaves additional information content to be used in other contexts. We explore this information in the context of classifying terrain conditions. An outline for gait adaptation based on this is suggested for future work.

ICRA Conference 2000 Conference Paper

A Miniature Robotic System for Reconnaissance and Surveillance

Dean F. Hougen
Saifallah Benjaafar
Jordan Bonney
John Budenske
Mark Dvorak
Maria L. Gini
Howard French
Donald G. Krantz

Presents a miniature robotic system ("scout") useful for reconnaissance and surveillance missions. A large number of scout robots are deployed and controlled by humans and/or larger "ranger" robots. The specially designed and constructed scouts are extremely small (roughly 116cc volume) yet are readily deployable (by tossing or launching), have multiple mobility modes, have multiple sensing capabilities, can transmit and receive data and instructions, and have a limited capability for autonomous action. The rangers are significantly larger vehicles, based on a commercial-off-the-shelf platform, augmented with scout launchers, radios, and additional sensors. Together, the scouts and rangers form a hierarchical team capable of carrying out complex missions in a wide variety of environments.

ICRA Conference 2000 Conference Paper

Active Video System for a Miniature Reconnaissance Robot

Kemal Berk Yesin
Bradley J. Nelson
Nikolaos P. Papanikolopoulos
Richard M. Voyles
Donald G. Krantz

In this paper we present an active video module that consists of a miniature video sensor, a wireless video transmitter and a pan-tilt mechanism driven by micromotors. The video module is part of a miniature mobile robot that is projected to areas of the environment to be surveyed. A single-chip CMOS video sensor and miniature brushless DC gearmotors are used to comply with restrictions imposed by the robotic system in terms of payload weight volume and power consumption. Different types of actuation are analyzed for compatibility with a mesoscale robotic system. Applications of an active video module are discussed.

ICRA Conference 2000 Conference Paper

Micropart Feature Design for Visually Servoed Microassembly

Bharath Mukundakrishnan
Bradley J. Nelson

This project develops rules for micropart design to aid in device assemblability with visual servoing techniques by ensuring that the microparts can be easily tracked and controlled using vision feedback. A criterion is presented that estimates part trackability based on the visual appearance of the part. This criterion is then used to microfabricate features to improve part trackability and hence, the assemblability of the device. The criterion considers the feature appearance when the part lies out of the optical system's depth-of-field. A Fourier optics based approach is used to simulate the visual appearance of microparts represented by CAD models using high resolution optical systems. This simulation is used to automatically design microfabricated features on microparts. These features are used to estimate the tracking accuracy of the MEMS parts to subpixel levels using interpolation techniques in optical flow based tracking. This allows MEMS parts to be assembled with a precision on the order of 20 nm with high magnification lens, using visual servoing strategies.

ICRA Conference 2000 Conference Paper

The Effect of Material Properties and Gripping Force on Micrograsping

Yu Zhou
Bradley J. Nelson

This paper presents our work in developing a force controlled microgripper and micrograsping strategies using optical beam deflection techniques. The optical beam deflection sensor is based on modified atomic force microscopy techniques and is able to resolve forces below a nano-Newton. A variety of gripper fingers made from materials with different conductivity and surface roughness is analyzed theoretically and experimentally using the force sensor. These results provide insight into the mechanics of micromanipulation, and the results are used to develop micrograsping strategies. A design of a microfabricated force controlled microgripper is presented along with initial experimental results in applying various gripping forces to microparts. The results demonstrate the important role gripping force plays in the grasping and releasing of microparts.

ICRA Conference 1998 Conference Paper

Fusing Force and Vision Feedback for Micromanipulation

Yu Zhou
Bradley J. Nelson
Barmeshwar Vikramaditya

We present experimental results that investigate the integration of two disparate sensing modalities, force and vision, for sensor-based microassembly. By integrating these sensing modes, we are able to provide feedback in a task-oriented frame of reference over a broad range of motion with an extremely high precision. An optical microscope is used to provide visual feedback down to micron resolutions. We have developed an optical beam deflection sensor to provide nanonewton level force feedback or nanometric level position feedback. The value of integrating these two disparate sensing modalities is demonstrated during controlled micropart impact experiments. Using force feedback alone to control micropart contact transitions, impact forces of over 140 nN were generated before the desired contact force of 2 nN was achieved. When visual servoing is integrated with the force control framework, micropart impact forces of only 9 nN and final contact forces of 2 nN were easily achieved.

IROS Conference 1998 Conference Paper

Teleoperated micromanipulation within a VRML environment using Java

Joseph Alex
Barmeshwar Vikramaditya
Bradley J. Nelson

A teleoperated microassembly workcell that integrates a VRML-based virtual microworld with visual servoing micromanipulation strategies for assembly of hybrid MEMS prototypes is described. Java is used to program the VRML-based supervisory interface and to communicate with the microassembly workcell. This provides platform independence and allows remote teleoperation of the microassembly workcell over the Internet. A key aspect of our approach entails the integration of teleoperation and visual servoing strategies. This allows a supervisor to guide the task remotely, while visual servoing strategies compensate for the imprecisely calibrated microworld. Results are presented that demonstrate system performance when a supervisor manipulates a microobject remotely. Though Internet delays impact the dynamic performance of the system, teleoperated relative parts placements with submicron precision is successfully demonstrated.

ICRA Conference 1997 Conference Paper

Integrating force and vision feedback within virtual environments for telerobotic systems

Bradley J. Nelson
Pradeep K. Khosla

Traditional telerobotic systems often provide feedback to the user through a variety of sensing modalities, for example through live video imagery, force reflection, or acoustic signals. When a supervisor guides a robotic task while immersed in a virtual environment, geometric representations of the world are provided as feedback. In this case, the supervisor interacts with the environment by visually observing these virtual objects and directing their motion. At issue is how to appropriately use various sensing modalities provided by disparate sensors in a system of this configuration. This paper focuses on assimilating the disparate feedback provided by force and vision sensors for telerobotic systems guided from within virtual environments. A framework for feedback assimilation is described based on the concept of sensor resolvability. Sensor resolvability is used in two ways, to update the virtual environment and to guide the desired task in the real world. Resolvability selects the appropriate sensing modality to use in updating the virtual environment within which the supervisor is immersed. It is also used to direct semi-autonomous agents that interact directly with the real world to perform the desired task. Experimental results demonstrate the significant advantages of assimilating disparate sensory feedback throughout a telerobotic task using the concept of sensor resolvability.

ICRA Conference 1997 Conference Paper

Visually guided manipulation using active camera-lens systems

Bradley J. Nelson
Pradeep K. Khosla

Visual servoing is a robust technique for aligning both static and moving parts using imprecisely calibrated camera-lens-manipulator systems. An important limitation of these systems is the workspace within which the alignment task can be successfully performed due to the position and orientation of the camera. An active camera can extend this region, however this changes the visual representation of the task itself. Therefore, the reference input that drives the visually servoed manipulator must change appropriately. In this paper a framework that allows for camera-lens motion during visually servoed manipulation is described. The main components of the framework include object schemas and port-based agents. Object schemas represent the task internally in terms of geometric models with attached sensor mappings. Object schemas are dynamically updated by sensor feedback, and thus provide an ability to perform three dimensional spatial reasoning during task execution, a capability traditional image-based visual servoing lacks. Object schemas are also able to dynamically create desired visual representations of the task from which reference inputs for vision-based control strategies are derived. The sensor mappings of object schemas are also used to guide camera motion based on task characteristics. Port-based agents are the executors of the visual reference inputs and the camera motion commands. They interact with the real world through visual servoing control laws. Experimental results that demonstrate system capabilities and performance are presented.

ICRA Conference 1997 Conference Paper

Visually guided microassembly using optical microscopes and active vision techniques

Barmeshwar Vikramaditya
Bradley J. Nelson

Robust micromanipulation strategies are needed in order to overcome a potential technology barrier to the commercial development of complex microelectromechanical devices. In this paper we present experimental results that investigate the use of high resolution optical systems with controllable intrinsic and extrinsic parameters for microassembly applications. Results of two active vision strategies, depth-from-defocus and visual servoing, using an optical microscope are presented. Depth-from-defocus results indicate a repeatability of 8 /spl mu/m (just one order of magnitude greater than the wavelength of light) is achievable. Visual servoing results demonstrate visually servoed motion over large scales of motion with submicron repeatability has been achieved. The ability to servo over large scales is made possible by the use of a unique formulation of the image Jacobian for an optical microscope.

ICRA Conference 1996 Conference Paper

A multi-agent framework for grasping using visual servoing and collision avoidance

Lars Overgaard
Bradley J. Nelson
Pradeep K. Khosla

This paper describes a multi-agent approach to grasping that combines collision avoidance and visual servoing strategies. Control of the system is distributed among five different types of agents: link agents, joint agents, end-effector agents, task agents, and object agents. As each agent attempts to achieve a desired individual behavior, the manipulator itself exhibits an emergent behavior that avoids obstacles while approaching an object to be grasped. As the object is approached, object agents that represent the object to be grasped and the gripper performing the grasp are instantiated. The instantiation of each of these two object agents, which are guided by visual feedback, automatically changes the manipulator control structure. Once both object agents become active, closed-loop visual servoing of the end-effector during the final stage of grasping is realized. Experimental results are presented that illustrate the three types of manipulator control realized during a typical grasping task.

ICRA Conference 1995 Conference Paper

An extendable Framework for Expectation-Based Visual Servoing Using Enviroment Models

Bradley J. Nelson
Pradeep K. Khosla

Visual servoing is a manipulation control strategy that precisely positions objects using imprecisely calibrated camera-lens-manipulator systems. In order to quickly and easily integrate sensor-based manipulation strategies such as visual servoing into robotic systems, a system framework and a task representation must exist which facilitates this integration. The framework must also be extendable so that obsolete sensor systems can be easily replaced or extended as new technologies become available. In this paper we present a framework for expectation-based visual servoing which visually guides tasks based on the expected visual appearance of the task. The appearance of the task is generated by a model of the environment that uses texture-mapped geometric models to represent objects. A system structure which facilitates the integration of various configurations of visual servoing systems is presented, as well as a hardware implementation of the proposed system and experimental results using a stereo camera system.

IROS Conference 1995 Conference Paper

Fast stable contact transitions with a stiff manipulator using force and vision feedback

Bradley J. Nelson
J. Daniel Morrow
Pradeep K. Khosla

Ideal manipulator end-effector transitions from noncontact to contact states should be fast and stable with minimal impact forces and without bounce. These specifications, however are difficult to simultaneously achieve, especially for the most common manipulator force control configuration employing a wrist force sensor and a stiff manipulator position loop. In this paper the authors present a control strategy that uses high bandwidth vision feedback (30 Hz) in addition to force feedback (100 Hz) for contact transient control. A nonlinear control strategy is proposed that considers force and vision feedback simultaneously and then switches to pure force control when the camera-lens system becomes unable to accurately resolve the location of the end-effector relative to the surface to be contacted. Experimental results are presented which demonstrate that a stiff manipulator can quickly contact a stiff surface stably without bounce. Results also show that the magnitude of the impact force spike is directly related to the accuracy with which contact surfaces are visually observed.

IROS Conference 1995 Conference Paper

Vision and force driven sensorimotor primitives for robotic assembly skills

J. Daniel Morrow
Bradley J. Nelson
Pradeep K. Khosla

Integrating sensors into robot systems is an important step towards increasing the flexibility of robotic manufacturing systems. Current sensor integration is largely task-specific which hinders flexibility. The authors are developing a sensorimotor command layer that encapsulates useful combinations of sensing and action which can be applied to many tasks within a domain. The sensorimotor commands provide a higher-level in which to terminate task strategy plans, which eases the development of sensor-driven robot programs. This paper reports on the development of both force and vision driven commands which are successfully applied to two different connector insertion experiments.

ICRA Conference 1994 Conference Paper

Integrating Sensor Placement and Visual Tracking Strategies

Bradley J. Nelson
Pradeep K. Khosla

Real-time visual feedback is an important capability that many robotic systems must possess if these systems are to operate successfully in dynamically varying and imprecisely calibrated environments. An eye-in-hand system is a common technique for providing camera motion to increase the working region of a visual sensor. Although eye-in-hand robotic systems have been well-studied, several deficiencies in proposed systems make them inadequate for actual use. Typically, the systems fail if manipulators pass through singularities or joint limits. Objects being tracked can be lost if the objects become defocused, occluded, or if features on the objects lie outside the field of view of the camera. In this paper, a technique is introduced for integrating a visual tracking strategy with dynamically determined sensor placement criteria. This allows the system to automatically determine, in real-time, proper camera motion for tracking objects successfully while accounting for the undesirable, but often unavoidable, characteristics of camera-lens and manipulator systems. The sensor placement criteria considered include focus, field-of-view spatial resolution, manipulator configuration, and a newly introduced measure called resolvability. Experimental results are presented. >

ICRA Conference 1994 Conference Paper

SixDegree-of-Freedom Hand/Eye Visual Thacking with Uncertain Parameters

Nikolaos P. Papanikolopoulos
Bradley J. Nelson
Pradeep K. Khosla

Algorithms for 3D robotic visual tracking of moving targets whose motion is 3D and consists of translational and rotational components are presented. The objective of the system is to track selected features on moving objects and to place their projections on the image plane at desired positions by appropriate camera motion. The most important characteristics of the proposed algorithms are the use of a single camera mounted on the end-effector of a robotic manipulator (eye-in-hand configuration), and the fact that these algorithms do not require accurate knowledge of the relative distance of the target object from the camera frame. This fact makes these algorithms particularly useful in environments that are difficult to calibrate. The camera model used introduces a number of parameters that are estimated on-line, further reducing the algorithms' reliance on precise calibration of the system. An adaptive control algorithm compensates for modeling errors, tracking errors, and unavoidable computational delays which result from time-consuming image processing. Experimental results are presented to verify the efficacy of the proposed algorithms. These experiments were performed using a multi-robotic system consisting of Puma 560 manipulators. >