George Kantor

ICRA Conference 2025 Conference Paper

Autonomous Sensor Exchange and Calibration for Cornstalk Nitrate Monitoring Robot

• Janice Seungyeon Lee
• Thomas Detlefsen
• Shara Lawande
• Saudamini Ghatge
• Shrudhi Ramesh Shanthi
• Sruthi Mukkamala
• George Kantor
• Oliver Kroemer

Interactive sensors are an important component of robotic systems but often require manual replacement due to wear and tear. Automating this process can enhance system autonomy and facilitate long-term deployment. We developed an autonomous sensor exchange and calibration system for an agriculture crop monitoring robot that inserts a nitrate sensor into cornstalks. A novel gripper and replacement mechanism, featuring a reliable funneling design, were developed to enable efficient and reliable sensor exchanges. To maintain consistent nitrate sensor measurement, an on-board sensor calibration station was integrated to provide in-field sensor cleaning and calibration. The system was deployed at the Ames Curtis Farm in June 2024, where it successfully inserted nitrate sensors with high accuracy into $\mathbf{3 0}$ cornstalks with a $\mathbf{7 7} {\%}$ success rate.

ICRA Conference 2025 Conference Paper

SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

• Mohammad Nomaan Qureshi
• Sparsh Garg
• Francisco Yandún
• David Held
• George Kantor
• Abhisesh Silwal

Sim2Real transfer, particularly for manipulation policies relying on RGB images, remains a critical challenge in robotics due to the significant domain shift between syn-thetic and real-world visual data. In this paper, we propose SplatSim, a novel framework that leverages Gaussian Splatting as the primary rendering primitive to reduce the Sim2Real gap for RGB-based manipulation policies. By replacing traditional mesh representations with Gaussian Splats in simulators, SplatSim produces highly photorealistic synthetic data while maintaining the scalability and cost-efficiency of simulation. We demonstrate the effectiveness of our framework by training manipulation policies within SplatSim and deploying them in the real world in a zero-shot manner, achieving an average success rate of 86. 25%, compared to 97. 5% for policies trained on real-world data. Videos can be found on our project page: https://splatsim.github.io

ICRA Conference 2025 Conference Paper

Towards Over-Canopy Autonomous Navigation: Crop-Agnostic LiDAR-Based Crop-Row Detection in Arable Fields

• Ruiji Liu
• Francisco Yandún
• George Kantor

Autonomous navigation is crucial for various robotics applications in agriculture. However, many existing methods depend on RTK-GPS devices, which can be susceptible to loss of radio signal or intermittent reception of corrections from the internet. Consequently, research has increasingly focused on using RGB cameras for crop-row detection, though challenges persist when dealing with grown plants. This paper introduces a LiDAR-based navigation system that can achieve crop-agnostic over-canopy autonomous navigation in row-crop fields, even when the canopy fully blocks the inter-row spacing. Our algorithm can detect crop rows across diverse scenarios, encompassing various crop types, growth stages, illumination conditions, the presence of weeds, curved rows, and discontinuities. Without utilizing a global localization method (i. e. , based on GPS), our navigation system can perform autonomous navigation in these challenging scenarios, detect the end of the crop rows, and navigate to the next crop row autonomously, providing a crop-agnostic approach to navigate an entire field. The proposed navigation system has undergone tests in various simulated and real agricultural fields, achieving an average cross-track error of 3. 55 cm without human intervention. The system has been deployed on a customized UGV robot, which can be reconfigured depending on the field conditions.

IROS Conference 2025 Conference Paper

Transformer-Based Spatio-Temporal Association of Apple Fruitlets

• Harry Freeman
• George Kantor

In this paper, we present a transformer-based method to spatio-temporally associate apple fruitlets in stereo-images collected on different days and from different camera poses. State-of-the-art association methods in agriculture are dedicated towards matching larger crops using either high-resolution point clouds or temporally stable features, which are both difficult to obtain for smaller fruit in the field. To address these challenges, we propose a transformer-based architecture that encodes the shape and position of each fruitlet, and propagates and refines these features through a series of transformer encoder layers with alternating self and cross-attention. We demonstrate that our method is able to achieve an F1-score of 92. 4% on data collected in a commercial apple orchard and outperforms all baselines and ablations. The code and data can be found at https://kantor-lab.github.io/fruitassociator/

ICRA Conference 2024 Conference Paper

Autonomous Apple Fruitlet Sizing with Next Best View Planning

• Harry Freeman
• George Kantor

In this paper, we present a next-best-view planning approach to autonomously size apple fruitlets. State-of-the-art viewpoint planners in agriculture are designed to size large and more sparsely populated fruit. They rely on lower resolution maps and sizing methods that do not generalize to smaller fruit sizes. To overcome these limitations, our method combines viewpoint sampling around semantically labeled regions of interest, along with an attention-guided information gain mechanism to more strategically select viewpoints that target the small fruits’ volume. Additionally, we integrate a dual-map representation of the environment that is able to both speed up expensive ray casting operations and maintain the high occupancy resolution required to informatively plan around the fruit. When sizing, a robust estimation and graph clustering approach is introduced to associate fruit detections across images. Through simulated experiments, we demonstrate that our viewpoint planner improves sizing accuracy compared to state of the art and ablations. We also provide quantitative results on data collected by a real robotic system in the field.

ICRA Conference 2024 Conference Paper

Towards Robotic Tree Manipulation: Leveraging Graph Representations

• Chung Hee Kim
• Moonyoung Lee
• Oliver Kroemer
• George Kantor

There is growing interest in automating agricultural tasks that require intricate and precise interaction with specialty crops, such as trees and vines. However, developing robotic solutions for crop manipulation remains a difficult challenge due to complexities involved in modeling their deformable behavior. In this study, we present a framework for learning the deformation behavior of tree-like crops under contact interaction. Our proposed method involves encoding the state of a spring-damper modeled tree crop as a graph. This representation allows us to employ graph networks to learn both a forward model for predicting resulting deformations, and a contact policy for inferring actions to manipulate tree crops. We conduct a comprehensive set of experiments in a simulated environment and demonstrate generalizability of our method on previously unseen trees. Videos can be found on the project website: https://kantor-lab.github.io/tree_gnn

ICRA Conference 2023 Conference Paper

3D Reconstruction-Based Seed Counting of Sorghum Panicles for Agricultural Inspection

• Harry Freeman
• Eric Schneider
• Chung Hee Kim
• Moonyoung Lee
• George Kantor

In this paper, we present a method for creating high-quality 3D models of sorghum panicles for phenotyping in breeding experiments. This is achieved with a novel reconstruction approach that uses seeds as semantic landmarks in both 2D and 3D. To evaluate the performance, we develop a new metric for assessing the quality of reconstructed point clouds without ground-truth. Finally, a counting method is presented where the density of seed centers in the 3D model allows 2D counts from multiple views to be effectively combined into a whole-panicle count. We demonstrate that using this method to estimate seed count and weight for sorghum outperforms count extrapolation from 2D images, an approach used in most state of the art methods for seeds and grains of comparable size.

IROS Conference 2023 Conference Paper

3D Skeletonization of Complex Grapevines for Robotic Pruning

• Eric Schneider
• Sushanth Jayanth
• Abhisesh Silwal
• George Kantor

Robotic pruning of dormant grapevines is an area of active research in order to promote vine balance and grape quality, but so far robotic efforts have largely focused on planar, simplified vines not representative of commercial vineyards. This paper aims to advance the robotic perception capabilities necessary for pruning in denser and more complex vine structures by extending plant skeletonization techniques. The proposed pipeline generates skeletal grapevine models that have lower reprojection error and higher connectivity than baseline algorithms. We also show how 3D and skeletal information enables prediction accuracy of pruning weight for dense vines surpassing prior work, where pruning weight is an important vine metric influencing pruning site selection.

ICRA Conference 2023 Conference Paper

Occlusion Reasoning for Skeleton Extraction of Self-Occluded Tree Canopies

• Chung Hee Kim
• George Kantor

In this work, we present a method to extract the skeleton of a self-occluded tree canopy by estimating the unobserved structures of the tree. A tree skeleton compactly describes the topological structure and contains useful information such as branch geometry, positions and hierarchy. This can be critical to planning contact interactions for agricultural manipulation, yet is difficult to gain due to occlusion by leaves, fruits and other branches. Our method uses an instance segmentation network to detect visible trunk, branches, and twigs. Then, based on the observed tree structures, we build a custom 3D likelihood map in the form of an occupancy grid to hypothesize on the presence of occluded skeletons through a series of minimum cost path searches. We show that our method outperforms baseline methods in highly occluded scenes, demonstrated through a set of experiments on a synthetic tree dataset. Qualitative results are also presented on a real tree dataset collected from the field.

IROS Conference 2021 Conference Paper

A Robust Illumination-Invariant Camera System for Agricultural Applications

• Abhisesh Silwal
• Tanvir Parhar
• Francisco Yandún
• Harjatin Singh Baweja
• George Kantor

Object detection and semantic segmentation are two of the most widely adopted deep learning algorithms in agricultural applications. One of the major sources of variability in image quality acquired outdoors for such tasks is changing lighting conditions that can alter the appearance of the objects or the contents of the entire image. While transfer learning and data augmentation reduce the need for large amount of data to train deep neural networks to some extent, the large variety of cultivars and the lack of shared datasets in agriculture makes wide-scale field deployments difficult. In this paper, we present an active lighting-based camera system that generates robust and uniform images in any lighting conditions. We provide extensive validation experiments to evaluate the consistency in the quality of the images. Metrics for assessing image uniformity like Structural Similarity (SSIM) index and the Peak Signal to Noise Ratio (PSNR) ranged from 83. 78 to 93. 79 and from 25. 30 to 31. 78 respectively, showing stability over a day with changing sunlight. The validation stage also showed that the generated images effectively reduce the amount of training samples for object detection using deep neural networks. The camera system was then deployed in real field experiments for counting buds in dormant vines and shoots in early season grape vines as well as for counting apples in orchards. The mean absolute errors obtained when compared to ground truth were 5%, 2. 55% and 8. 57%, respectively.

ICRA Conference 2021 Conference Paper

Reaching Pruning Locations in a Vine Using a Deep Reinforcement Learning Policy

• Francisco Yandún
• Tanvir Parhar
• Abhisesh Silwal
• David Clifford
• Zhiqiang Yuan
• Gabriella Levine
• Sergey Yaroshenko
• George Kantor

We outline a neural network-based pipeline for perception, control and planning of a 7 DoF robot for tasks that involve reaching into a dormant grapevine canopy. The proposed system consists of a 6 DoF industrial robot arm and a linear slider that can actuate on an entire grape vine. Our approach uses Convolutional Neural Networks to detect buds in dormant grape vines and a Reinforcement Learning based control strategy to reach desired cut-point locations for pruning tasks. Within this framework, three methodologies are developed and compared to reach the desired locations: the learned policy-based approach (RL), a hybrid method that uses the learned policy and an inverse kinematics solver (RL+IK), and lastly a classical approach commonly used in robotics. We first tested and validated the suitability of the proposed learning methodology in a simulated environment that resembled laboratory conditions. A reaching accuracy of up to 61. 90% and 85. 71% for the RL and RL+IK approaches respectively was obtained for a vine that the agent observed while learning. When testing in a new vine, the accuracy was up to 66. 66% and 76. 19% for RL and RL+IK, respectively. The same methods were then deployed on a real system in an end to end procedure: autonomously scan the vine using a vision system, create its model and finally use the learned policy to reach cutting points. The reaching accuracy obtained in these tests was 73. 08%.

AAMAS Conference 2019 Conference Paper

Active Learning with Gaussian Processes for High Throughput Phenotyping

• Sumit Kumar
• Wenhao Luo
• George Kantor
• Katia Sycara

A looming question that must be solved before robotic plant phenotyping capabilities can have significant impact to crop improvement programs is scalability. High Throughput Phenotyping (HTP) uses robotic technologies to analyze crops in order to determine species with favorable traits, however, the current practices rely on exhaustive coverage and data collection from the entire crop field being monitored under the breeding experiment. This works well in relatively small agricultural fields but can not be scaled to the larger ones, thus limiting the progress of genetics research. In this work, we propose an active learning algorithm to enable an autonomous system to collect the most informative samples in order to accurately learn the distribution of phenotypes in the field with the help of a Gaussian Process model. We demonstrate the superior performance of our proposed algorithm compared to the current practices on sorghum phenotype data collection.

NeurIPS Conference 2019 Conference Paper

Adaptive Auxiliary Task Weighting for Reinforcement Learning

• Xingyu Lin
• Harjatin Baweja
• George Kantor
• David Held

Reinforcement learning is known to be sample inefficient, preventing its application to many real-world problems, especially with high dimensional observations like images. Transferring knowledge from other auxiliary tasks is a powerful tool for improving the learning efficiency. However, the usage of auxiliary tasks has been limited so far due to the difficulty in selecting and combining different auxiliary tasks. In this work, we propose a principled online learning algorithm that dynamically combines different auxiliary tasks to speed up training for reinforcement learning. Our method is based on the idea that auxiliary tasks should provide gradient directions that, in the long term, help to decrease the loss of the main task. We show in various environments that our algorithm can effectively combine a variety of different auxiliary tasks and achieves significant speedup compared to previous heuristic approches of adapting auxiliary task weights.

AAMAS Conference 2019 Conference Paper

Distributed Environmental Modeling and Adaptive Sampling for Multi-Robot Sensor Coverage

• Wenhao Luo
• Changjoo Nam
• George Kantor
• Katia Sycara

We consider the problem of online distributed environmental modeling and adaptive sampling for multi-robot sensor coverage, where a team of robots spread out over the workspace in order to optimize the sensing performance over environmental phenomena, whose distribution is often referred to as a density function. Unlike most existing works that either assume certain knowledge of the density function beforehand or centrally learn the density function assuming global knowledge of collected data from all the robots, we propose a fully distributed adaptive sampling approach to allow robots to efficiently learn the unknown density function online. In particular, we developed adaptive coverage controllers based on the learned density functions for minimizing the sensing cost. To capture significantly different components of the environmental phenomenon with only locally collected data for each robot when global knowledge is not available, we propose a distributed mixture of Gaussian Processes algorithm that enables robots to collaboratively learn the global density function by exchanging only modelrelated parameters. We empirically demonstrate the effectiveness of our algorithm via evaluation on real-world data gathered from agricultural field robot and indoor static sensors.

IROS Conference 2019 Conference Paper

Stereo Visual Inertial LiDAR Simultaneous Localization and Mapping

• Weizhao Shao
• Srinivasan Vijayarangan
• Cong Li
• George Kantor

Simultaneous Localization and Mapping (SLAM) is a fundamental task to mobile and aerial robotics. LiDAR based systems have proven to be superior compared to vision based systems due to its accuracy and robustness. In spite of its superiority, pure LiDAR based systems fail in certain degenerate cases like traveling through a tunnel. We propose Stereo Visual Inertial LiDAR (VIL) SLAM that performs better on these degenerate cases and has comparable performance on all other cases. VIL-SLAM accomplishes this by incorporating tightly-coupled stereo visual inertial odometry (VIO) with LiDAR mapping and LiDAR enhanced visual loop closure. The system generates loop-closure corrected 6-DOF LiDAR poses in real-time and lcm voxel dense maps near real-time. VIL-SLAM demonstrates improved accuracy and robustness compared to state-of-the-art LiDAR methods.

ICRA Conference 2018 Conference Paper

A Deep Learning-Based Stalk Grasping Pipeline

• Tanvir Parhar
• Harjatin Singh Baweja
• Merritt Jenkins
• George Kantor

The need for fast and precise measurements of plant attributes makes robotic solutions an ideal replacement for labor-intensive phenotyping processes. In this work we present a deep learning-based high throughput, online pipeline for in-situ sorghum stalk detection and grasping. We use a variation of Generative Adversarial Network (GAN) for stalk segmentation trained on a relatively small number of images followed by a grasp point generation pipeline. The presented pipeline is robust to field challenges such as occlusions, high stalk density and lighting variation, and was deployed on a custom-built ground robot. We tested our end-to-end system in a field of Sorghum bicolor in South Carolina, USA, achieving an average grasping accuracy of 74. 13% and a stalk detection F1 score of 0. 90. Grasp point detection for plant manipulation takes an average of 0. 98 seconds, and pixel-wise stalk detection takes 0. 2 seconds per image.

IROS Conference 2018 Conference Paper

Compensating for Context by Learning Local Models of Perception Performance

• Humphrey Hu
• George Kantor

Perception system performance can vary dramatically with contextual factors such as environmental geometry, appearance, and other phenomena. In this work we present a theoretical framework for understanding the role of context in perception and discuss three approaches for predicting probabilistic performance from observations by efficiently learning local performance models. We compare these approaches with experiments on the monocular and stereo visual odometry systems for a ground robot, and show that they can effectively predict system failures in a wide variety of environments.

RLDM Conference 2017 Conference Abstract

Multi-modal Deep Reinforcement Learning with a Novel Sensor-based Dropout

• Guan-Horng Liu
• Avinash Siravuru
• Sai Prab-
• Manuela Veloso
• George Kantor

Sensor fusion is a key driver in the success of autonomous driving, given how instrumental it is to improve accuracy and robustness in the vehicle’s algorithmic decision making. However, in the space of end-to-end sensorimotor control, this multi-modal outlook has not received much attention. In the interest of enhancing safety and accuracy in control, a multi-modal approach to end-to-end autonomous navigation is need of the hour. Here, we introduce Multi-modal Deep Reinforcement Learning, and demonstrate how the use of multiple sensors improves the reward for an agent. For this purpose, we augment using both DDPG and NAF algorithms to admit multiple sensor input. The efficacy of a multi-modal policy is shown through extensive simulations experiments in TORCS, a popular open-source racing car game. Additionally, we introduce a new stochastic regularization technique, called Sensor Dropout to reduces the network’s sensitivity to any one sensor. Suitable metrics have been devised to study this behavior and highlight its applicability to other domains that operate in multi-modal settings.

IROS Conference 2017 Conference Paper

Online detection of occluded plant stalks for manipulation

• Merritt Jenkins
• George Kantor

This paper describes an algorithm for visually detecting crop stalks in-situ. Field-based crop stalk detection is a challenging computer vision problem due to occlusion by leaves, similarity of color and texture between stalks and surrounding foliage, and high stalk density within rows. Detecting stalks for grasping adds further challenges such as computation time and computing hardware. The proposed algorithm utilizes multiple stereo images taken from an unmanned ground vehicle and exploits geometric features specific to stalks such as vertical continuity, height, and the direction of surface normals. The described hardware and software pipeline is capable of detecting stalks for a manipulator grasp in approximately 12 seconds, and preliminary results show that hardware improvements can reduce detection time to 4 seconds. The algorithm is tested on 378 point clouds generated from 916 images of Sorghum bicolor, a grain crop. Performance is evaluated according to two methods, demonstrating precision greater than 93%.

ICRA Conference 2017 Conference Paper

The Robotanist: A ground-based agricultural robot for high-throughput crop phenotyping

• Tim Mueller-Sim
• Merritt Jenkins
• Justin Abel
• George Kantor

The established processes for measuring physiological and morphological traits (phenotypes) of crops in outdoor test plots are labor intensive and error-prone. Low-cost, reliable, field-based robotic phenotyping will enable geneticists to more easily map genotypes to phenotypes, which in turn will improve crop yields. In this paper, we present a novel robotic ground-based platform capable of autonomously navigating below the canopy of row crops such as sorghum or corn. The robot is also capable of deploying a manipulator to measure plant stalk strength and gathering phenotypic data with a modular array of non-contact sensors. We present data obtained from deployments to Sorghum bicolor test plots at various sites in South Carolina, USA.

ICRA Conference 2016 Conference Paper

Instance selection for efficient and reliable camera calibration

• Humphrey Hu
• George Kantor

The popularity of cameras for perception is enabled in part by powerful intrinsic calibration routines, commonly requiring a user to manually collect images of a known calibration target. The manual nature of this process produces training data that is unevenly spread in the camera relative pose space. If we desire camera parameters that perform well on average over the entire relative pose space, training on such a dataset results in poor performance. To address this, we show that reasoning about the training data distribution to select a more uniformly-spread subset of images produces more accurate and stable calibrations with fewer images. Our approach can be used easily with most camera calibration algorithms. We demonstrate in large-scale physical experiments the effect of non-uniform training data and show that our approach outperforms baselines in reprojection error and parameter variance.

ICRA Conference 2015 Conference Paper

Mobile manufacturing of large structures

• David A. Bourne
• Howie Choset
• Humphrey Hu
• George Kantor
• Chris Niessl
• Zachary B. Rubinstein
• Reid G. Simmons
• Stephen F. Smith

Assembly of large structures requires large fixtures, often referred to as monuments. Their cost and massive size limit flexibility and scalability of the manufacturing process. Numerous small mobile robots can replace these large structures and, therefore, replicate the efficiency of the assembly line with far more flexibility. An assembly line made up of mobile manipulators can easily and rapidly be reconfigured to support scalability and a varied product mix, while allowing for near optimal resource assignment. The challenge to using small robots in place of monuments is making their joint behavior precise enough to accomplish the task and efficient enough to execute subtasks in a reasonable period of time. In this paper, we describe a set of techniques that we combine to achieve the necessary precision and overall efficiency to build a large structure. We describe and demonstrate these techniques in the context of a testbed we implemented for assembling a wing ladder.

ICRA Conference 2015 Conference Paper

Operation of the ballbot on slopes and with center-of-mass offsets

• Bhaskar Vaidya
• Michael Shomin
• Ralph L. Hollis
• George Kantor

The ballbot is a human sized, dynamically stable mobile robot that balances on a single, spherical wheel. The current framework for navigation and control makes the assumption that the robot is operating on a level surface without any center-of-mass offset; however, in practice, such a dynamically stable robot has to be able to successfully navigate sloped surfaces and with such offsets. This work develops the equations of motion for the ballbot system on a sloped surface with a center-of-mass offset. The equilibria of this system are analyzed, and a compensation strategy is formulated that allows the ballbot to operate in the presence of slopes and center-of-mass offsets. This work also develops estimation algorithms for slope and center-of-mass offset angles during station-keeping and trajectory following. Results for compensation and estimation are demonstrated experimentally.

IROS Conference 2015 Conference Paper

Parametric covariance prediction for heteroscedastic noise

• Humphrey Hu
• George Kantor

The ubiquitous additive Gaussian noise model is favored in statistical modeling applications for its flexibility and ease of use. Often noise is assumed to be well-represented by a constant covariance, while in reality error characteristics may change predictably. We present an efficient parametric covariance predictor based on the modified Cholesky decomposition that maps from features of the input to covariance matrices. In addition, we discuss fitting the predictor parameters using noise samples with simple regularization techniques. We demonstrate our approach by estimating observation covariances for range-bearing localization with simulated and experimental datasets and show that this results in increased filtering performance compared to traditional covariance adaptation and constant covariance baselines.

ICRA Conference 2013 Conference Paper

Monocular feature-based periodic motion estimation for surgical guidance

• Stephen Tully
• George Kantor
• Howie Choset

In this paper, we present a novel approach for mapping periodically moving visual features with a monocular camera. Our target application is the estimation of moving surfaces during minimally invasive surgery for the purpose of aiding in the guidance of surgical tools. Our approach uses a bank of Kalman filters to estimate FFT parameters that encode the periodic motion of visually detected features. To ensure convergent estimation for this highly nonlinear problem, we have developed an iterative update procedure that treats the Kalman filter measurement update step as an optimization problem. Unlike existing solutions that rely on stereo vision, our approach estimates periodic motion with a single moving camera. With an experiment involving a beating heart phantom, we have shown that our approach is able to successfully estimate the periodic motion of visual features.

ICRA Conference 2012 Conference Paper

Constrained filtering with contact detection data for the localization and registration of continuum robots in flexible environments

• Stephen Tully
• Andrea Bajo
• George Kantor
• Howie Choset
• Nabil Simaan

This paper presents a novel filtering technique that uses contact detection data and environmental stiffness estimates to register and localize a robot with respect to an a priori 3D surface model. The algorithm leverages geometric constraints within a Kalman filter framework and relies on two distinct update procedures: 1) an equality constrained step for when the robot is forcefully contacting the environment, and 2) an inequality constrained step for when the robot lies in the free-space of the environment. This filtering procedure registers the robot by incrementally eliminating probabilistically infeasible state space regions until a high likelihood solution emerges. In addition to registration and localization, the algorithm can estimate the deformation of the surface model and can detect false positives with respect to contact estimation. This method is experimentally evaluated with an experiment involving a continuum robot interacting with a bench-top flexible structure. The presented algorithm produces an experimental error in registration (with respect to the end-effector position) of 1. 1 mm, which is less than 0. 8 percent of the robot length.

ICRA Conference 2012 Conference Paper

Integrated planning and control for graceful navigation of shape-accelerated underactuated balancing mobile robots

• Umashankar Nagarajan
• George Kantor
• Ralph L. Hollis

This paper presents controllers called motion policies that achieve fast, graceful motions in small, collision-free domains of the position space for balancing mobile robots like the ballbot. The motion policies are designed such that their valid compositions will produce overall graceful motions. An automatic instantiation procedure deploys motion policies on a 2D map of the environment to form a library and the validity of their composition is given by a gracefully prepares graph. Dijsktra's algorithm is used to plan in the space of these motion policies to achieve the desired navigation task. A hybrid controller is used to switch between the motion policies. The results of successful experimental testing of two navigation tasks, namely, point-point and surveillance motions on the ballbot platform are presented.

IROS Conference 2012 Conference Paper

Monocular visual navigation of an autonomous vehicle in natural scene corridor-like environments

• Ji Zhang 0003
• George Kantor
• Marcel Bergerman
• Sanjiv Singh

We present a monocular visual navigation methodology for autonomous orchard vehicles. Modern orchards are usually planted with straight and parallel tree rows that form a corridor-like environment. Our task consists of driving a vehicle autonomously along the tree rows. The original contributions of this paper are: 1) a method to recover vehicle rotation independently of translation by modeling the vehicle as a car-like robot driving on a 3D ground surface-the rotation is estimated from monocular images while the translation is measured by a wheel encoder; and 2) a method to fit the 3D points corresponding to the trees into straight lines via an optimization algorithm that minimizes the error variance on the robot lookahead point. Additionally, we use a simple vanishing point detection approach to find the ends of the tree rows. The vanishing point detection is integrated into the system via an extended Kalman filter. The methodology's robustness to environmental changes is validated in more than fifty experiments in research and commercial orchards, six of which are presented and discussed in detail.

ICRA Conference 2012 Conference Paper

Multi-agent deterministic graph mapping via robot rendezvous

• Chaohui Gong
• Stephen Tully
• George Kantor
• Howie Choset

In this paper, we present a novel algorithm for deterministically mapping an undirected graph-like world with multiple synchronized agents. The application of this algorithm is the collective mapping of an indoor environment with multiple mobile robots while leveraging an embedded topological decomposition of the environment. Our algorithm relies on a group of agents that all depart from the same initial vertex in the graph and spread out to explore the graph. A centralized tree of graph hypotheses is maintained to consider loop-closure, which is deterministically verified when agents observe each other at a common vertex. To achieve efficient mapping, we introduce an active exploration method in which agents dynamically request rendezvous tasks from other available agents to validate graph hypotheses.

AAMAS Conference 2012 Conference Paper

Real-World Testing of a Multi-Robot Team

• Paul Scerri
• Prasanna Velagapudi
• Balajee Kannan
• Abhinav Valada
• Christopher Tomaszewski
• John Dolan
• Adrian Scerri
• Kumar Shaurya Shankar

ICRA Conference 2011 Conference Paper

Deployment of a point and line feature localization system for an outdoor agriculture vehicle

• Jacqueline Libby
• George Kantor

This paper presents a perception-based GPS free approach for localizing a mobile robot in an orchard environment. An extended Kalman filter (EKF) algorithm is presented that uses a wheel odometry prediction step and laser rangefinder update steps. There are two update steps, one that uses measurements to reflective point features and one that uses measurements to linear features formed by tree rows. The features are associated to landmarks in previously surveyed maps. The practical issues of dealing with uncertainty both from the environment and the on-board sensors are discussed and accounted for. The resulting algorithm is demonstrated in over 20km of online operation in a variety of real orchard environments.

ICRA Conference 2011 Conference Paper

Incremental construction of the saturated-GVG for multi-hypothesis topological SLAM

• Tong Tao
• Stephen Tully
• George Kantor
• Howie Choset

The generalized Voronoi graph (GVG) is a topological representation of an environment that can be incrementally constructed with a mobile robot using sensor-based control. However, because of sensor range limitations, the GVG control law will fail when the robot moves into a large open area. This paper discusses an extended GVG approach to topological navigation and mapping: the saturated generalized Voronoi graph (S-GVG), for which the robot employs an additional wall-following behavior to navigate along obstacles at the range limit of the sensor. In this paper, we build upon previous work related to the S-GVG and provide two important contributions: 1) a rigorous discussion of the control laws and algorithm modifications that are necessary for incremental construction of the S-GVG with a mobile robot, and 2) a method for incorporating the S-GVG into a novel multi-hypothesis SLAM algorithm for loop-closing and localization. Experiments with a wheeled mobile robot in an office-like environment validate the effectiveness of the proposed approach.

IROS Conference 2011 Conference Paper

Inequality constrained Kalman filtering for the localization and registration of a surgical robot

• Stephen Tully
• George Kantor
• Howie Choset

We present a novel method for enforcing nonlinear inequality constraints in the estimation of a high degree of freedom robotic system within a Kalman filter. Our constrained Kalman filtering technique is based on a new concept, which we call uncertainty projection, that projects the portion of the uncertainty ellipsoid that does not satisfy the constraint onto the constraint surface. A new PDF is then generated with an efficient update procedure that is guaranteed to reduce the uncertainty of the system. The application we have targeted for this work is the localization and automatic registration of a robotic surgical probe relative to preoperative images during image-guided surgery. We demonstrate the feasibility of our constrained filtering approach with data collected from an experiment involving a surgical robot navigating on the epicardial surface of a porcine heart.

IROS Conference 2011 Conference Paper

Monte Carlo Localization using 3D texture maps

• Yu Fu
• Stephen Tully
• George Kantor
• Howie Choset

This paper uses KLD-based (Kullback-Leibler Divergence) Monte Carlo Localization (MCL) to localize a mobile robot in an indoor environment represented by 3D texture maps. A 3D texture map is a simplified model that includes vertical planes with colored texture information associated with each vertical plane. At each time step, a distance measurement and an observed texture from an omnidirectional camera are compared to the expected distance measurement and the expected texture according to each hypothesis of the robot's pose in an MCL framework. Compared to previous implementations of MCL, our proposed approach converges faster than distance-only MCL and localizes the robot more precisely than SIFT-based MCL. We demonstrate this new MCL algorithm for robot localization with experiments in several hallways.

IROS Conference 2011 Conference Paper

Shape estimation for image-guided surgery with a highly articulated snake robot

• Stephen Tully
• George Kantor
• Marco A. Zenati
• Howie Choset

In this paper, we present a filtering method for estimating the shape and end effector pose of a highly articulated surgical snake robot. Our algorithm introduces new kinematic models that are used in the prediction step of an extended Kalman filter whose update step incorporates measurements from a 5-DOF electromagnetic tracking sensor situated at the distal end of the robot. A single tracking sensor is sufficient for estimating the shape of the system because the robot is inherently a follow-the-leader mechanism with well defined motion characteristics. We therefore show that, with appropriate steering motion, the state of the filter is fully observable. The goal of our shape estimation algorithm is to create a more accurate and representative 3D rendered visualization for image-guided surgery. We demonstrate the feasibility of our method with results from an animal experiment in which our shape and pose estimate was used as feedback in a control scheme that semi-autonomously drove the robot along the epicardial surface of a porcine heart.

AAAI Conference 2010 Conference Paper

A Single-Step Maximum A Posteriori Update for Bearing-Only SLAM

• Stephen Tully
• George Kantor
• Howie Choset

This paper presents a novel recursive maximum a posteriori update for the Kalman formulation of undelayed bearing-only SLAM. The estimation update step is cast as an optimization problem for which we can prove the global minimum is reachable via a bidirectional search using Gauss-Newton’s method along a one-dimensional manifold. While the filter is designed for mapping just one landmark, it is easily extended to full-scale multiple-landmark SLAM. We provide this extension via a formulation of bearing-only FastSLAM. With experiments, we demonstrate accurate and convergent estimation in situations where an EKF solution would diverge.

IROS Conference 2009 Conference Paper

A multi-hypothesis topological SLAM approach for loop closing on edge-ordered graphs

• Stephen Tully
• George Kantor
• Howie Choset
• Felix Werner

We present a method for topological SLAM that specifically targets loop closing for edge-ordered graphs. Instead of using a heuristic approach to accept or reject loop closing, we propose a probabilistically grounded multi-hypothesis technique that relies on the incremental construction of a map/state hypothesis tree. Loop closing is introduced automatically within the tree expansion, and likely hypotheses are chosen based on their posterior probability after a sequence of sensor measurements. Careful pruning of the hypothesis tree keeps the growing number of hypotheses under control and a recursive formulation reduces storage and computational costs. Experiments are used to validate the approach.

ICRA Conference 2009 Conference Paper

State transition, balancing, station keeping, and yaw control for a dynamically stable single spherical wheel mobile robot

• Umashankar Nagarajan
• Anish Mampetta
• George Kantor
• Ralph L. Hollis

Unlike statically stable wheeled mobile robots, dynamically stable mobile robots can have higher centers of gravity, smaller bases of support and can be tall and thin resembling the shape of an adult human. This paper concerns the ballbot mobile robot, which balances dynamically on a single spherical wheel. The ballbot is omni-directional and can also rotate about its vertical axis (yaw motion). It uses a triad of legs to remain statically stable when powered off. This paper presents the evolved design with a four-motor inverse mouse-ball drive, yaw drive, leg drive, control system, and results including dynamic balancing, station keeping, yaw motion while balancing, and automatic transition between statically stable and dynamically stable states.

IROS Conference 2009 Conference Paper

Topological SLAM using neighbourhood information of places

• Felix Werner
• Frédéric Maire
• Joaquin Sitte
• Howie Choset
• Stephen Tully
• George Kantor

Perceptual aliasing makes topological navigation a difficult task. In this paper we present a general approach for topological SLAM (simultaneous localisation and mapping) which does not require motion or odometry information but only a sequence of noisy measurements from visited places. We propose a particle filtering technique for topological SLAM which relies on a method for disambiguating places which appear indistinguishable using neighbourhood information extracted from the sequence of observations. The algorithm aims to induce a small topological map which is consistent with the observations and simultaneously estimate the location of the robot. The proposed approach is evaluated using a data set of sonar measurements from an indoor environment which contains several similar places. It is demonstrated that our approach is capable of dealing with severe ambiguities and, and that it infers a small map in terms of vertices which is consistent with the sequence of observations.

ICRA Conference 2009 Conference Paper

Trajectory planning and control of an underactuated dynamically stable single spherical wheeled mobile robot

• Umashankar Nagarajan
• George Kantor
• Ralph L. Hollis

The ballbot is a dynamically stable mobile robot that moves on a single spherical wheel and is capable of omnidirectional movement. The ballbot is an underactuated system with nonholonomic dynamic constraints. The authors propose an offline trajectory planning algorithm that provides a class of parametric trajectories to the unactuated joint in order to reach desired static configurations of the system with regard to the dynamic constraint. The parameters of the trajectories are obtained using optimization techniques. A feedback controller is proposed that ensures accurate trajectory tracking. The trajectory planning algorithm and tracking controller are validated experimentally. The authors also extend the offline trajectory planning algorithm to a generalized case of motion between non-static configurations.

ICRA Conference 2008 Conference Paper

Iterated filters for bearing-only SLAM

• Stephen Tully
• Hyungpil Moon
• George Kantor
• Howie Choset

This paper discusses the importance of iteration when performing the measurement update step for the problem of bearing-only SLAM. We focus on an undelayed approach that initializes a landmark after only one bearing measurement. Traditionally, the extended Kalman filter (EKF) has been used for SLAM, but the EKF measurement update rule can often lead to a divergent state estimate due to its inconsistency in linearization. We discuss the flaws of the EKF in this paper, and show that even the well established inverse-depth parametrization for bearing-only SLAM can be affected. We then show that representing the bearing-only update as a numerical optimization problem (solved with an iterative approach such as Gauss-Newton minimization) prevents divergence of the Kalman filter state and produces accurate SLAM results for a bearing-only sensor. More specifically, we propose the use of an iterated Kalman filter to resolve the issues normally associated with the EKF measurement update. Two outdoor mobile robot experiments are discussed to compare algorithm performance.

IROS Conference 2007 Conference Paper

Hybrid localization using the hierarchical atlas

• Stephen Tully
• Hyungpil Moon
• Deryck Morales
• George Kantor
• Howie Choset

This paper presents a hybrid localization scheme for a mobile robot using the hierarchical atlas. The hierarchical atlas is a map that consists of a higher level topological graph with lower level feature-based metric submaps associated with the graph edges. Our method employs both a discrete Bayes filter and a Kalman filter to localize the robot in the map. This framework accommodates localization in a map with no prior information (global localization) and localization in a map with an incorrect pose estimate (kidnapped robot). Our approach efficiently scales to large environments without sacrificing accuracy or robustness. We have verified our method with large-scale experiments in a multi-floor office environment.

ICRA Conference 2006 Conference Paper

A Dynamically Stable Single-wheeled Mobile Robot with Inverse Mouse-ball Drive

• Tom Lauwers
• George Kantor
• Ralph L. Hollis

Multi-wheel statically-stable mobile robots tall enough to interact meaningfully with people must have low centers of gravity, wide bases of support, and low accelerations to avoid tipping over. These conditions present a number of performance limitations. Accordingly, we are developing an inverse of this type of mobile robot that is the height, width, and weight of a person, having a high center of gravity, that balances dynamically on a single spherical wheel. Unlike balancing 2-wheel platforms which must turn before driving in some direction, the single-wheel robot can move directly in any direction. We present the overall design, actuator mechanism based on an inverse mouse-ball drive, control system, and initial results including dynamic balancing, station keeping, and point-to-point motion

ICRA Conference 2006 Conference Paper

Range-only SLAM for Robots Operating Cooperatively with Sensor Networks

• Joseph Djugash
• Sanjiv Singh
• George Kantor
• Wei Zhang 0023

A mobile robot we have developed is equipped with sensors to measure range to landmarks and can simultaneously localize itself as well as locate the landmarks. This modality is useful in those cases where environmental conditions preclude measurement of bearing (typically done optically) to landmarks. Here we extend the paradigm to consider the case where the landmarks (nodes of a sensor network) are able to measure range to each other. We show how the two capabilities are complimentary in being able to achieve a map of the landmarks and to provide localization for the moving robot. We present recent results with experiments on a robot operating in a randomly arranged network of nodes that can communicate via radio and range to each other using sonar. We find that incorporation of inter-node measurements helps reduce drift in positioning as well as leads to faster convergence of the map of the nodes. We find that addition of a mobile node makes the SLAM feasible in a sparsely connected network of nodes

ICRA Conference 2006 Conference Paper

Towards Particle Filter SLAM with Three Dimensional Evidence Grids in a Flooded Subterranean Environment

• Nathaniel Fairfield
• George Kantor
• David Wettergreen

This paper describes the application of a RaoBlackwellized Particle Filter to the problem of simultaneous localization and mapping onboard a hovering autonomous underwater vehicle. This vehicle, called DEPTHX, equipped with a large array of pencil-beam sonars for mapping, and autonomously explore a system of flooded tunnels associated with the Zacaton sinkhole in Tamaulipas, Mexico. Due to the three-dimensional nature of the tunnels, we describe an extension of traditional two dimensional evidence grids to three dimensions. In May 2005, we collected a sonar data set in Zacaton. We present successful SLAM results using both the real-world data and simulated data

ICRA Conference 2004 Conference Paper

Bearing-only Landmark Initialization with Unknown Data Association

• Albert Costa
• George Kantor
• Howie Choset

It is essential in many applications that mobile robots localize themselves with respect to an unknown environment. This means that the robot must build a map of its environment and then localize using the map. This process is called simultaneous localization and mapping (SLAM). This paper presents an iterative solution to the landmark initialization problem inherent in a bearing-only implementation of SLAM. No prior knowledge of the environment is required, and furthermore, there are no requirements about having the data association problem solved. Once landmarks are initialized, they are inserted into an extended Kalman Filter (EKF) to solve the SLAM problem. Both indoor and outdoor experiments are presented to validate the method.

IROS Conference 2003 Conference Paper

Experimental results in range-only localization with radio

• Derek Kurth
• George Kantor
• Sanjiv Singh

We present an early experimental result toward solving the localization problem with range-only sensors. We perform an experiment in which a mobile robot localizes using dead reckoning and range measurements to stationary radio-frequency beacons in its environment, incorporating the range measurements into the position estimate using a Kalman filter. This data set involves over 20, 000 range readings to surveyed beacons while a robot moved continuously over a path for nearly 1 hour. Careful groundtruth accurate to a few centimeters was recorded during this motion. We show the improvement of the robot's position estimate over dead reckoning even when the range readings are very noisy. We extend this approach to the problem of simultaneous localization and mapping (SLAM), localizing both the robot and tag positions from noisy initial estimates.

IROS Conference 2003 Conference Paper

Hierarchical simultaneous localization and mapping

• Brad Lisien
• Deryck Morales
• David Silver 0002
• George Kantor
• Ioannis M. Rekleitis
• Howie Choset

This paper presents a novel method of combining topological and feature-based mapping strategies to create a hierarchical approach to simultaneous localization and mapping (SLAM). More than simply running both processes in parallel, we use the topological mapping procedure to organize local feature-based methods. The result is an autonomous exploration and mapping strategy that scales well to large environments and higher dimensions while confronting the issue of obstacle avoidance. We have obtained successful results of our approach in an area spanning 5000 square meters.

IROS Conference 2003 Conference Paper

Inertial navigation and visual line following for a dynamical hexapod robot

• Sarjoun Skaff
• George Kantor
• David Maiwand
• Alfred A. Rizzi

This paper presents preliminary development of autonomous sensor-guided behaviors for a six-legged dynamical robot (RHex). The behaviors represent the exteroceptive closed-loop locomotion strategies for this system. Simple motion models for RHex are described and used to deploy controllers for inertial straight line locomotion and visual line following. Results are experimentally validated on the robot.

ICRA Conference 2002 Conference Paper

Preliminary Results in Range-Only Localization and Mapping

• George Kantor
• Sanjiv Singh

This paper presents methods of localization using cooperating landmarks (beacons) that provide the ability to measure range only. Recent advances in radio frequency technology make it possible to measure range between inexpensive beacons and a transponder Such a method has tremendous benefit since line of sight is not required between the beacons and the transponder and because the data association problem can be completely avoided. If the positions of the beacons are known, measurements from multiple beacons can be combined using probability grids to provide an accurate estimate of robot location. This estimate can be improved by using Monte Carlo techniques and Kalman filters to incorporate odometry data. Similar methods can be used to solve the simultaneous localization and mapping problem (SLAM) when beacon locations are uncertain. Experimental results are presented for robot localization. Tracking and SLAM algorithms are demonstrated in simulation.