James J. Little

TMLR Journal 2026 Journal Article

Segmentation From Attention: Training-Free Layer Selection and One-Shot Tuning for Segmentation in VLMs

• Mir Rayat Imtiaz Hossain
• Mennatullah Siam
• Leonid Sigal
• James J. Little

Large-scale vision-language models (VLMs), trained on extensive datasets of image-text pairs, exhibit strong multimodal understanding capabilities by implicitly learning associations between textual descriptions and image regions. This emergent ability enables zero-shot object detection and segmentation, using techniques that rely on text-image attention maps, without necessarily training on abundant labeled segmentation datasets. However, performance of such methods depends heavily on prompt engineering and manually selected layers or head choices for the attention layers. In this work, we propose a training-free entropy-based measure, InfoScore, to identify the best image-text attention layers for segmentation, providing a more flexible and scalable solution for training-free open-vocabulary segmentation, reducing the additional burden of hyperparamter search. We empirically show that our training-free selection strategy is superior to naive selection strategies. Additionally, we demonstrate that instead of solely relying on text prompts, fine-tuning the image-text attention layer with a single visual example of each class significantly improves segmentation without the need of additional parameters or decoders. Moreover, we show that our methods and findings are general and can be applied across various vision-language models (VLMs).

IROS Conference 2018 Conference Paper

Exploiting Points and Lines in Regression Forests for RGB-D Camera Relocalization

• Lili Meng
• Frederick Tung
• James J. Little
• Julien Valentin
• Clarence W. de Silva

Camera relocalization plays a vital role in many robotics and computer vision applications, such as self-driving cars and virtual reality. Recent random forests based methods exploit randomly sampled pixel comparison features to predict 3D world locations for 2D image locations to guide the camera pose optimization. However, these point features are only sampled randomly in images, without considering geometric information such as lines, leading to large errors with the existence of poorly textured areas or in motion blur. Line segments are more robust in these environments. In this work, we propose to jointly exploit points and lines within the framework of uncertainty driven regression forests. The proposed approach is thoroughly evaluated on three publicly available datasets against several strong state-of-the-art baselines in terms of several different error metrics. Experimental results prove the efficacy of our method, showing superior or on-par state-of-the-art performance.

IROS Conference 2017 Conference Paper

Backtracking regression forests for accurate camera relocalization

• Lili Meng
• Jianhui Chen
• Frederick Tung
• James J. Little
• Julien Valentin
• Clarence W. de Silva

Camera relocalization plays a vital role in many robotics and computer vision tasks, such as global localization, recovery from tracking failure, and loop closure detection. Recent random forests based methods directly predict 3D world locations for 2D image locations to guide the camera pose optimization. During training, each tree greedily splits the samples to minimize the spatial variance. However, these greedy splits often produce uneven sub-trees in training or incorrect 2D-3D correspondences in testing. To address these problems, we propose a sample-balanced objective to encourage equal numbers of samples in the left and right sub-trees, and a novel backtracking scheme to remedy the incorrect 2D-3D correspondence predictions. Furthermore, we extend the regression forests based methods to use local features in both training and testing stages for outdoor RGB-only applications. Experimental results on publicly available indoor and outdoor datasets demonstrate the efficacy of our approach, which shows superior or on-par accuracy with several state-of-the-art methods.

ICRA Conference 2017 Conference Paper

MF3D: Model-free 3D semantic scene parsing

• Frederick Tung
• James J. Little

We present a novel model-free method for online 3D semantic scene parsing from video sequences. MF3D (Model-Free 3D) is different from conventional methods for 3D scene parsing in that voxel labelling is approached via search-based label transfer instead of discriminative classification. This non-parametric approach makes MF3D easy to scale with an online growth in the database, as no model re-training is required with the addition of new examples or categories. Experimental results on the KITTI benchmark demonstrate that our model-free approach enables accurate online 3D scene parsing while retaining extensibility to new categories. In addition, we show that unsupervised binary encoding (hashing) techniques can be easily incorporated into our framework for scalability to larger databases.

ICRA Conference 2014 Conference Paper

Ensuring safety in human-robot dialog - A cost-directed approach

• Junaed Sattar
• James J. Little

We present an approach for detecting potentially unsafe commands in human-robot dialog, where a robotic system evaluates task cost in input commands to ask input-specific, directed questions to ensure safe task execution. The goal is to reduce risk, both to the robot and the environment, by asking context-appropriate questions. Given an input program, (i. e. , a sequence of commands) the system evaluates a set of likely alternate programs along with their likelihood and cost, and these are given as input to a Decision Function to decide whether to execute the task or confirm the plan from the human partner. A process called token-risk grounding identifies the costly commands in the programs, and specifically asks the human user to clarify those commands. We evaluate our system in two simulated robot tasks, and also on-board the Willow Garage PR2 and TurtleBot robots in an indoor task setting. In both sets of evaluations, the results show that the system is able to identify specific commands that contribute to high task cost, and present users the option to either confirm or modify those commands. In addition to ensuring task safety, this results in an overall reduction in robot reprogramming time.

ICRA Conference 2013 Conference Paper

3D spatial relationships for improving object detection

• Tristram Southey
• James J. Little

This work demonstrates how 3D qualitative spatial relationships can be used to improve object detection by differentiating between true and false positive detections. Our method identifies the most likely subset of 3D detections using seven types of 3D relationships and adjusts detection confidence scores to improve the average precision. A model is learned using a structured support vector machine [1] from examples of 3D layouts of objects in offices and kitchens. We test our method on synthetic detections to determine how factors such as localization accuracy, number of detections and detection scores change the effectiveness of 3D spatial relationships for improving object detection rates. Finally, we describe a technique for generating 3D detections from 2D image-based object detections and demonstrate how our method improves the average precision of these 3D detections.

ICRA Conference 2013 Conference Paper

Modeling nonconvex workspace constraints from diverse demonstration sets for Constrained Manipulator Visual Servoing

• Ambrose Chan
• Elizabeth A. Croft
• James J. Little

This paper presents a novel framework for solving the Constrained Manipulator Visual Servoing (CMVS) problem. Classical eye-in-hand visual servoing relies on a reference image to capture the end-effector positioning task, but non-convex workspace constraints (such as whole-arm collision and camera occlusion constraints) are not represented. An explicit CAD model of the workspace is typically required for collision avoidance and visibility planning algorithms. In our novel CMVS framework, during the reference image capture process, we leverage the user's kinesthetic and visual capabilities to obtain a set of qualitatively-diverse demonstrations that provide information about the robot's work environment. We investigate methods for identifying the topology of the feasible regions represented directly in the control space of the robot (i. e. , image-space and joint-space). We use a combination of stochastic modeling and graphical methods to describe the feasible space, capturing both the inter-group and intra-group variations. Specifically, our method uses the inter-groups variations to build a map that describes the global connectivity of the space, while exploiting the intra-group variations to automatically derive the appropriate gains in the control law. For a given target object, we apply online Gaussian Mixture Regression to the relevant feasible space regions to provide an idealized trajectory for tracking in image-space and in joint-space. We illustrate the key advantages of our approach through a set of visual servoing experiments on a Barrett WAM 7-DOF manipulator with a Sony XC-HR70 camera.

ICRA Conference 2013 Conference Paper

Overcoming unknown occlusions in eye-in-hand visual search

• Sina Radmard
• David Meger
• Elizabeth A. Croft
• James J. Little

We propose a method for handling persistent visual occlusions that disrupt visual tracking for eye-in-hand systems. Our approach allows a robot to “look behind” an occluder and re-acquire its target. To allow efficient planning, we avoid exhaustive mapping of the 3D occluder into configuration space, and instead use informed samples to strike a balance between target search and information gain. A particle filter continuously estimates the target location when it is not visible. Meanwhile, we build a simple but effective map of the occluder's extents to compute potential occlusion-clearing motions using very few calls to efficient approximations of inverse kinematics. Our mixed-initiative cost function balances the goal of directly locating the target with the goal of gaining information through mapping the occluder. Monte-Carlo optimization with efficient data-driven proposals allows us to approximate one-step solutions efficiently. Experimental evaluation performed on a realistic simulator shows that our method can quickly obtain clear views of the target, even when occlusions are persistent and significant camera motion is required.

IROS Conference 2011 Conference Paper

Constrained manipulator visual servoing (CMVS): Rapid robot programming in cluttered workspaces

• Ambrose Chan
• Elizabeth A. Croft
• James J. Little

This paper presents a model-free optimization framework for the visual servoing of eye-in-hand manipulators in cluttered environments. Visual feedback is used to solve for a set of feasible trajectories that bring the robot end-effector to a target object at a previously untaught location under a number of challenging constraints (i. e. , whole-arm collisions, object occlusions, robot's joint limits, camera's sensing limits). A novel controller is proposed, which exploits the natural by-products of the teach-by-showing process, to help the robot navigate this non-convex space. Examining the user-demonstrated trajectories that lead up to the reference image, we use a combination of stochastic optimization techniques and classical optimization techniques to extract the relevant cost functions and constraints for servoing. We hypothesize that we can leverage the user's sensory capabilities and knowledge of the workspace to alleviate the burden of modeling system constraints explicitly. We verify this hypothesis via realistic experiments on a Barrett WAM 7-DOF manipulator equipped with a Sony XC-HR70 camera to show the comparative efficacy of this approach.

IROS Conference 2011 Conference Paper

Mobile 3D object detection in clutter

• David Meger
• James J. Little

This paper presents a method for multi-view 3D robotic object recognition targeted for cluttered indoor scenes. We explicitly model occlusions that cause failures in visual detectors by learning a generative appearance-occlusion model from a training set containing annotated 3D objects, images and point clouds. A Bayesian 3D object likelihood incorporates visual information from many views as well as geometric priors for object size and position. An iterative, sampling-based inference technique determines object locations based on the model. We also contribute a novel robot-collected data set with images and point clouds from multiple views of 60 scenes, with over 600 manually annotated 3D objects accounting for over ten thousand bounding boxes. This data has been released to the community. Our results show that our system is able to robustly recognize objects in realistic scenes, significantly improving recognition performance in clutter.

ICRA Conference 2010 Conference Paper

Viewpoint detection models for sequential embodied object category recognition

• David Meger
• Ankur Gupta 0004
• James J. Little

This paper proposes a method for learning viewpoint detection models for object categories that facilitate sequential object category recognition and viewpoint planning. We have examined such models for several state-of-the-art object detection methods. Our learning procedure has been evaluated using an exhaustive multiview category database recently collected for multiview category recognition research. Our approach has been evaluated on a simulator that is based on real images that have previously been collected. Simulation results verify that our viewpoint planning approach requires fewer viewpoints for confident recognition. Finally, we illustrate the applicability of our method as a component of a completely autonomous visual recognition platform that has previously been demonstrated in an object category recognition competition.

IROS Conference 2009 Conference Paper

Planning collision-free and occlusion-free paths for industrial manipulators with eye-to-hand configuration

• Simon Léonard
• Elizabeth A. Croft
• James J. Little

This paper presents a motion planning algorithm for industrial manipulators with the simultaneous constraints of avoiding collisions and avoiding the occlusion of specified pixellated regions of an eye-to-hand camera. The system uses a probabilistic roadmap to satisfy the constraints imposed by the command interface of typical industrial manipulators and uses dynamic collision checking to ensure collision-free motion. In the context of a task monitored by a camera, we enhance a probabilistic roadmap with a dynamic occlusion checking algorithm that is able to determine which pixels of the camera are occluded by the robot during each motion segment. The occlusion algorithm is formulated as collision algorithm where the field of view of the camera is represented as a quadtree of frustums. The proposed algorithm is demonstrated in industrial bin picking simulations where the gripper must not occlude the targeted object throughout the task.

ICRA Conference 2008 Conference Paper

Dynamic visibility checking for vision-based motion planning

• Simon Léonard
• Elizabeth A. Croft
• James J. Little

An important problem in position-based visual servoing (PBVS) is to guarantee that a target will remain within the field of view for the duration of the task. In this paper, we propose a dynamic visibility checking algorithm that, given a parametrized trajectory of the camera, determines if an arbitrary 3D target will remain within the field of view. We reformulate this problem as the problem of determining if the 3D coordinates of the target collide with the frustum formed by the camera field of view during the camera trajectory. To solve this problem, our algorithm computes and compares the shortest distance between the target and the frustum with the length of the trajectory described by the target in the camera's coordinate frame. Furthermore, we demonstrate that our algorithm can be combined with path planning algorithms and, in particular, probabilistic roadmaps (PRM). Results suggest that our algorithm is computationally efficient even when the target moves in the vicinity of image borders. In simulations, we use our dynamic visibility checking algorithm in conjunction with a PRM to plan collision free paths while providing the guarantee that a specific target will not leave the field of view.

ICRA Conference 2008 Conference Paper

Informed visual search: Combining attention and object recognition

• Per-Erik Forssén
• David Meger
• Kevin Lai
• Scott Helmer
• James J. Little
• David G. Lowe

This paper studies the sequential object recognition problem faced by a mobile robot searching for specific objects within a cluttered environment. In contrast to current state-of-the-art object recognition solutions which are evaluated on databases of static images, the system described in this paper employs an active strategy based on identifying potential objects using an attention mechanism and planning to obtain images of these objects from numerous viewpoints. We demonstrate the use of a bag-of-features technique for ranking potential objects, and show that this measure outperforms geometric matching for invariance across viewpoints. Our system implements informed visual search by prioritising map locations and re-examining promising locations first. Experimental results demonstrate that our system is a highly competent object recognition system that is capable of locating numerous challenging objects amongst distractors.

IROS Conference 2008 Conference Paper

Occlusion-free path planning with a probabilistic roadmap

• Matthew A. Baumann
• Donna C. Dupuis
• Simon Léonard
• Elizabeth A. Croft
• James J. Little

We present a novel algorithm for path planning that avoids occlusions of a visual target for an ldquoeye-in-handrdquo sensor on an articulated robot arm. We compute paths using a probabilistic roadmap to avoid collisions between the robot and obstacles, while penalizing trajectories that do not maintain line-of-sight. The system determines the space from which line-of-sight is unimpeded to the target (the visible region). We assign penalties to trajectories within the roadmap proportional to the distance the camera travels while outside the visible region. Using Dijkstrapsilas algorithm, we compute paths of minimal occlusion (maximal visibility) through the roadmap. In our experiments, we compare a shortest-distance path to the minimal-occlusion path and discuss the impact of the improved visibility.

ICRA Conference 2008 Conference Paper

Trajectory specification via sparse waypoints for eye-in-hand robots requiring continuous target visibility

• Ambrose Chan
• Elizabeth A. Croft
• James J. Little

This paper presents several methods of managing field of view constraints of an eye-in-hand system for vision- based pose control with limited controller input. Herein, the possible inverse kinematic solutions for a desired relative camera pose are evaluated to determine whether the interpolated trajectories satisfy field of view constraints for the target of interest. If no immediately feasible trajectory exists, additional waypoints are specified to guide the robot towards its goal while maintaining visibility. The insertion of an additional visible and feasible waypoint divides the problem into two sub-problems of the same form, but of lesser difficulty by reducing the robot's interpolation distance. Virtual image-based visual servoing (IBVS) is used to generate an ideal image trajectory to guide the selection of waypoints. A damped least- squares inverse kinematics solution is implemented to handle robot singularities. The methods are simulated for a CRS-A465 robot with a Sony XC-HR70 camera.

IROS Conference 2007 Conference Paper

Decision theoretic task coordination for a visually-guided interactive mobile robot

• Pantelis Elinas
• James J. Little

In this paper, we present a visually-guided mobile robot that is capable of executing a task requiring complex human-robot interaction (HRI). The robot delivers verbal messages among the inhabitants of an office-like environment. Essential to the robot's robust performance is our behavior-based robot control architecture enhanced with a state of the art decision theoretic planner that takes into account the temporal characteristics of the robot's actions. The decision theoretic layer is based on the partially observable Markov decision process (POMDP) framework allowing us to achieve principled coordination of complex subtasks implemented as robot behaviors/skills. We compute approximate POMDP policies using the randomized point-based value iteration algorithm and we present heuristics for improving its computational efficiency.

IROS Conference 2006 Conference Paper

Autonomous vision-based exploration and mapping using hybrid maps and Rao-Blackwellised particle filters

• Robert Sim
• James J. Little

This paper addresses the problem of exploring and mapping an unknown environment using a robot equipped with a stereo vision sensor. The main contribution of our work is a fully automatic mapping system that operates without the use of active ranger sensors (such as laser or sonic transducers), can operate in real-time and can consistently produce accurate maps of large-scale environments. Our approach implements a Rao-Blackwellised particle filter (RBPF) to solve the simultaneous localization and mapping problem and uses efficient data structures for real-time data association, mapping, and spatial reasoning. We employ a hybrid map representation that infers 3D point landmarks from image features to achieve precise localization, coupled with occupancy grids for safe navigation. This paper describes our framework and implementation, and presents our exploration method, and experimental results illustrating the functionality of the system

ICRA Conference 2006 Conference Paper

σSLAM: Stereo Vision SLAM using the Rao-Blackwellised Particle Filter and a Novel Mixture Proposal Distribution

• Pantelis Elinas
• Robert Sim
• James J. Little

We consider the problem of simultaneous localization and mapping (SLAM) using the Rao-Blackwellised particle filter (RBPF) for the class of indoor mobile robots equipped only with stereo vision. Our goal is to construct dense metric maps of natural 3D point landmarks for large cyclic environments in the absence of accurate landmark position measurements and motion estimates. Our work differs from other approaches because landmark estimates are derived from stereo vision and motion estimates are based on sparse optical flow. We distinguish between landmarks using the scale invariant feature transform (SIFT). This is in contrast to current popular approaches that rely on reliable motion models derived from odometric hardware and accurate landmark measurements obtained with laser sensors. Since our approach depends on a particle filter whose main component is the proposal distribution, we develop and evaluate a novel mixture proposal distribution that allows us to robustly close large loops. We validate our approach experimentally for long camera trajectories processing thousands of images at reasonable frame rates

IROS Conference 2004 Conference Paper

Environment modeling with stereo vision

• Don Ray Murray
• James J. Little

We consider the problem of creating compact surface-based environment models from stereo vision images taken from a stereo-camera equipped mobile robot. The stereo images can be quite complex and correlation stereo suffers from considerable noise at ranges over a few metres. We construct the environment models by segmenting the scene viewed from a stereo camera into rectangular planar surfaces through the use of the patchlets surface element data structure. Patchlets are the projection of the stereo pixels onto detected surfaces in the scene. They have position, orientation, size and sensor-based confidence measures. The confidence measures allow proper weighting of patchlet parameters when aggregating patchlets into larger surfaces.

IROS Conference 2002 Conference Paper

Global localization using distinctive visual features

• Stephen Se
• David G. Lowe
• James J. Little

We have previously developed a mobile robot system which uses scale invariant visual landmarks to localize and simultaneously build a 3D map of the environment In this paper, we look at global localization, also known as the kidnapped robot problem, where the robot localizes itself globally, without any prior location estimate. This is achieved by matching distinctive landmarks in the current frame to a database map. A Hough transform approach and a random sample consensus (RANSAC) approach for global localization are compared, showing that RANSAC is much more efficient. Moreover, robust global localization can be achieved by matching a small sub-map of the local region built from multiple frames.

IROS Conference 2002 Conference Paper

Vision-based mapping with backward correction

• Stephen Se
• David G. Lowe
• James J. Little

We consider the problem of creating a consistent alignment of multiple 3D submaps containing distinctive visual landmarks in an unmodified environment. An efficient map alignment algorithm based on landmark specificity is proposed to align submaps. This is followed by a global minimization using the close-the-loop constraint. Landmark uncertainty is taken into account in the pairwise alignment and the global minimization process. Experiments show that the pairwise alignment of submaps with backward correction produces a consistent global 3D map. Our vision-based mapping approach using sparse 3D data is different from other existing approaches which use dense 2D range data from laser or sonar rangefinders.

ICRA Conference 2002 Conference Paper

Waiting with José, a Vision-Based Mobile Robot

• Pantelis Elinas
• Jesse Hoey
• Darrell Lahey
• Jefferson D. Montgomery
• Don Ray Murray
• Stephen Se
• James J. Little

Jose is a visually guided autonomous robotic waiter. He circulates around a room populated by groups of people, politely serving appetizers to humans. The serving task combines elements of robotics with human computer interaction, challenging control architecture with multiple task integration. This paper describes our purely vision-based approach to this task. Methods for mapping, localization and navigation are presented and discussed, including issues of safety for both robots and humans. Our work on human-robot interaction is covered, as well as our solutions to various tasks specific to serving food. We present results of our methods from sample experiments in our laboratory. We further discuss our experiences at the 2001 AAAI mobile robot "Hors D'oeuvres Anyone? " competition, at which Jose took first prize.

IROS Conference 2001 Conference Paper

Local and global localization for mobile robots using visual landmarks

• Stephen Se
• David G. Lowe
• James J. Little

Our mobile robot system uses scale-invariant visual landmarks to localize itself and build a 3D map of the environment simultaneously. As image features are not noise-free, we carry out error analysis and use Kalman filters to track the 3D landmarks, resulting in a database map with landmark positional uncertainty. By matching a set of landmarks as a whole, our robot can localize itself globally based on the database containing landmarks of sufficient distinctiveness. Experiments show that recognition of position within a map without any prior estimate can be achieved using the scale-invariant landmarks.

ICRA Conference 2001 Conference Paper

Vision-based Mobile Robot Localization And Mapping using Scale-Invariant Features

• Stephen Se
• David G. Lowe
• James J. Little

A key component of a mobile robot system is the ability to localize itself accurately and build a map of the environment simultaneously. In this paper, a vision-based mobile robot localization and mapping algorithm is described which uses scale-invariant image features as landmarks in unmodified dynamic environments. These 3D landmarks are localized and robot ego-motion is estimated by matching them, taking into account the feature viewpoint variation. With our Triclops stereo vision system, experiments show that these features are robustly matched between views, 3D landmarks are tracked, robot pose is estimated and a 3D map is built.

ICRA Conference 1999 Conference Paper

Cooperative Robot Localization with Vision-Based Mapping

• Cullen Jennings
• Don Ray Murray
• James J. Little

Two stereo vision-based mobile robots navigate and autonomously explore their environment safely while building occupancy grid maps of the environment. A novel landmark recognition system allows one robot to automatically find suitable landmarks in the environment. The second robot uses these landmarks to localize itself relative to the first robot's reference frame, even when the current state of the map is incomplete. The robots have a common local reference frame so that they can collaborate on tasks, without having a prior map of the environment. Stereo vision processing and map updates are done at 5 Hz and the robots move at 200 cm/s. Using occupancy grids the robots can robustly explore unstructured and dynamic environments. The map is used for path planning and landmark detection. Landmark detection uses the map's corner features and least-squares optimization to find the transformation between the robots' coordinate frames. The results provide very accurate relative localization without requiring highly accurate sensors. Accuracy of better than 2 cm was achieved in experiments.

IROS Conference 1998 Conference Paper

Selecting stable image features for robot localization using stereo

• James J. Little
• Jiping Lu
• Don Ray Murray

To navigate and recognize where it is, a mobile robot must be able to identify its current location. In an unknown initial position, a robot needs to refer to its environment to determine its location in an external coordinate system. Even with a known initial position, drift in odometry causes the estimated position to deviate from the correct position, requiring correction. We show how to find landmarks without models. We use dense stereo data from our mobile robot's trinocular system to discover image regions that will be stable over widely differing viewpoints. We find image brightness "corners" in images and select those that do not straddle depth discontinuities in the stereo depth data. Selecting corners only in regions of nearly planar stereo data results in landmarks that can be seen in images taken from different viewpoints.

IJCAI Conference 1985 Conference Paper

Determining Object Attitude from Extended Gaussian Images

• James J. Little

AAAI Conference 1983 Conference Paper

An Iterative Method for Reconstructing Convex Polyhedra from External Guassian Images

• James J. Little

In computing a scene description from an image, a useful intermediate representation of a scene object is given by the orientation and area of the constituent surface facets, termed the Extended Gaussian Image (EGI) of the object. The EGI of a convex object uniquely represents that object. We are con- cerned with the computational task of reconstructing the shape of scene objects from their Extended Gaussian Images, where the objects are restricted to convex polyhedra. We present an iterative method for reconstructing convex polyhe- dra from their Extended Gaussian Images.