Rahul Sukthankar

ICLR Conference 2022 Conference Paper

Discrete Representations Strengthen Vision Transformer Robustness

• Chengzhi Mao
• Lu Jiang 0004
• Mostafa Dehghani 0001
• Carl Vondrick
• Rahul Sukthankar
• Irfan Essa

Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs are overly reliant on local features (\eg, nuisances and texture) and fail to make adequate use of global context (\eg, shape and structure). As a result, ViTs fail to generalize to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different from the standard continuous pixel tokens, discrete tokens are invariant under small perturbations and contain less information individually, which promote ViTs to learn global information that is invariant. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12\% across seven ImageNet robustness benchmarks while maintaining the performance on ImageNet.

ICRA Conference 2022 Conference Paper

UFO Depth: Unsupervised learning with flow-based odometry optimization for metric depth estimation

• Vlad Licaret
• Victor Robu
• Alina Marcu
• Dragos Costea
• Emil Slusanschi
• Rahul Sukthankar
• Marius Leordeanu

We propose an efficient method for unsupervised learning of metric depth estimation from a single image in the context of unconstrained videos captured from UAVs. We combine the accuracy of an analytical solution based on odometry with the power of deep learning. First, we show how to correct the noisy odometric measurements by optimizing the alignment between the derotated optical flow and the projected linear speed in the image. Then, we detail an analytical depth estimation method based on optical flow and corrected camera velocities. Subsequently, the improved depth and camera veloc-ities obtained analytically are used, as additional cost terms, for training our novel unsupervised learning architecture for metric depth estimation. We extensively test on a recent UAV dataset, which we significantly extend by adding completely novel scenes. We outperform by significant margins different kinds of state-of-the-art approaches, ranging from analytical and unsupervised solutions to transformer-based architectures that require heavy computation and pre-training. The resulting algorithm could be deployed on embedded devices, being a good candidate for practical robotics use cases, such as obstacle avoidance and safe landing for UAV s.

AAAI Conference 2021 Conference Paper

Semi-Supervised Learning for Multi-Task Scene Understanding by Neural Graph Consensus

• Marius Leordeanu
• Mihai Cristian Pîrvu
• Dragos Costea
• Alina E Marcu
• Emil Slusanschi
• Rahul Sukthankar

We address the challenging problem of semi-supervised learning in the context of multiple visual interpretations of the world by finding consensus in a graph of neural networks. Each graph node is a scene interpretation layer, while each edge is a deep net that transforms one layer at one node into another from a different node. During the supervised phase edge networks are trained independently. During the next unsupervised stage edge nets are trained on the pseudo-ground truth provided by consensus among multiple paths that reach the nets’ start and end nodes. These paths act as ensemble teachers for any given edge and strong consensus is used for high-confidence supervisory signal. The unsupervised learning process is repeated over several generations, in which each edge becomes a ”student” and also part of different ensemble ”teachers” for training other students. By optimizing such consensus between different paths, the graph reaches consistency and robustness over multiple interpretations and generations, in the face of unknown labels. We give theoretical justifications of the proposed idea and validate it on a large dataset. We show how prediction of different representations such as depth, semantic segmentation, surface normals and pose from RGB input could be effectively learned through self-supervised consensus in our graph. We also compare to state-of-the-art methods for multi-task and semi-supervised learning and show superior performance.

IROS Conference 2020 Conference Paper

SelfieDroneStick: A Natural Interface for Quadcopter Photography

• Saif Alabachi
• Gita Sukthankar
• Rahul Sukthankar

A physical selfie stick extends the user's reach, enabling the acquisition of personal photos that include more of the background scene. Similarly, a quadcopter can capture photos from vantage points unattainable by the user; but teleoperating a quadcopter to good viewpoints is a difficult task. This paper presents a natural interface for quadcopter photography, the SelfieDroneStick that allows the user to guide the quadcopter to the optimal vantage point based on the phone's sensors. Users specify the composition of their desired long-range selfies using their smartphone, and the quadcopter autonomously flies to a sequence of vantage points from where the desired shots can be taken. The robot controller is trained from a combination of real-world images and simulated flight data. This paper describes two key innovations required to deploy deep reinforcement learning models on a real robot: 1) an abstract state representation for transferring learning from simulation to the hardware platform, and 2) reward shaping and staging paradigms for training the controller. Both of these improvements were found to be essential in learning a robot controller from simulation that transfers successfully to the real robot.

ICRA Conference 2019 Conference Paper

Customizing Object Detectors for Indoor Robots

• Saif Alabachi
• Gita Sukthankar
• Rahul Sukthankar

Object detection models based on convolutional neural networks (CNNs) demonstrate impressive performance when trained on large-scale labeled datasets. While a generic object detector trained on such a dataset performs adequately in applications where the input data is similar to user photographs, the detector performs poorly on small objects, particularly ones with limited training data or imaged from uncommon viewpoints. Also, a specific room will have many objects that are missed by standard object detectors, frustrating a robot that continually operates in the same indoor environment. This paper describes a system for rapidly creating customized object detectors. Data is collected from a quadcopter that is teleoperated with an interactive interface. Once an object is selected, the quadcopter autonomously photographs the object from multiple viewpoints to collect data to train DUNet (Dense Upscaled Network), our proposed model for learning customized object detectors from scratch given limited data. Our experiments compare the performance of learning models from scratch with DUNet vs. fine tuning existing state of the art object detectors, both on our indoor robotics domain and on standard datasets.

ICML Conference 2017 Conference Paper

Robust Adversarial Reinforcement Learning

• Lerrel Pinto
• James Davidson
• Rahul Sukthankar
• Abhinav Gupta 0001

Deep neural networks coupled with fast simulation and improved computational speeds have led to recent successes in the field of reinforcement learning (RL). However, most current RL-based approaches fail to generalize since: (a) the gap between simulation and real world is so large that policy-learning approaches fail to transfer; (b) even if policy learning is done in real world, the data scarcity leads to failed generalization from training to test scenarios (e. g. , due to different friction or object masses). Inspired from H-infinity control methods, we note that both modeling errors and differences in training and test scenarios can just be viewed as extra forces/disturbances in the system. This paper proposes the idea of robust adversarial reinforcement learning (RARL), where we train an agent to operate in the presence of a destabilizing adversary that applies disturbance forces to the system. The jointly trained adversary is reinforced – that is, it learns an optimal destabilization policy. We formulate the policy learning as a zero-sum, minimax objective function. Extensive experiments in multiple environments (InvertedPendulum, HalfCheetah, Swimmer, Hopper, Walker2d and Ant) conclusively demonstrate that our method (a) improves training stability; (b) is robust to differences in training/test conditions; and c) outperform the baseline even in the absence of the adversary.

AAAI Conference 2016 Conference Paper

Labeling the Features Not the Samples: Efficient Video Classification with Minimal Supervision

• Marius Leordeanu
• Alexandra Radu
• Shumeet Baluja
• Rahul Sukthankar

Feature selection is essential for effective visual recognition. We propose an efficient joint classifier learning and feature selection method that discovers sparse, compact representations of input features from a vast sea of candidates, with an almost unsupervised formulation. Our method requires only the following knowledge, which we call the feature sign—whether or not a particular feature has on average stronger values over positive samples than over negatives. We show how this can be estimated using as few as a single labeled training sample per class. Then, using these feature signs, we extend an initial supervised learning problem into an (almost) unsupervised clustering formulation that can incorporate new data without requiring ground truth labels. Our method works both as a feature selection mechanism and as a fully competitive classifier. It has important properties, low computational cost and excellent accuracy, especially in difficult cases of very limited training data. We experiment on large-scale recognition in video and show superior speed and performance to established feature selection approaches such as AdaBoost, Lasso, greedy forward-backward selection, and powerful classifiers such as SVM.

ICRA Conference 2013 Conference Paper

Multi-armed recommendation bandits for selecting state machine policies for robotic systems

• Pyry Matikainen
• P. Michael Furlong
• Rahul Sukthankar
• Martial Hebert

We investigate the problem of selecting a state-machine from a library to control a robot. We are particularly interested in this problem when evaluating such state machines on a particular robotics task is expensive. As a motivating example, we consider a problem where a simulated vacuuming robot must select a driving state machine well-suited for a particular (unknown) room layout. By borrowing concepts from collaborative filtering (recommender systems such as Netflix and Amazon.com), we present a multi-armed bandit formulation that incorporates recommendation techniques to efficiently select state machines for individual room layouts. We show that this formulation outperforms the individual approaches (recommendation, multi-armed bandits) as well as the baseline of selecting the `average best' state machine across all rooms.

NeurIPS Conference 2009 Conference Paper

An Integer Projected Fixed Point Method for Graph Matching and MAP Inference

• Marius Leordeanu
• Martial Hebert
• Rahul Sukthankar

Graph matching and MAP inference are essential problems in computer vision and machine learning. We introduce a novel algorithm that can accommodate both problems and solve them efficiently. Recent graph matching algorithms are based on a general quadratic programming formulation, that takes in consideration both unary and second-order terms reflecting the similarities in local appearance as well as in the pairwise geometric relationships between the matched features. In this case the problem is NP-hard and a lot of effort has been spent in finding efficiently approximate solutions by relaxing the constraints of the original problem. Most algorithms find optimal continuous solutions of the modified problem, ignoring during the optimization the original discrete constraints. The continuous solution is quickly binarized at the end, but very little attention is put into this final discretization step. In this paper we argue that the stage in which a discrete solution is found is crucial for good performance. We propose an efficient algorithm, with climbing and convergence properties, that optimizes in the discrete domain the quadratic score, and it gives excellent results either by itself or by starting from the solution returned by any graph matching algorithm. In practice it outperforms state-or-the art algorithms and it also significantly improves their performance if used in combination. When applied to MAP inference, the algorithm is a parallel extension of Iterated Conditional Modes (ICM) with climbing and convergence properties that make it a compelling alternative to the sequential ICM. In our experiments on MAP inference our algorithm proved its effectiveness by outperforming ICM and Max-Product Belief Propagation.

NeurIPS Conference 2008 Conference Paper

Semi-supervised Learning with Weakly-Related Unlabeled Data : Towards Better Text Categorization

• Liu Yang
• Rong Jin
• Rahul Sukthankar

The cluster assumption is exploited by most semi-supervised learning (SSL) methods. However, if the unlabeled data is merely weakly related to the target classes, it becomes questionable whether driving the decision boundary to the low density regions of the unlabeled data will help the classification. In such case, the cluster assumption may not be valid; and consequently how to leverage this type of unlabeled data to enhance the classification accuracy becomes a challenge. We introduce Semi-supervised Learning with Weakly-Related Unlabeled Data" (SSLW), an inductive method that builds upon the maximum-margin approach, towards a better usage of weakly-related unlabeled information. Although the SSLW could improve a wide range of classification tasks, in this paper, we focus on text categorization with a small training pool. The key assumption behind this work is that, even with different topics, the word usage patterns across different corpora tends to be consistent. To this end, SSLW estimates the optimal word-correlation matrix that is consistent with both the co-occurrence information derived from the weakly-related unlabeled documents and the labeled documents. For empirical evaluation, we present a direct comparison with a number of state-of-the-art methods for inductive semi-supervised learning and text categorization; and we show that SSLW results in a significant improvement in categorization accuracy, equipped with a small training set and an unlabeled resource that is weakly related to the test beds. "

UAI Conference 2007 Conference Paper

Bayesian Active Distance Metric Learning

• Liu Yang 0001
• Rong Jin 0001
• Rahul Sukthankar

Distance metric learning is an important component for many tasks, such as statistical classification and content-based image retrieval. Existing approaches for learning distance metrics from pairwise constraints typically suffer from two major problems. First, most algorithms only offer point estimation of the distance metric and can therefore be unreliable when the number of training examples is small. Second, since these algorithms generally select their training examples at random, they can be inefficient if labeling effort is limited. This paper presents a Bayesian framework for distance metric learning that estimates a posterior distribution for the distance metric from labeled pairwise constraints. We describe an efficient algorithm based on the variational method for the proposed Bayesian approach. Furthermore, we apply the proposed Bayesian framework to active distance metric learning by selecting those unlabeled example pairs with the greatest uncertainty in relative distance. Experiments in classification demonstrate that the proposed framework achieves higher classification accuracy and identifies more informative training examples than the non-Bayesian approach and state-of-the-art distance metric learning algorithms.

AAAI Conference 2006 Conference Paper

An Efficient Algorithm for Local Distance Metric Learning

• Liu Yang
• Rahul Sukthankar

Learning application-specific distance metrics from labeled data is critical for both statistical classification and information retrieval. Most of the earlier work in this area has focused on finding metrics that simultaneously optimize compactness and separability in a global sense. Specifically, such distance metrics attempt to keep all of the data points in each class close together while ensuring that data points from different classes are separated. However, particularly when classes exhibit multimodal data distributions, these goals conflict and thus cannot be simultaneously satisfied. This paper proposes a Local Distance Metric (LDM) that aims to optimize local compactness and local separability. We present an efficient algorithm that employs eigenvector analysis and bound optimization to learn the LDM from training data in a probabilistic framework. We demonstrate that LDM achieves significant improvements in both classification and retrieval accuracy compared to global distance learning and kernel-based KNN.

NeurIPS Conference 2006 Conference Paper

Distributed Inference in Dynamical Systems

• Stanislav Funiak
• Carlos Guestrin
• Rahul Sukthankar
• Mark Paskin

We present a robust distributed algorithm for approximate probabilistic inference in dynamical systems, such as sensor networks and teams of mobile robots. Using assumed density filtering, the network nodes maintain a tractable representation of the belief state in a distributed fashion. At each time step, the nodes coordinate to condition this distribution on the observations made throughout the network, and to advance this estimate to the next time step. In addition, we identify a significant challenge for probabilistic inference in dynamical systems: message losses or network partitions can cause nodes to have inconsistent beliefs about the current state of the system. We address this problem by developing distributed algorithms that guarantee that nodes will reach an informative consistent distribution when communication is re-established. We present a suite of experimental results on real-world sensor data for two real sensor network deployments: one with 25 cameras and another with 54 temperature sensors.

ICRA Conference 2005 Conference Paper

A Robust Visual Odometry and Precipice Detection System Using Consumer-grade Monocular Vision

• Jason Campbell
• Rahul Sukthankar
• Illah R. Nourbakhsh
• Aroon Pahwa

We describe a monocular robot vision system which accomplishes accurate 3-DOF dead-reckoning, closed loop motion control, and precipice and obstacle detection, all in dynamic environments, using a single, consumer-grade web cam and typical laptop computer hardware. Simultaneous translation and rotation are accurately measured, and the camera need not be placed at the robot’s center of rotation. The algorithm is straightforward to implement and robust to noisy measurements. The software is based on open source computer vision libraries and is itself open source. It has been tested in a wide variety of real-world environments and on several different mobile robot platforms.

IROS Conference 2004 Conference Paper

Techniques for evaluating optical flow for visual odometry in extreme terrain

• Jason Campbell
• Rahul Sukthankar
• Illah R. Nourbakhsh

Motion vision (visual odometry, the estimation of camera egomotion) is a well researched field, yet has seen relatively limited use despite strong evidence from biological systems that vision can be extremely valuable for navigation. The limited use of such vision techniques has been attributed to a lack of good algorithms and insufficient computer power, but both of those problems were resolved as long as a decade ago. A gap presently yawns between theory and practice, perhaps due to perceptions of robot vision as less reliable and more complex than other types of sensing. We present an experimental methodology for assessing the real world precision and reliability of visual odometry techniques in both normal and extreme terrain. This paper evaluates the performance of a mobile robot equipped with a simple vision system in common outdoor and indoor environments, including grass, pavement, ice, and carpet. Our results show that motion vision algorithms can be robust and effective, and suggest a number of directions for further development.

ICML Conference 2000 Conference Paper

Complete Cross-Validation for Nearest Neighbor Classifiers

• Matthew D. Mullin
• Rahul Sukthankar

AAAI Conference 1999 Conference Paper

ARGUS: An Automated Multi-Agent Visitor Identification System

• Rahul Sukthankar
• Carnegie Mellon University;
• Robert G. Stockton
• Just Research

ARGUS is a multi-agent visitor identification system distributed over several workstations. Human faces are extracted from security cameraimagesby a neuralnetwork-based face detector, and identified as frequent visitors by ARENA, a memory-basedface recognition system. ARGUS then uses a messaging system to notify hosts that their guests have arrived. Aninterface agent enables users to submit feedback, whichis immediately incorporated by ARENA to improve its face recognition performance. The ARGUS components were rapidly developed using JGram, an agent frameworkthat is also detailed in this paper. JGram automatically converts high-level agent specifications into Java source code, and assembles complextasks by composingindividual agent services into a JGram pipeline. ARGUS has been operating successfully in an outdoor environmentfor several months.

ICRA Conference 1997 Conference Paper

Evolving an intelligent vehicle for tactical reasoning in traffic

• Rahul Sukthankar
• Shumeet Baluja
• John A. Hancock

Recent research in automated highway systems has ranged from low-level vision-based controllers to high-level route-guidance software. However there is currently no system for tactical-level reasoning. Such a system should address tasks such as passing cars, making exits on time, and merging into a traffic stream. Our approach to this intermediate-level planning combines a distributed reasoning system (PolySAPIENT) with a novel evolutionary optimization strategy (PBIL). PBIL automatically tunes PolySAPIENT module parameters in simulation by evaluating candidate modules on various traffic scenarios. Since the control interface to the simulated vehicles is identical to that on the Carnegie Mellon Navlab vehicles, modules developed using this process can be directly ported to existing hardware. This method is currently being applied to the automated highway system domain; it also generalizes to many complex robotics tasks where multiple interacting modules must simultaneously be configured without individual module feedback.