Arrow Research search

Author name cluster

David Meger

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

59 papers
2 author rows

Possible papers

59

AAMAS Conference 2026 Conference Paper

Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddings

  • Zhizun Wang
  • David Meger

Learning to coordinate many agents in partially observable and highly dynamic environments requires both informative representations and data-efficient training. To address this challenge, we present a novel model-based multi-agent reinforcement learning framework that unifies joint state-action representation learning with imaginative roll-outs. We design a world model trained with variational auto-encoders and augment the model using the state-action learned embedding (SALE). SALE is injected into both the imagination module that forecasts plausible future roll-outs and the joint agent network whose individual action values are combined through a mixing network to estimate the joint actionvalue function. By coupling imagined trajectories with SALE-based action values, the agents acquire a richer understanding of how their choices influence collective outcomes, leading to improved long-termplanningandoptimizationunderlimitedreal-environment interactions. Empiricalstudiesonwell-establishedmulti-agentbenchmarks, including StarCraft II Micro-Management, Multi-Agent Mu- JoCo, and Level-Based Foraging challenges, demonstrate consistent gains of our method over baseline algorithms and highlight the effectiveness of joint state-action learned embeddings within a multi-agent model-based paradigm.

PDF

NeurIPS Conference 2025 Conference Paper

Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning

  • Yash Jhaveri
  • Harley Wiltzer
  • Patrick Shafto
  • Marc Bellemare
  • David Meger

In the pursuit of finding an optimal policy, reinforcement learning (RL) methods generally ignore the properties of learned policies apart from their expected return. Thus, even when successful, it is difficult to characterize which policies will be learned and what they will do. In this work, we present a theoretical framework for policy optimization that guarantees convergence to a particular optimal policy, via vanishing entropy regularization and a temperature decoupling gambit. Our approach realizes an interpretable, diversity-preserving optimal policy as the regularization temperature vanishes and ensures the convergence of policy derived objects--value functions and return distributions. In a particular instance of our method, for example, the realized policy samples all optimal actions uniformly. Leveraging our temperature decoupling gambit, we present an algorithm that estimates, to arbitrary accuracy, the return distribution associated to its interpretable, diversity-preserving optimal policy.

PDF Details

NeurIPS Conference 2025 Conference Paper

Epistemic Uncertainty Estimation in Regression Ensemble Models with Pairwise Epistemic Estimators

  • Lucas Berry
  • David Meger

This work introduces a novel approach, Pairwise Epistemic Estimators (PairEpEsts), for epistemic uncertainty estimation in ensemble models for regression tasks using pairwise-distance estimators (PaiDEs). By utilizing the pairwise distances between model components, PaiDEs establish bounds on entropy. We leverage this capability to enhance the performance of Bayesian Active Learning by Disagreement (BALD). Notably, unlike sample-based Monte Carlo estimators, PairEpEsts can estimate epistemic uncertainty up to 100 times faster and demonstrate superior performance in higher dimensions. To validate our approach, we conducted a varied series of regression experiments on commonly used benchmarks: 1D sinusoidal data, Pendulum, Hopper, Ant, and Humanoid, demonstrating PairEpEsts’ advantage over baselines in high-dimensional regression active learning.

PDF Details

ICRA Conference 2025 Conference Paper

Generalizable Imitation Learning Through Pre-Trained Representations

  • Wei-Di Chang
  • Francois Robert Hogan
  • Scott Fujimoto
  • David Meger
  • Gregory Dudek

In this paper, we leverage self-supervised vision transformer models and their emergent semantic abilities to improve the generalization abilities of imitation learning policies. We introduce DVK, an imitation learning algorithm that leverages rich pre-trained Visual Transformer patch-level embeddings to obtain better generalization when learning through demonstrations. Our learner sees the world by clustering appearance features into groups associated with semantic concepts, forming stable keypoints that generalize across a wide range of appearance variations and object types. We demonstrate how this representation enables generalized behaviour by evaluating imitation learning across a diverse dataset of object manipulation tasks. To facilitate further study of generalization in Imitation Learning, all of our code for the method and evaluation, as well as the dataset, is made available.

Details

ICRA Conference 2025 Conference Paper

Learning Active Tactile Perception Through Belief-Space Control

  • Jean-François Tremblay
  • David Meger
  • Francois Robert Hogan
  • Gregory Dudek

Robots operating in an open world will encounter novel objects with unknown physical properties, such as mass, friction, or size. These robots will need to sense these properties through interaction prior to performing downstream tasks with the objects. We propose a method that autonomously learns tactile exploration policies by developing a generative world model that is leveraged to 1) estimate the object's physical parameters using a differentiable Bayesian filtering algorithm and 2) develop an exploration policy using an information-gathering model predictive controller. We evaluate our method on three simulated tasks where the goal is to estimate a desired object property (mass, height or toppling height) through physical interaction. We find that our method is able to discover policies that efficiently gather information about the desired property in an intuitive manner. Finally, we validate our method on a real robot system for the height estimation task, where our method is able to successfully learn and execute an information-gathering policy from scratch.

Details

RLDM Conference 2025 Conference Abstract

RLDM 2025 Abstract Booklet 42 Tractable Representations for Convergent Approximation of Distributional HJB Equations Julie Alhosh Harley Wiltzer

  • ractable Representations for Convergent
  • Approximation of Distributional HJB Equations
  • Julie Alhosh Harley Wiltzer
  • David Meger

Booklet 42 Tractable Representations for Convergent Approximation of Distributional HJB Equations Julie Alhosh Harley Wiltzer School of Computer Science School of Computer Science McGill University Mila, McGill University Montréal, Québec, Canada Montréal, Québec, Canada julie. alhosh@mail. mcgill. ca harley. wiltzer@mail. mcgill. ca David Meger Centre for Intelligent Machines School of Computer Science McGill University Montréal, Québec, Canada dmeger@cim. mcgill. ca Abstract In reinforcement learning (RL), the long-term behavior of decision-making policies is evaluated based on their average returns. Distributional RL has emerged, presenting techniques for learning return distributions, which provide additional statistics for evaluating policies, incorporating risk- sensitive considerations. When the passage of time cannot naturally be divided into discrete time increments, researchers have studied the continuous-time RL (CTRL) problem, where agent states and decisions evolve continuously. In this setting, the Hamilton-Jacobi-Bellman (HJB) equation is well established as the characterization of the expected return, and many solution methods exist. However, the study of distributional RL in the continuous-time setting is in its infancy. Recent work has established a distributional HJB (DHJB) equation, providing the first characterization of return distributions in CTRL. These equations and their solutions are intractable to solve and rep- resent exactly, requiring novel approximation techniques. This work takes strides towards this end, establishing conditions on the method of parameterizing return distributions under which the DHJB equation can be approximately solved. Particularly, we show that under a certain topo- logical property of the mapping between statistics learned by a distributional RL algorithm and corresponding distributions, approximation of these statistics leads to close approximations of the solution of the DHJB equation. Concretely, we demonstrate that the quantile representation com- mon in distributional RL satisfies this topological property, certifying an efficient approximation algorithm for continuous-time distributional RL.

PDF Details

ICRA Conference 2025 Conference Paper

Topological Mapping for Traversability-Aware Long-Range Navigation in Off-Road Terrain

  • Jean-François Tremblay
  • Julie Alhosh
  • Louis Petit
  • Faraz Lotfi
  • Lara Landauro
  • David Meger

Autonomous robots navigating in off-road terrain like forests open new opportunities for automation. While off-road navigation has been studied, existing work often relies on clearly delineated pathways. We present a method allowing for long-range planning, exploration and low-level control in unknown off-trail forest terrain, using vision and GPS only. We represent outdoor terrain with a topological map, which is a set of panoramic snapshots connected with edges containing traversability information. A novel traversability analysis method is demonstrated, predicting the existence of a safe path towards a target in an image. Navigating between nodes is done using goal-conditioned behavior cloning, leveraging the power of a pretrained vision transformer. An exploration planner is presented, efficiently covering an unknown off-road area with unknown traversability using a frontiers-based approach. The approach is successfully deployed to autonomously explore two 400 m 2 forest sites unseen during training, in difficult conditions for navigation.

Details

NeurIPS Conference 2024 Conference Paper

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning

  • Harley Wiltzer
  • Marc G. Bellemare
  • David Meger
  • Patrick Shafto
  • Yash Jhaveri

When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In turn, their performance is inconsistent and often poor. Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, we establish that DRL agents are sensitive to the decision frequency. We prove that action-conditioned return distributions collapse to their underlying policy's return distribution as the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse at different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the superiority as a probabilistic generalization of the advantage---the core object of approaches to mitigating performance issues in high-frequency value-based RL. In addition, we build a superiority-based DRL algorithm. Through simulations in an option-trading domain, we validate that proper modeling of the superiority distribution produces improved controllers at high decision frequencies.

PDF Details DOI

RLC Conference 2024 Conference Paper

Imitation Learning from Observation through Optimal Transport

  • Wei-Di Chang
  • Scott Fujimoto
  • David Meger
  • Gregory Dudek

Imitation Learning from Observation (ILfO) is a setting in which a learner tries to imitate the behavior of an expert, using only observational data and without the direct guidance of demonstrated actions. In this paper, we re-examine optimal transport for IL, in which a reward is generated based on the Wasserstein distance between the state trajectories of the learner and expert. We show that existing methods can be simplified to generate a reward function without requiring learned models or adversarial learning. Unlike many other state-of-the-art methods, our approach can be integrated with any RL algorithm and is amenable to ILfO. We demonstrate the effectiveness of this simple approach on a variety of continuous control tasks and find that it surpasses the state of the art in the IlfO setting, achieving expert-level performance across a range of evaluation domains even when observing only a single expert trajectory without actions.

PDF Details

RLJ Journal 2024 Journal Article

Imitation Learning from Observation through Optimal Transport

  • Wei-Di Chang
  • Scott Fujimoto
  • David Meger
  • Gregory Dudek

Imitation Learning from Observation (ILfO) is a setting in which a learner tries to imitate the behavior of an expert, using only observational data and without the direct guidance of demonstrated actions. In this paper, we re-examine optimal transport for IL, in which a reward is generated based on the Wasserstein distance between the state trajectories of the learner and expert. We show that existing methods can be simplified to generate a reward function without requiring learned models or adversarial learning. Unlike many other state-of-the-art methods, our approach can be integrated with any RL algorithm and is amenable to ILfO. We demonstrate the effectiveness of this simple approach on a variety of continuous control tasks and find that it surpasses the state of the art in the IlfO setting, achieving expert-level performance across a range of evaluation domains even when observing only a single expert trajectory without actions.

PDF Details

NeurIPS Conference 2024 Conference Paper

Parseval Regularization for Continual Reinforcement Learning

  • Wesley Chung
  • Lynn Cherif
  • David Meger
  • Doina Precup

Plasticity loss, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequences of tasks---referring to the increased difficulty in training on new tasks. We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in a continual reinforcement learning setting. We show that it provides significant benefits to RL agents on a suite of gridworld, CARL and MetaWorld tasks. We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated to network trainability including weight matrix rank, weight norms and policy entropy.

PDF Details DOI

JMLR Journal 2024 Journal Article

Policy Gradient Methods in the Presence of Symmetries and State Abstractions

  • Prakash Panangaden
  • Sahand Rezaei-Shoshtari
  • Rosie Zhao
  • David Meger
  • Doina Precup

Reinforcement learning (RL) on high-dimensional and complex problems relies on abstraction for improved efficiency and generalization. In this paper, we study abstraction in the continuous-control setting, and extend the definition of Markov decision process (MDP) homomorphisms to the setting of continuous state and action spaces. We derive a policy gradient theorem on the abstract MDP for both stochastic and deterministic policies. Our policy gradient results allow for leveraging approximate symmetries of the environment for policy optimization. Based on these theorems, we propose a family of actor-critic algorithms that are able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. Finally, we introduce a series of environments with continuous symmetries to further demonstrate the ability of our algorithm for action abstraction in the presence of such symmetries. We demonstrate the effectiveness of our method on our environments, as well as on challenging visual control tasks from the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance, and the visualizations of the latent space clearly demonstrate the structure of the learned abstraction. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2024. ( edit, beta )

PDF Details

UAI Conference 2024 Conference Paper

Shedding Light on Large Generative Networks: Estimating Epistemic Uncertainty in Diffusion Models

  • Lucas Berry
  • Axel Brando
  • David Meger

Generative diffusion models, notable for their large parameter count (exceeding 100 million) and operation within high-dimensional image spaces, pose significant challenges for traditional uncertainty estimation methods due to computational demands. In this work, we introduce an innovative framework, Diffusion Ensembles for Capturing Uncertainty (DECU), designed for estimating epistemic uncertainty for diffusion models. The DECU framework introduces a novel method that efficiently trains ensembles of conditional diffusion models by incorporating a static set of pre-trained parameters, drastically reducing the computational burden and the number of parameters that require training. Additionally, DECU employs Pairwise-Distance Estimators (PaiDEs) to accurately measure epistemic uncertainty by evaluating the mutual information between model outputs and weights in high-dimensional spaces. The effectiveness of this framework is demonstrated through experiments on the ImageNet dataset, highlighting its capability to capture epistemic uncertainty, specifically in under-sampled image classes.

Details

ICRA Conference 2024 Conference Paper

Uncertainty-aware hybrid paradigm of nonlinear MPC and model-based RL for offroad navigation: Exploration of transformers in the predictive model

  • Faraz Lotfi
  • Khalil Virji
  • Farnoosh Faraji
  • Lucas Berry
  • Andrew Holliday
  • David Meger
  • Gregory Dudek

In this paper, we investigate a hybrid scheme that combines nonlinear model predictive control (MPC) and model-based reinforcement learning (RL) for navigation planning of an autonomous model car across offroad, unstructured terrains without relying on predefined maps. Our innovative approach takes inspiration from BADGR, an LSTM-based network that primarily concentrates on environment modeling, but distinguishes itself by substituting LSTM modules with transformers to greatly elevate the performance of our model. Addressing uncertainty within the system, we train an ensemble of predictive models and estimate the mutual information between model weights and outputs, facilitating dynamic horizon planning through the introduction of variable speeds. Further enhancing our methodology, we incorporate a nonlinear MPC controller that accounts for the intricacies of the vehicle’s model and states. The model-based RL facet produces steering angles and quantifies inherent uncertainty. At the same time, the nonlinear MPC suggests optimal throttle settings, striking a balance between goal attainment speed and managing model uncertainty influenced by velocity. In the conducted studies, our approach excels over the existing baseline by consistently achieving higher metric values in predicting future events and seamlessly integrating the vehicle’s kinematic model for enhanced decision-making. The code and the evaluation data are available at (Github-repo).

Details

ICRA Conference 2023 Conference Paper

ANSEL Photobot: A Robot Event Photographer with Semantic Intelligence

  • Dmitriy Rivkin
  • Gregory Dudek
  • Nikhil Kakodkar
  • David Meger
  • Oliver Limoyo
  • Michael Jenkin
  • Xue Liu 0004
  • Francois Robert Hogan

Our work examines the way in which large language models can be used for robotic planning and sampling in the context of automated photographic documentation. Specifically, we illustrate how to produce a photo-taking robot with an exceptional level of semantic awareness by leveraging recent advances in general purpose language (LM) and vision-language (VLM) models. Given a high-level description of an event we use an LM to generate a natural-language list of photo descriptions that one would expect a photographer to capture at the event. We then use a VLM to identify the best matches to these descriptions in the robot's video stream. The photo portfolios generated by our method are consistently rated as more appropriate to the event by human evaluators than those generated by existing methods.

Details

NeurIPS Conference 2023 Conference Paper

For SALE: State-Action Representation Learning for Deep Reinforcement Learning

  • Scott Fujimoto
  • Wei-Di Chang
  • Edward Smith
  • Shixiang (Shane) Gu
  • Doina Precup
  • David Meger

In reinforcement learning (RL), representation learning is a proven tool for complex image-based tasks, but is often overlooked for environments with low-level states, such as physical control problems. This paper introduces SALE, a novel approach for learning embeddings that model the nuanced interaction between state and action, enabling effective representation learning from low-level states. We extensively study the design space of these embeddings and highlight important design considerations. We integrate SALE and an adaptation of checkpoints for RL into TD3 to form the TD7 algorithm, which significantly outperforms existing continuous control algorithms. On OpenAI gym benchmark tasks, TD7 has an average performance gain of 276. 7% and 50. 7% over TD3 at 300k and 5M time steps, respectively, and works in both the online and offline settings.

PDF Details

AAAI Conference 2023 Conference Paper

Hypernetworks for Zero-Shot Transfer in Reinforcement Learning

  • Sahand Rezaei-Shoshtari
  • Charlotte Morissette
  • Francois R. Hogan
  • Gregory Dudek
  • David Meger

In this paper, hypernetworks are trained to generate behaviors across a range of unseen task conditions, via a novel TD-based training objective and data from a set of near-optimal RL solutions for training tasks. This work relates to meta RL, contextual RL, and transfer learning, with a particular focus on zero-shot performance at test time, enabled by knowledge of the task parameters (also known as context). Our technical approach is based upon viewing each RL algorithm as a mapping from the MDP specifics to the near-optimal value function and policy and seek to approximate it with a hypernetwork that can generate near-optimal value functions and policies, given the parameters of the MDP. We show that, under certain conditions, this mapping can be considered as a supervised learning problem. We empirically evaluate the effectiveness of our method for zero-shot transfer to new reward and transition dynamics on a series of continuous control tasks from DeepMind Control Suite. Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Normalizing Flow Ensembles for Rich Aleatoric and Epistemic Uncertainty Modeling

  • Lucas Berry
  • David Meger

In this work, we demonstrate how to reliably estimate epistemic uncertainty while maintaining the flexibility needed to capture complicated aleatoric distributions. To this end, we propose an ensemble of Normalizing Flows (NF), which are state-of-the-art in modeling aleatoric uncertainty. The ensembles are created via sets of fixed dropout masks, making them less expensive than creating separate NF models. We demonstrate how to leverage the unique structure of NFs, base distributions, to estimate aleatoric uncertainty without relying on samples, provide a comprehensive set of baselines, and derive unbiased estimates for differential entropy. The methods were applied to a variety of experiments, commonly used to benchmark aleatoric and epistemic uncertainty estimation: 1D sinusoidal data, 2D windy grid-world (Wet Chicken), Pendulum, and Hopper. In these experiments, we setup an active learning framework and evaluate each model's capability at measuring aleatoric and epistemic uncertainty. The results show the advantages of using NF ensembles in capturing complicated aleatoric while maintaining accurate epistemic uncertainty estimates.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

Continuous MDP Homomorphisms and Homomorphic Policy Gradient

  • Sahand Rezaei-Shoshtari
  • Rosie Zhao
  • Prakash Panangaden
  • David Meger
  • Doina Precup

Abstraction has been widely studied as a way to improve the efficiency and generalization of reinforcement learning algorithms. In this paper, we study abstraction in the continuous-control setting. We extend the definition of MDP homomorphisms to encompass continuous actions in continuous state spaces. We derive a policy gradient theorem on the abstract MDP, which allows us to leverage approximate symmetries of the environment for policy optimization. Based on this theorem, we propose an actor-critic algorithm that is able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. We demonstrate the effectiveness of our method on benchmark tasks in the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance when learning from pixel observations.

PDF Details

ICML Conference 2022 Conference Paper

Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning

  • Harley Wiltzer
  • David Meger
  • Marc G. Bellemare

Continuous-time reinforcement learning offers an appealing formalism for describing control problems in which the passage of time is not naturally divided into discrete increments. Here we consider the problem of predicting the distribution of returns obtained by an agent interacting in a continuous-time, stochastic environment. Accurate return predictions have proven useful for determining optimal policies for risk-sensitive control, learning state representations, multiagent coordination, and more. We begin by establishing the distributional analogue of the Hamilton-Jacobi-Bellman (HJB) equation for Ito diffusions and the broader class of Feller-Dynkin processes. We then specialize this equation to the setting in which the return distribution is approximated by N uniformly-weighted particles, a common design choice in distributional algorithms. Our derivation highlights additional terms due to statistical diffusivity which arise from the proper handling of distributions in the continuous-time setting. Based on this, we propose a tractable algorithm for approximately solving the distributional HJB based on a JKO scheme, which can be implemented in an online, control algorithm. We demonstrate the effectiveness of such an algorithm in a synthetic control problem.

Details

ICRA Conference 2022 Conference Paper

Visuotactile-RL: Learning Multimodal Manipulation Policies with Deep Reinforcement Learning

  • Johanna Hansen
  • Francois Robert Hogan
  • Dmitriy Rivkin
  • David Meger
  • Michael Jenkin
  • Gregory Dudek

Manipulating objects with dexterity requires timely feedback that simultaneously leverages the senses of vision and touch. In this paper, we focus on the problem setting where both visual and tactile sensors provide pixel-level feedback for Visuotactile reinforcement learning agents. We investigate the challenges associated with multimodal learning and propose several improvements to existing RL methods; including tactile gating, tactile data augmentation, and visual degradation. When compared with visual-only and tactile-only baselines, our Visuotactile-RL agents showcase (1) significant improvements in contact-rich tasks; (2) improved robustness to visual changes (lighting/camera view) in the workspace; and (3) resilience to physical changes in the task environment (weight/friction of objects).

Details

ICML Conference 2022 Conference Paper

Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error

  • Scott Fujimoto
  • David Meger
  • Doina Precup
  • Ofir Nachum
  • Shixiang Gu

In this work, we study the use of the Bellman equation as a surrogate objective for value prediction accuracy. While the Bellman equation is uniquely solved by the true value function over all state-action pairs, we find that the Bellman error (the difference between both sides of the equation) is a poor proxy for the accuracy of the value function. In particular, we show that (1) due to cancellations from both sides of the Bellman equation, the magnitude of the Bellman error is only weakly related to the distance to the true value function, even when considering all state-action pairs, and (2) in the finite data regime, the Bellman equation can be satisfied exactly by infinitely many suboptimal solutions. This means that the Bellman error can be minimized without improving the accuracy of the value function. We demonstrate these phenomena through a series of propositions, illustrative toy examples, and empirical analysis in standard benchmark domains.

Details

ICML Conference 2021 Conference Paper

A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation

  • Scott Fujimoto
  • David Meger
  • Doina Precup

Marginalized importance sampling (MIS), which measures the density ratio between the state-action occupancy of a target policy and that of a sampling distribution, is a promising approach for off-policy evaluation. However, current state-of-the-art MIS methods rely on complex optimization tricks and succeed mostly on simple toy problems. We bridge the gap between MIS and deep reinforcement learning by observing that the density ratio can be computed from the successor representation of the target policy. The successor representation can be trained through deep reinforcement learning methodology and decouples the reward optimization from the dynamics of the environment, making the resulting algorithm stable and applicable to high-dimensional domains. We evaluate the empirical performance of our approach on a variety of challenging Atari and MuJoCo environments.

Details

NeurIPS Conference 2021 Conference Paper

Active 3D Shape Reconstruction from Vision and Touch

  • Edward Smith
  • David Meger
  • Luis Pineda
  • Roberto Calandra
  • Jitendra Malik
  • Adriana Romero Soriano
  • Michal Drozdzal

Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. However, in 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings, leaving the active exploration of the shape largely unexplored. In active touch sensing for 3D reconstruction, the goal is to actively select the tactile readings that maximize the improvement in shape reconstruction accuracy. However, the development of deep learning-based active touch models is largely limited by the lack of frameworks for shape exploration. In this paper, we focus on this problem and introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile signals; and 3) a set of data-driven solutions with either tactile or visuotactile priors to guide the shape exploration. Our framework enables the development of the first fully data-driven solutions to active touch on top of learned models for object understanding. Our experiments show the benefits of such solutions in the task of 3D shape understanding where our models consistently outperform natural baselines. We provide our framework as a tool to foster future research in this direction.

PDF Details

IROS Conference 2021 Conference Paper

Latent Attention Augmentation for Robust Autonomous Driving Policies

  • Ran Cheng
  • Christopher Agia
  • Florian Shkurti
  • David Meger
  • Gregory Dudek

Model-free reinforcement learning has become a viable approach for vision-based robot control. However, sample complexity and adaptability to domain shifts remain persistent challenges when operating in high-dimensional observation spaces (images, LiDAR), such as those that are involved in autonomous driving. In this paper, we propose a flexible framework by which a policy’s observations are augmented with robust attention representations in the latent space to guide the agent’s attention during training. Our method encodes local and global descriptors of the augmented state representations into a compact latent vector, and scene dynamics are approximated by a recurrent network that processes the latent vectors in sequence. We outline two approaches for constructing attention maps; a supervised pipeline leveraging semantic segmentation networks, and an unsupervised pipeline relying only on classical image processing techniques. We conduct our experiments in simulation and test the learned policy against varying seasonal effects and weather conditions. Our design decisions are supported in a series of ablation studies. The results demonstrate that our state augmentation method both improves learning efficiency and encourages robust domain adaptation when compared to common end-to-end frameworks and methods that learn directly from intermediate representations.

Details

AAAI Conference 2021 Conference Paper

Learning Intuitive Physics with Multimodal Generative Models

  • Sahand Rezaei-Shoshtari
  • Francois R. Hogan
  • Michael Jenkin
  • David Meger
  • Gregory Dudek

Predicting the future interaction of objects when they come into contact with their environment is key for autonomous agents to take intelligent and anticipatory actions. This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. Visual information captures object properties such as 3D shape and location, while tactile information provides critical cues about interaction forces and resulting object motion when it makes contact with the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces, our system captures both the visual appearance and the tactile properties of objects. We interpret the dual stream signals from the sensor using a Multimodal Variational Autoencoder (MVAE), allowing us to capture both modalities of contacting objects and to develop a mapping from visual to tactile interaction and vice-versa. Additionally, the perceptual system can be used to infer the outcome of future physical interactions, which we validate through simulated and realworld experiments in which the resting state of an object is predicted from given initial conditions.

PDF Details

ICRA Conference 2021 Conference Paper

Multimodal dynamics modeling for off-road autonomous vehicles

  • Jean-François Tremblay
  • Travis Manderson
  • Aurélio Noca
  • Gregory Dudek
  • David Meger

Dynamics modeling in outdoor and unstructured environments is difficult because different elements in the environment interact with the robot in ways that can be hard to predict. Leveraging multiple sensors to perceive maximal information about the robot’s environment is thus crucial when building a model to perform predictions about the robot’s dynamics with the goal of doing motion planning. We design a model capable of long-horizon motion predictions, leveraging vision, lidar and proprioception, which is robust to arbitrarily missing modalities at test time. We demonstrate in simulation that our model is able to leverage vision to predict traction changes. We then test our model using a real-world challenging dataset of a robot navigating through a forest, performing predictions in trajectories unseen during training. We try different modality combinations at test time and show that, while our model performs best when all modalities are present, it is still able to perform better than the baseline even when receiving only raw vision input and no proprioception, as well as when only receiving proprioception. Overall, our study demonstrates the importance of leveraging multiple sensors when doing dynamics modeling in outdoor conditions.

Details

IROS Conference 2021 Conference Paper

Trajectory-Constrained Deep Latent Visual Attention for Improved Local Planning in Presence of Heterogeneous Terrain

  • Stefan Wapnick
  • Travis Manderson
  • David Meger
  • Gregory Dudek

We present a reward-predictive, model-based learning method featuring trajectory-constrained visual attention for use in mapless, local visual navigation tasks. Our method learns to place visual attention at locations in latent image space which follow trajectories caused by vehicle control actions to later enhance predictive accuracy during planning. Our attention model is jointly optimized by the task-specific loss and additional trajectory-constraint loss, allowing adaptability yet encouraging a regularized structure for improved generalization and reliability. Importantly, visual attention is applied in latent feature map space instead of raw image space to promote efficient planning. We validated our model in visual navigation tasks of planning low turbulence, collision-free trajectories in off-road settings and hill climbing with locking differentials in the presence of slippery terrain. Experiments involved randomized procedural generated simulation and real-world environments. We found our method improved generalization and learning efficiency when compared to no-attention and self-attention alternatives.

Details

NeurIPS Conference 2020 Conference Paper

3D Shape Reconstruction from Vision and Touch

  • Edward Smith
  • Roberto Calandra
  • Adriana Romero
  • Georgia Gkioxari
  • David Meger
  • Jitendra Malik
  • Michal Drozdzal

When a toddler is presented a new toy, their instinctual behaviour is to pick it up and inspect it with their hand and eyes in tandem, clearly searching over its surface to properly understand what they are playing with. At any instance here, touch provides high fidelity localized information while vision provides complementary global context. However, in 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. In this paper, we study this problem and present an effective chart-based approach to multi-modal shape understanding which encourages a similar fusion vision and touch information. To do so, we introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects. Our results show that (1) leveraging both vision and touch signals consistently improves single- modality baselines; (2) our approach outperforms alternative modality fusion methods and strongly benefits from the proposed chart-based structure; (3) the reconstruction quality increases with the number of grasps provided; and (4) the touch information not only enhances the reconstruction at the touch site but also extrapolates to its local neighborhood.

PDF Details

NeurIPS Conference 2020 Conference Paper

An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay

  • Scott Fujimoto
  • David Meger
  • Doina Precup

Prioritized Experience Replay (PER) is a deep reinforcement learning technique in which agents learn from transitions sampled with non-uniform probability proportionate to their temporal-difference error. We show that any loss function evaluated with non-uniformly sampled data can be transformed into another uniformly sampled loss function with the same expected gradient. Surprisingly, we find in some environments PER can be replaced entirely by this new loss function without impact to empirical performance. Furthermore, this relationship suggests a new branch of improvements to PER by correcting its uniformly sampled loss function equivalent. We demonstrate the effectiveness of our proposed modifications to PER and the equivalent loss function in several MuJoCo and Atari environments.

PDF Details

IROS Conference 2020 Conference Paper

Learning Domain Randomization Distributions for Training Robust Locomotion Policies

  • Melissa Mozifian
  • Juan Camilo Gamboa Higuera
  • David Meger
  • Gregory Dudek

This paper considers the problem of learning behaviors in simulation without knowledge of the precise dynamical properties of the target robot platform(s). In this context, our learning goal is to mutually maximize task efficacy on each environment considered and generalization across the widest possible range of environmental conditions. The physical parameters of the simulator are modified by a component of our technique that learns the Domain Randomization (DR) that is appropriate at each learning epoch to maximally challenge the current behavior policy, without being overly challenging, which can hinder learning progress. This so-called sweet spot distribution is a selection of simulated domains with the following properties: 1) The trained policy should be successful in environments sampled from the domain randomization distribution; and 2) The DR distribution made as wide as possible, to increase variability in the environments. These properties aim to ensure the trajectories encountered in the target system are close to those observed during training, as existing methods in machine learning are better suited for interpolation than extrapolation. We show how adapting the DR distribution while training context-conditioned policies results in improvements on jump-start and asymptotic performance when transferring a learned policy to the target environment 1.

Details

IROS Conference 2020 Conference Paper

Learning the Latent Space of Robot Dynamics for Cutting Interaction Inference

  • Sahand Rezaei-Shoshtari
  • David Meger
  • Inna Sharf

Utilization of latent space to capture a lower-dimensional representation of a complex dynamics model is explored in this work. The targeted application is of a robotic manipulator executing a complex environment interaction task, in particular, cutting a wooden object. We train two flavours of Variational Autoencoders-standard and Vector-Quantised-to learn the latent space which is then used to infer certain properties of the cutting operation, such as whether the robot is cutting or not, as well as, material and geometry of the object being cut. The two VAE models are evaluated with reconstruction, prediction and a combined reconstruction/prediction decoders. The results demonstrate the expressiveness of the latent space for robotic interaction inference and the competitive prediction performance against recurrent neural networks.

Details

ICRA Conference 2020 Conference Paper

Learning to Drive Off Road on Smooth Terrain in Unstructured Environments Using an On-Board Camera and Sparse Aerial Images

  • Travis Manderson
  • Stefan Wapnick
  • David Meger
  • Gregory Dudek

We present a method for learning to drive on smooth terrain while simultaneously avoiding collisions in challenging off-road and unstructured outdoor environments using only visual inputs. Our approach applies a hybrid model-based and model-free reinforcement learning method that is entirely self-supervised in labeling terrain roughness and collisions using on-board sensors. Notably, we provide both first-person and overhead aerial image inputs to our model. We nd that the fusion of these complementary inputs improves planning foresight and makes the model robust to visual obstructions. Our results show the ability to generalize to environments with plentiful vegetation, various types of rock, and sandy trails. During evaluation, our policy attained 90% smooth terrain traversal and reduced the proportion of rough terrain driven over by 6. 1 times compared to a model using only first-person imagery. Video and project details can be found at www. cim. mcgill. ca/mrl/offroad_driving/.

Details

IROS Conference 2020 Conference Paper

PresSense: Passive Respiration Sensing via Ambient WiFi Signals in Noisy Environments

  • Yi Tian Xu
  • Xi Chen 0009
  • Xue Liu 0004
  • David Meger
  • Gregory Dudek

Passive sensing with ambient WiFi signals is a promising technique that will enable new types of human-robot interactions while preserving users' privacy. Here, we present PresSense, a system for human respiration sensing in noisy environments. Unlike existing WiFi-based respiration sensors, we employ a human presence detector, improving the robustness in scenarios where no human is present in an Area Of Interest (AOI). We also integrate our novel feature, Peak Distance Histogram (PDH), with other classic WiFi features to achieve better accuracy when someone is present in the AOI. We tested our system using commodity WiFi devices in an office room. Our PresSense outperforms the state of the arts in both respiration rate estimation and presence detection.

Details

ICRA Conference 2020 Conference Paper

View-Invariant Loop Closure with Oriented Semantic Landmarks

  • Jimmy Li 0001
  • Karim Koreitem
  • David Meger
  • Gregory Dudek

Recent work on semantic simultaneous localization and mapping (SLAM) have shown the utility of natural objects as landmarks for improving localization accuracy and robustness. In this paper we present a monocular semantic SLAM system that uses object identity and inter-object geometry for view-invariant loop detection and drift correction. Our system's ability to recognize an area of the scene even under large changes in viewing direction allows it to surpass the mapping accuracy of ORB-SLAM, which uses only local appearance-based features that are not robust to large viewpoint changes. Experiments on real indoor scenes show that our method achieves mean drift reduction of 70% when compared directly to ORB-SLAM. Additionally, we propose a method for object orientation estimation, where we leverage the tracked pose of a moving camera under the SLAM setting to overcome ambiguities caused by object symmetry. This allows our SLAM system to produce geometrically detailed semantic maps with object orientation, translation, and scale.

Details

IROS Conference 2019 Conference Paper

Cascaded Gaussian Processes for Data-efficient Robot Dynamics Learning

  • Sahand Rezaei-Shoshtari
  • David Meger
  • Inna Sharf

Motivated by the recursive Newton-Euler formulation, we propose a novel cascaded Gaussian process learning framework for the inverse dynamics of robot manipulators. This approach leads to a significant dimensionality reduction which in turn results in better learning and data efficiency. We explore two formulations for the cascading: the inward and outward, both along the manipulator chain topology. The learned modeling is tested in conjunction with the classical inverse dynamics model (semi-parametric) and on its own (non-parametric) in the context of feed-forward control of the arm. Experimental results are obtained with Jaco 2 six-DOF and SARCOS seven-DOF manipulators for randomly defined sinusoidal motions of the joints in order to evaluate the performance of cascading against the standard GP learning. In addition, experiments are conducted using Jaco 2 on a task emulating a pouring maneuver. Results indicate a consistent improvement in learning speed with the inward cascaded GP model and an overall improvement in data efficiency and generalization.

Details

ICML Conference 2019 Conference Paper

GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects

  • Edward J. Smith
  • Scott Fujimoto
  • Adriana Romero
  • David Meger

Mesh models are a promising approach for encoding the structure of 3D objects. Current mesh reconstruction systems predict uniformly distributed vertex locations of a predetermined graph through a series of graph convolutions, leading to compromises with respect to performance or resolution. In this paper, we argue that the graph representation of geometric objects allows for additional structure, which should be leveraged for enhanced reconstruction. Thus, we propose a system which properly benefits from the advantages of the geometric structure of graph-encoded objects by introducing (1) a graph convolutional update preserving vertex information; (2) an adaptive splitting heuristic allowing detail to emerge; and (3) a training objective operating both on the local surfaces defined by vertices as well as the global structure defined by the mesh. Our proposed method is evaluated on the task of 3D object reconstruction from images with the ShapeNet dataset, where we demonstrate state of the art performance, both visually and numerically, while having far smaller space requirements by generating adaptive meshes.

Details

RLDM Conference 2019 Conference Abstract

Off-Policy Deep Reinforcement Learning without Exploration

  • Scott Fujimoto
  • David Meger
  • Doina Precup

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this pa- per, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distri- bution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

PDF Details

ICML Conference 2019 Conference Paper

Off-Policy Deep Reinforcement Learning without Exploration

  • Scott Fujimoto
  • David Meger
  • Doina Precup

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

Details

ICRA Conference 2019 Conference Paper

Semantic Mapping for View-Invariant Relocalization

  • Jimmy Li 0001
  • David Meger
  • Gregory Dudek

We propose a system for visual simultaneous localization and mapping (SLAM) that combines traditional local appearance-based features with semantically meaningful object landmarks to achieve both accurate local tracking and highly view-invariant object-driven relocalization. Our mapping process uses a sampling-based approach to efficiently infer the 3D pose of object landmarks from 2D bounding box object detections. These 3D landmarks then serve as a view-invariant representation which we leverage to achieve camera relocalization even when the viewing angle changes by more than 125 degrees. This level of view-invariance cannot be attained by local appearance-based features (e. g. SIFT) since the same set of surfaces are not even visible when the viewpoint changes significantly. Our experiments show that even when existing methods fail completely for viewpoint changes of more than 70 degrees, our method continues to achieve a relocalization rate of around 90%, with a mean rotational error of around 8 degrees.

Details

ICRA Conference 2019 Conference Paper

Uncertainty Aware Learning from Demonstrations in Multiple Contexts using Bayesian Neural Networks

  • Sanjay Thakur
  • Herke van Hoof
  • Juan Camilo Gamboa Higuera
  • Doina Precup
  • David Meger

Diversity of environments is a key challenge that causes learned robotic controllers to fail due to the discrepancies between the training and evaluation conditions. Training from demonstrations in various conditions can mitigate - but not completely prevent - such failures. Learned controllers such as neural networks typically do not have a notion of uncertainty that allows to diagnose an offset between training and testing conditions, and potentially intervene. In this work, we propose to use Bayesian Neural Networks, which have such a notion of uncertainty. We show that uncertainty can be leveraged to consistently detect situations in high-dimensional simulated and real robotic domains in which the performance of the learned controller would be sub-par. Also, we show that such an uncertainty based solution allows making an informed decision about when to invoke a fallback strategy. One fallback strategy is to request more data. We empirically show that providing data only when requested results in increased data-efficiency.

Details

ICML Conference 2018 Conference Paper

Addressing Function Approximation Error in Actor-Critic Methods

  • Scott Fujimoto
  • Herke van Hoof
  • David Meger

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

Details

IROS Conference 2018 Conference Paper

Cost Adaptation for Robust Decentralized Swarm Behaviour

  • Peter Henderson 0002
  • Matthew Vertescher
  • David Meger
  • Mark Coates

Decentralized receding horizon control (D-RHC) provides a mechanism for coordination in multiagent settings without a centralized command center. However, combining a set of different goals, costs, and constraints to form an efficient optimization objective for D-RHC can be difficult. To allay this problem, we use a meta-learning process - cost adaptation - which generates the optimization objective for D-RHC to solve based on a set of human-generated priors (cost and constraint functions) and an auxiliary heuristic. We use this adaptive D-RHC method for control of mesh-networked swarm agents. This formulation allows a wide range of tasks to be encoded and can account for network delays, heterogeneous capabilities, and increasingly large swarms through the adaptation mechanism. We leverage the Unity3D game engine to build a simulator capable of introducing artificial networking failures and delays in the swarm. Using the simulator we validate our method on an example coordinated exploration task. We demonstrate that cost adaptation allows for more efficient and safer task completion under varying environment conditions and increasingly large swarm sizes. We release our simulator and code to the community for future work.

Details

AAAI Conference 2018 Conference Paper

Deep Reinforcement Learning That Matters

  • Peter Henderson
  • Riashat Islam
  • Philip Bachman
  • Joelle Pineau
  • Doina Precup
  • David Meger

In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

PDF Details

ICRA Conference 2018 Conference Paper

Eager and Memory-Based Non-Parametric Stochastic Search Methods for Learning Control

  • Victor Barbaros
  • Herke van Hoof
  • Abbas Abdolmaleki
  • David Meger

Direct policy search has shown to be a successful method to optimize robot controller parameters. However, defining a good parametric form for the controller can be challenging for complex problems. Non-parametric methods provide a flexible alternative and are thus a promising tool in robot skill learning. In this paper, we investigate two nonparametric methods based on similar principles but utilizing differing computing schedules: an eager learner and a memory-based learner. We compare the methods experimentally on two different control problems. Furthermore, we define and evaluate a new `hybrid' controller that combines the strong points of both of these methods.

Details

NeurIPS Conference 2018 Conference Paper

Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation

  • Edward Smith
  • Scott Fujimoto
  • David Meger

We consider the problem of scaling deep generative shape models to high-resolution. Drawing motivation from the canonical view representation of objects, we introduce a novel method for the fast up-sampling of 3D objects in voxel space through networks that perform super-resolution on the six orthographic depth projections. This allows us to generate high-resolution objects with more efficient scaling than methods which work directly in 3D. We decompose the problem of 2D depth super-resolution into silhouette and depth prediction to capture both structure and fine detail. This allows our method to generate sharp edges more easily than an individual network. We evaluate our work on multiple experiments concerning high-resolution 3D objects, and show our system is capable of accurately predicting novel objects at resolutions as large as 512x512x512 -- the highest resolution reported for this task. We achieve state-of-the-art performance on 3D object reconstruction from RGB images on the ShapeNet dataset, and further demonstrate the first effective 3D super-resolution method.

PDF Details

AAAI Conference 2018 Conference Paper

OptionGAN: Learning Joint Reward-Policy Options Using Generative Adversarial Inverse Reinforcement Learning

  • Peter Henderson
  • Wei-Di Chang
  • Pierre-Luc Bacon
  • David Meger
  • Joelle Pineau
  • Doina Precup

Reinforcement learning has shown promise in learning policies that can solve complex problems. However, manually specifying a good reward function can be difficult, especially for intricate tasks. Inverse reinforcement learning offers a useful paradigm to learn the underlying reward function directly from expert demonstrations. Yet in reality, the corpus of demonstrations may contain trajectories arising from a diverse set of underlying reward functions rather than a single one. Thus, in inverse reinforcement learning, it is useful to consider such a decomposition. The options framework in reinforcement learning is specifically designed to decompose policies in a similar light. We therefore extend the options framework and propose a method to simultaneously recover reward options in addition to policy options. We leverage adversarial methods to learn joint reward-policy options using only observed expert states. We show that this approach works well in both simple and complex continuous control tasks and shows significant performance increases in one-shot transfer learning.

PDF Details

IROS Conference 2018 Conference Paper

Synthesizing Neural Network Controllers with Probabilistic Model-Based Reinforcement Learning

  • Juan Camilo Gamboa Higuera
  • David Meger
  • Gregory Dudek

We present an algorithm for rapidly learning neural network policies for robotics systems. The algorithm follows the model-based reinforcement learning paradigm and improves upon existing algorithms: PILeO and a sample-based version of PILeo with neural network dynamics (Deep-PILeO). To improve convergence, we propose a model-based algorithm that uses fixed random numbers and clips gradients during optimization. We propose training a neural network dynamics model using variational dropout with truncated Log-Normal noise. These improvements enable data-efficient synthesis of complex neural network policies. We test our approach on a variety of benchmark tasks, demonstrating data-efficiency that is competitive with that of PILeO, while being able to optimize complex neural network controllers. Finally, we assess the performance of the algorithm for learning motor controllers for a six legged autonomous underwater vehicle. This demonstrates the potential of the algorithm for scaling up the dimensionality and dataset sizes, in more complex tasks.

Details

ICRA Conference 2017 Conference Paper

Adapting learned robotics behaviours through policy adjustment

  • Juan Camilo Gamboa Higuera
  • David Meger
  • Gregory Dudek

We present an approach to learning control policies for physical robots that achieves high efficiency by adjusting existing policies that have been learned on similar source systems, such as a similar robot with different physical parameters, or an approximate dynamics model simulator. This can be viewed as calibrating a policy learned on a source system, to match a desired behaviour in similar target systems. Our approach assumes that the trajectories described by the source robot are feasible on the target robot. By making this assumption, we only need to learn a mapping from the source robot state and action spaces to the target robot action space, which we call a policy adjustment model. We demonstrate our approach in simulation in the cart-pole balancing task and a two link double pendulum. We also validate our approach with a physical cart-pole system, where we adjust a learned policy under changes to the weight of the pole.

Details

IROS Conference 2017 Conference Paper

Context-coherent scenes of objects for camera pose estimation

  • Jimmy Li 0001
  • David Meger
  • Gregory Dudek

We propose an approach to vision-based pose estimation using object recognition and identity. Whereas feature based scene recognition and pose estimation methods are well established as effective means for estimating motion and recognizing locations, feature-based methods depend critically on the detection of common local features from one view of a scene to another. We focus on place recognition and pose change estimation in the context of large changes in viewing position, even to the extent that no common surfaces are seen between the two views. Our approach is based on using object identities and their inter-relationship to compute pose change. An important secondary outcome of our method is that it simultaneously infers the 3D poses of objects in the scene that are used as features. Such an object-based approach is inspired by a vast literature on human perception and has the potential for great robustness, albeit at the expense of accuracy. We propose a formulation of the problem using pairwise contextual constraints and develop an efficient algorithmic solution. We validate the approach and quantify its performance using the publicly available TUM SLAM dataset [1].

Details

ICRA Conference 2016 Conference Paper

Learning to generalize 3D spatial relationships

  • Jimmy Li 0001
  • David Meger
  • Gregory Dudek

This paper presents an approach to learn meaningful spatial relationships in an unsupervised fashion from the distribution of 3D object poses in the real world. Our approach begins by extracting an over-complete set of features to describe the relative geometry of two objects. Each relationship type is modeled using a relevance-weighted distance over this feature space. This effectively ignores irrelevant feature dimensions. Our algorithm RANSEM for determining subsets of data that share a relationship as well as the model to describe each relationship is based on robust sample-based clustering. This approach combines the search for consistent groups of data with the extraction of models that precisely capture the geometry of those groups. An iterative refinement scheme has shown to be an effective approach for finding concepts of differing degrees of geometric specificity. Our results show that the models learned by our approach correlate strongly with the English labels that have been given by a human annotator to a set of validation data drawn from the NYUv2 real-world Kinect dataset, demonstrating that these concepts can be automatically acquired given sufficient experience. Additionally, the results of our method significantly out-perform K-means, a standard baseline for unsupervised cluster extraction.

Details

ICRA Conference 2015 Conference Paper

Learning legged swimming gaits from experience

  • David Meger
  • Juan Camilo Gamboa Higuera
  • Anqi Xu 0003
  • Philippe Giguère
  • Gregory Dudek

We present an end-to-end framework for realizing fully automated gait learning for a complex underwater legged robot. Using this framework, we demonstrate that a hexapod flipper-propelled robot can learn task-specific control policies purely from experience data. Our method couples a state-of-the-art policy search technique with a family of periodic low-level controls that are well suited for underwater propulsion. We demonstrate the practical efficacy of tabula rasa learning, that is, learning without the use of any prior knowledge, of policies for a six-legged swimmer to carry out a variety of acrobatic maneuvers in three dimensional space. We also demonstrate informed learning that relies on simulated experience from a realistic simulator. In numerous cases, novel emergent gait behaviors have arisen from learning, such as the use of one stationary flipper to create drag while another oscillates to create thrust. Similar effective results have been demonstrated in under-actuated configurations, where as few as two flippers are used to maneuver the robot to a desired pose, or through an acrobatic motion such as a corkscrew. The success of our learning framework is assessed both in simulation and in the field using an underwater swimming robot.

Details

IROS Conference 2014 Conference Paper

3D trajectory synthesis and control for a legged swimming robot

  • David Meger
  • Florian Shkurti
  • David Cortés Poza
  • Philippe Giguère
  • Gregory Dudek

Inspection and exploration of complex underwater structures requires the development of agile and easy to program platforms. In this paper, we describe a system that enables the deployment of an autonomous underwater vehicle in 3D environments proximal to the ocean bottom. Unlike many previous approaches, our solution: uses oscillating hydrofoil propulsion; allows for stable control of the robot's motion and sensor directions; allows human operators to specify detailed trajectories in a natural fashion; and has been successfully demonstrated as a holistic system in the open ocean near both coral reefs and a sunken cargo ship. A key component of our system is the 3D control of a hexapod swimming robot, which can move the vehicle through agile sequences of orientations despite challenging marine conditions. We present two methods to easily generate robot trajectories appropriate for deployments in close proximity to challenging contours of the sea floor. Both offline recording of trajectories using augmented reality and online placement of fiducial tags in the marine environment are shown to have desirable properties, with complementary strengths and weaknesses. Finally, qualitative and quantitative results of the 3D control system are presented.

Details

ICRA Conference 2013 Conference Paper

Overcoming unknown occlusions in eye-in-hand visual search

  • Sina Radmard
  • David Meger
  • Elizabeth A. Croft
  • James J. Little

We propose a method for handling persistent visual occlusions that disrupt visual tracking for eye-in-hand systems. Our approach allows a robot to “look behind” an occluder and re-acquire its target. To allow efficient planning, we avoid exhaustive mapping of the 3D occluder into configuration space, and instead use informed samples to strike a balance between target search and information gain. A particle filter continuously estimates the target location when it is not visible. Meanwhile, we build a simple but effective map of the occluder's extents to compute potential occlusion-clearing motions using very few calls to efficient approximations of inverse kinematics. Our mixed-initiative cost function balances the goal of directly locating the target with the goal of gaining information through mapping the occluder. Monte-Carlo optimization with efficient data-driven proposals allows us to approximate one-step solutions efficiently. Experimental evaluation performed on a realistic simulator shows that our method can quickly obtain clear views of the target, even when occlusions are persistent and significant camera motion is required.

Details

IROS Conference 2011 Conference Paper

Mobile 3D object detection in clutter

  • David Meger
  • James J. Little

This paper presents a method for multi-view 3D robotic object recognition targeted for cluttered indoor scenes. We explicitly model occlusions that cause failures in visual detectors by learning a generative appearance-occlusion model from a training set containing annotated 3D objects, images and point clouds. A Bayesian 3D object likelihood incorporates visual information from many views as well as geometric priors for object size and position. An iterative, sampling-based inference technique determines object locations based on the model. We also contribute a novel robot-collected data set with images and point clouds from multiple views of 60 scenes, with over 600 manually annotated 3D objects accounting for over ten thousand bounding boxes. This data has been released to the community. Our results show that our system is able to robustly recognize objects in realistic scenes, significantly improving recognition performance in clutter.

Details

ICRA Conference 2010 Conference Paper

Viewpoint detection models for sequential embodied object category recognition

  • David Meger
  • Ankur Gupta 0004
  • James J. Little

This paper proposes a method for learning viewpoint detection models for object categories that facilitate sequential object category recognition and viewpoint planning. We have examined such models for several state-of-the-art object detection methods. Our learning procedure has been evaluated using an exhaustive multiview category database recently collected for multiview category recognition research. Our approach has been evaluated on a simulator that is based on real images that have previously been collected. Simulation results verify that our viewpoint planning approach requires fewer viewpoints for confident recognition. Finally, we illustrate the applicability of our method as a component of a completely autonomous visual recognition platform that has previously been demonstrated in an object category recognition competition.

Details

ICRA Conference 2009 Conference Paper

Inferring a probability distribution function for the pose of a sensor network using a mobile robot

  • David Meger
  • Dimitri Marinakis
  • Ioannis M. Rekleitis
  • Gregory Dudek

In this paper we present an approach for localizing a sensor network augmented with a mobile robot which is capable of providing inter-sensor pose estimates through its odometry measurements. We present a stochastic algorithm that samples efficiently from the probability distribution for the pose of the sensor network by employing Rao-Blackwellization and a proposal scheme which exploits the sequential nature of odometry measurements. Our algorithm automatically tunes itself to the problem instance and includes a principled stopping mechanism based on convergence analysis. We demonstrate the favourable performance of our approach compared to that of established methods via simulations and experiments on hardware.

Details

IROS Conference 2008 Conference Paper

Heuristic search planning to reduce exploration uncertainty

  • David Meger
  • Ioannis M. Rekleitis
  • Gregory Dudek

The path followed by a mobile robot while mapping an environment (i. e. an exploration trajectory) plays a large role in determining the efficiency of the mapping process and the accuracy of any resulting metric map of the environment. This paper examines some important aspects of path planning in this context: the trade-offs between the speed of the exploration process versus the accuracy of resulting maps; and alternating between exploration of new territory and planning through known maps. The resulting motion planning strategy and associated heuristic are targeted to a robot building a map of an environment assisted by a Sensor Network composed of uncalibrated monocular cameras. An adaptive heuristic exploration strategy based on A * search over a combined distance and uncertainty cost function allows for adaptation to the environment and improvement in mapping accuracy. We assess the technique using an illustrative experiment in a real environment and a set of simulations in a parametric family of idealized environments.

Details

ICRA Conference 2008 Conference Paper

Informed visual search: Combining attention and object recognition

  • Per-Erik Forssén
  • David Meger
  • Kevin Lai
  • Scott Helmer
  • James J. Little
  • David G. Lowe

This paper studies the sequential object recognition problem faced by a mobile robot searching for specific objects within a cluttered environment. In contrast to current state-of-the-art object recognition solutions which are evaluated on databases of static images, the system described in this paper employs an active strategy based on identifying potential objects using an attention mechanism and planning to obtain images of these objects from numerous viewpoints. We demonstrate the use of a bag-of-features technique for ranking potential objects, and show that this measure outperforms geometric matching for invariance across viewpoints. Our system implements informed visual search by prioritising map locations and re-examining promising locations first. Experimental results demonstrate that our system is a highly competent object recognition system that is capable of locating numerous challenging objects amongst distractors.

Details
v2026.09.13