Author name cluster

John Winder

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

12 papers

2 author rows

NeurIPS Conference 2024 Conference Paper

Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance

Joshua McClellan
Naveed Haghani
John Winder
Furong Huang
Pratap Tokekar

Multi-Agent Reinforcement Learning (MARL) struggles with sample inefficiency and poor generalization [1]. These challenges are partially due to a lack of structure or inductive bias in the neural networks typically used in learning the policy. One such form of structure that is commonly observed in multi-agent scenarios is symmetry. The field of Geometric Deep Learning has developed Equivariant Graph Neural Networks (EGNN) that are equivariant (or symmetric) to rotations, translations, and reflections of nodes. Incorporating equivariance has been shown to improve learning efficiency and decrease error [ 2 ]. In this paper, we demonstrate that EGNNs improve the sample efficiency and generalization in MARL. However, we also show that a naive application of EGNNs to MARL results in poor early exploration due to a bias in the EGNN structure. To mitigate this bias, we present Exploration-enhanced Equivariant Graph Neural Networks or E2GN2. We compare E2GN2 to other common function approximators using common MARL benchmarks MPE and SMACv2. E2GN2 demonstrates a significant improvement in sample efficiency, greater final reward convergence, and a 2x-5x gain in over standard GNNs in our generalization tests. These results pave the way for more reliable and effective solutions in complex multi-agent systems.

PDF Details DOI

NeurIPS Conference 2021 Conference Paper

A Spoken Language Dataset of Descriptions for Speech-Based Grounded Language Learning

Gaoussou Kebe
Padraig Higgins
Patrick Jenkins
Kasra Darvish
Rishabh Sachdeva
Ryan Barron
John Winder
Donald Engel

Grounded language acquisition is a major area of research combining aspects of natural language processing, computer vision, and signal processing, compounded by domain issues requiring sample efficiency and other deployment constraints. In this work, we present a multimodal dataset of RGB+depth objects with spoken as well as textual descriptions. We analyze the differences between the two types of descriptive language and our experiments demonstrate that the different modalities affect learning. This will enable researchers studying the intersection of robotics, NLP, and HCI to better investigate how the multiple modalities of image, depth, text, speech, and transcription interact, as well as how differences in the vernacular of these modalities impact results.

PDF Details

AAAI Conference 2020 Conference Paper

Planning with Abstract Learned Models While Learning Transferable Subtasks

John Winder
Stephanie Milani
Matthew Landen
Erebus Oh
Shane Parr
Shawn Squire
Marie desJardins
Cynthia Matuszek

We introduce an algorithm for model-based hierarchical reinforcement learning to acquire self-contained transition and reward models suitable for probabilistic planning at multiple levels of abstraction. We call this framework Planning with Abstract Learned Models (PALM). By representing subtasks symbolically using a new formal structure, the lifted abstract Markov decision process (L-AMDP), PALM learns models that are independent and modular. Through our experiments, we show how PALM integrates planning and execution, facilitating a rapid and efﬁcient learning of abstract, hierarchical models. We also demonstrate the increased potential for learned models to be transferred to new and related tasks.

PDF Details

AIIM Journal 2019 Journal Article

Segmentation of breast MR images using a generalised 2D mathematical model with inflation and deflation forces of active contours

Andrik Rampun
Bryan W. Scotney
Philip J. Morrow
Hui Wang
John Winder

In medical computer aided diagnosis systems, image segmentation is one of the major pre-processing steps used to ensure only the region of interest, such as the breast region, will be processed in subsequent steps. Nevertheless, breast segmentation is a difficult task due to low contrast and inhomogeneity, especially when estimating the chest wall in magnetic resonance (MR) images. In fact, the chest wall comprises fat, skin, muscles, and the thoracic skeleton, which can misguide automatic methods when attempting to estimate its location. The objective of the study is to develop a fully automated method for breast and pectoral muscle boundary estimation in MR images. Firstly, we develop a 2D breast mathematical model based on 30 MRI slices (from a patient) and identify important landmarks to obtain a model for the general shape of the breast in an axial plane. Subsequently, we use Otsu's thresholding approach and Canny edge detection to estimate the breast boundary. The active contour method is then employed using both inflation and deflation forces to estimate the pectoral muscle boundary by taking account of information obtained from the proposed 2D model. Finally, the estimated boundary is smoothed using a median filter to remove outliers. Our two datasets contain 60 patients in total and the proposed method is evaluated based on 59 patients (one patient is used to develop the 2D breast model). On the first dataset (9 patients) the proposed method achieved Jaccard = 81. 1% ±6. 1 % and dice coefficient= 89. 4% ±4. 1 % and on the second dataset (50 patients) Jaccard = 84. 9% ±5. 8 % and dice coefficient = 92. 3% ±3. 6 %. These results are qualitatively comparable with the existing methods in the literature.

Details DOI

IJCAI Conference 2019 Conference Paper

The Expected-Length Model of Options

David Abel
John Winder
Marie desJardins
Michael Littman

Effective options can make reinforcement learning easier by enhancing an agent's ability to both explore in a targeted manner and plan further into the future. However, learning an appropriate model of an option's dynamics in hard, requiring estimating a highly parameterized probability distribution. This paper introduces and motivates the Expected-Length Model (ELM) for options, an alternate model for transition dynamics. We prove ELM is a (biased) estimator of the traditional Multi-Time Model (MTM), but provide a non-vacuous bound on their deviation. We further prove that, in stochastic shortest path problems, ELM induces a value function that is sufficiently similar to the one induced by MTM, and is thus capable of supporting near-optimal behavior. We explore the practical utility of this option model experimentally, finding consistent support for the thesis that ELM is a suitable replacement for MTM. In some cases, we find ELM leads to more sample efficient learning, especially when options are arranged in a hierarchy.

PDF Details

TAAS Journal 2017 Journal Article

Feature Construction for Controlling Swarms by Visual Demonstration

Karan K. Budhraja
John Winder
Tim Oates

Agent-based modeling is a paradigm of modeling dynamic systems of interacting agents that are individually governed by specified behavioral rules. Training a model of such agents to produce an emergent behavior by specification of the emergent (as opposed to agent) behavior is easier from a demonstration perspective. While many approaches involve manual behavior specification via code or reliance on a defined taxonomy of possible behaviors, the meta-modeling framework in Miner [2010] generates mapping functions between agent-level parameters and swarm-level parameters, which are re-usable once generated. This work builds on that framework by integrating demonstration by image or video. The demonstrator specifies spatial motion of the agents over time and retrieves agent-level parameters required to execute that motion. The framework, at its core, uses computationally cheap image-processing algorithms. Our work is tested with a combination of primitive visual feature extraction methods (contour area and shape) and features generated using a pre-trained deep neural network in different stages of image featurization. The framework is also evaluated for its potential using complex visual features for all image featurization stages. Experimental results show significant coherence between demonstrated behavior and predicted behavior based on estimated agent-level parameters specific to the spatial arrangement of agents.

Details DOI

AIIM Journal 2017 Journal Article

Fully automated breast boundary and pectoral muscle segmentation in mammograms

Andrik Rampun
Philip J. Morrow
Bryan W. Scotney
John Winder

Breast and pectoral muscle segmentation is an essential pre-processing step for the subsequent processes in computer aided diagnosis (CAD) systems. Estimating the breast and pectoral boundaries is a difficult task especially in mammograms due to artifacts, homogeneity between the pectoral and breast regions, and low contrast along the skin-air boundary. In this paper, a breast boundary and pectoral muscle segmentation method in mammograms is proposed. For breast boundary estimation, we determine the initial breast boundary via thresholding and employ Active Contour Models without edges to search for the actual boundary. A post-processing technique is proposed to correct the overestimated boundary caused by artifacts. The pectoral muscle boundary is estimated using Canny edge detection and a pre-processing technique is proposed to remove noisy edges. Subsequently, we identify five edge features to find the edge that has the highest probability of being the initial pectoral contour and search for the actual boundary via contour growing. The segmentation results for the proposed method are compared with manual segmentations using 322, 208 and 100mammograms from the Mammographic Image Analysis Society (MIAS), INBreast and Breast Cancer Digital Repository (BCDR) databases, respectively. Experimental results show that the breast boundary and pectoral muscle estimation methods achieved dice similarity coefficients of 98. 8% and 97. 8% (MIAS), 98. 9% and 89. 6% (INBreast) and 99. 2% and 91. 9% (BCDR), respectively.

Details DOI

ICAPS Conference 2017 Conference Paper

Planning with Abstract Markov Decision Processes

Nakul Gopalan
Marie desJardins
Michael L. Littman
James MacGlashan
Shawn Squire
Stefanie Tellex
John Winder
Lawson L. S. Wong

Robots acting in human-scale environments must plan under uncertainty in large state–action spaces and face constantly changing reward functions as requirements and goals change. Planning under uncertainty in large state–action spaces requires hierarchical abstraction for efficient computation. We introduce a new hierarchical planning framework called Abstract Markov Decision Processes (AMDPs) that can plan in a fraction of the time needed for complex decision making in ordinary MDPs. AMDPs provide abstract states, actions, and transition dynamics in multiple layers above a base-level “flat” MDP. AMDPs decompose problems into a series of subtasks with both local reward and local transition functions used to create policies for subtasks. The resulting hierarchical planning method is independently optimal at each level of abstraction, and is recursively optimal when the local reward and transition functions are correct. We present empirical results showing significantly improved planning speed, while maintaining solution quality, in the Taxi domain and in a mobile-manipulation robotics problem. Furthermore, our approach allows specification of a decision-making model for a mobile-manipulation problem on a Turtlebot, spanning from low-level control actions operating on continuous variables all the way up through high-level object manipulation tasks.

Details

RLDM Conference 2017 Conference Abstract

Planning with Abstract Markov Decision Processes

Nakul Gopalan
Michael Littman
Shawn Squire
Stefanie Tellex
John Winder
Lawson Wong

Robots acting in human-scale environments must plan under uncertainty in large state–action spaces and face constantly changing reward functions as requirements and goals change. Planning un- der uncertainty in large state–action spaces requires hierarchical abstraction for efficient computation. We (Gopalan et al. 2017 In Press) introduce a new hierarchical planning framework called Abstract Markov Decision Processes (AMDPs) that can plan in a fraction of the time needed for complex decision making in ordinary MDPs. AMDPs provide abstract states, actions, and transition dynamics in multiple layers above a base-level “flat” MDP. AMDPs decompose problems into a series of subtasks with both local reward and local transition functions used to create policies for subtasks. The resulting hierarchical planning method is independently optimal at each level of abstraction, and is recursively optimal when the local reward and transition functions are correct. We present empirical results showing significantly improved planning speed, while maintaining solution quality, in the Taxi domain and in a mobile-manipulation robotics prob- lem. Furthermore, our approach allows specification of a decision-making model for a mobile-manipulation problem on a Turtlebot, spanning from low-level control actions operating on continuous variables all the way up through high-level object manipulation tasks.

PDF Details

RLDM Conference 2017 Conference Abstract

R-AMDP: Model-Based Learning for Abstract Markov Decision Process Hierarchies

Shawn Squire
John Winder
Matthew Landen
Stephanie Milani

Decision-making agents face immensely challenging planning problems when operating in large environments to solve complex tasks. A hierarchy of abstract Markov decision processes (AMDPs) provides a framework for decomposing such problems into distinct, related subtasks. AMDP hierarchies grant con- siderable speedup over related recursively and hierarchically optimal methods such as MAXQ and options. Each AMDP serves as a subgoal, and each is itself a planning problem with a local model and state space abstracted from a ground MDP. Agents are able to plan more efficiently by using a reduced state space at the appropriate level of abstraction; however, they require their subtask models to be specified by a human expert. We describe an approach for automating model estimation by combining the R-Max algorithm with AMDPs. We compare the resulting structures, R-AMDPs, with a similar approach, RMAXQ, and motivate its advantages. Ultimately, R-AMDPs represent the first step in learning AMDP hierarchies dynamically, completely from an agent’s experience.

PDF Details

IJCAI Conference 2016 Conference Paper

A Framework for Anomaly Reasoning: Interpretation through Concept Formation for Knowledge Transfer and Lifelong Learning

John Winder

I am developing a framework for anomaly reasoning for agents that plan and learn in complex, sometimes unfamiliar domains. Anomaly reasoning encompasses recognizing, interpreting, and reacting to unfamiliar objects or familiar objects appearing in unexpected contexts. As a first approach, I propose an interpretation method in which agents form concepts from perceptions to create new representations for use in planning and decision making. An anomaly reasoning framework will make agents more versatile, facilitate learning transfer by pruning irrelevant features, relate new to known phenomena with appropriate similarity metrics, and guide an agent to aspects of the environment most significant to its goals.

PDF Details

IJCAI Conference 2015 Conference Paper

Portable Option Discovery for Automated Learning Transfer in Object-Oriented Markov Decision Processes

Nicholay Topin
Nicholas Haltmeyer
Shawn Squire
John Winder
Marie desJardins
James MacGlashan

We introduce a novel framework for option discovery and learning transfer in complex domains that are represented as object-oriented Markov decision processes (OO-MDPs) [Diuk et al. , 2008]. Our framework, Portable Option Discovery (POD), extends existing option discovery methods, and enables transfer across related but different domains by providing an unsupervised method for finding a mapping between object-oriented domains with different state spaces. The framework also includes heuristic approaches for increasing the efficiency of the mapping process. We present the results of applying POD to Pickett and Barto’s [2002] Policy- Blocks and MacGlashan’s [2013] Option-Based Policy Transfer in two application domains. We show that our approach can discover options effectively, transfer options among different domains, and improve learning performance with low computational overhead.

PDF Details