Vishal Soni

IJCAI Conference 2007 Conference Paper

• David Wingate
• Vishal Soni
• Britton Wolfe
• Satinder Singh

Most work on Predictive Representations of State (PSRs) has focused on learning and planning in unstructured domains (for example, those represented by flat POMDPs). This paper extends PSRs to represent relational knowledge about domains, so that they can use policies that generalize across different tasks, capture knowledge that ignores irrelevant attributes of objects, and represent policies in a way that is independent of the size of the state space. Using a blocks world domain, we show how generalized predictions about the future can compactly capture relations between objects, which in turn can be used to naturally specify relational-style options and policies. Because our representation is expressed solely in terms of actions and observations, it has extensive semantics which are statistics about observable quantities.

AAAI Conference 2007 Conference Paper

Abstraction in Predictive State Representations

• Vishal Soni

Most work on Predictive Representations of State (PSRs) focuses on learning a complete model of the system that can be used to answer any question about the future. However, we may be interested only in answering certain kinds of abstract questions. For instance, we may only care about the presence of objects in an image rather than pixel level details. In such cases, we may be able to learn substantially smaller models that answer only such abstract questions. We present the framework of PSR homomorphisms for model abstraction in PSRs. A homomorphism transforms a given PSR into a smaller PSR that provides exact answers to abstract questions in the original PSR. As we shall show, this transformation captures structural and temporal abstractions in the original PSR.

AAMAS Conference 2007 Conference Paper

Constraint Satisfaction Algorithms for Graphical Games

• Vishal Soni
• Satinder Singh
• Michael P. Wellman

We formulate the problem of computing equilibria in multiplayer games represented by arbitrary undirected graphs as a constraint satisfaction problem and present two algorithms. The first is PureProp: an algorithm for computing approximate Nash equilibria in complete information one-shot games and approximate Bayes-Nash equilibria in one-shot games of incomplete information. PureProp unifies existing message-passing based algorithms for solving these classes of games. We also address repeated graphical games, and present a second algorithm, PureProp-R, for computing approximate Nash equilibria in these games.

AAAI Conference 2006 Conference Paper

Using Homomorphisms to Transfer Options across Continuous Reinforcement Learning Domains

• Vishal Soni

We examine the problem of Transfer in Reinforcement Learning and present a method to utilize knowledge acquired in one Markov Decision Process (MDP) to bootstrap learning in a more complex but related MDP. We build on work in model minimization in Reinforcement Learning to define relationships between state-action pairs of the two MDPs. Our main contribution in this work is to provide a way to compactly represent such mappings using relationships between state variables in the two domains. We use these functions to transfer a learned policy in the first domain into an option in the new domain, and apply intra-option learning methods to bootstrap learning in the new domain. We first evaluate our approach in the well known Blocksworld domain. We then demonstrate that our approach to transfer is viable in a complex domain with a continuous state space by evaluating it in the Robosoccer Keepaway domain.

ICAPS Conference 2004 Conference Paper

Distributed Feedback Control for Decision Making on Supply Chains

• Christopher Kiekintveld
• Michael P. Wellman
• Satinder Singh 0001
• Joshua Estelle
• Yevgeniy Vorobeychik
• Vishal Soni
• Matthew R. Rudary

Decision makers on supply chains face an uncertain, dynamic, and strategic multiagent environment. We report on Deep Maize, an agent we designed to participate in the 2003 Trading Agent Competition Supply Chain Management (TAC/SCM) game. Our design employs an idealized equilibrium analysis of the SCM game to factor out the strategic aspects of the environment and to define an expected profitable zone of operation. Deep Maize applies distributed feedback control to coordinate its separate functional modules and maintain its environment in the desired zone, despite the uncertainty and dynamism. We evaluate our design with results from the TAC/SCM tournament as well as from controlled experiments conducted after the competition.