Michael R. James

YNIMG Journal 2011 Journal Article

BDNF gene effects on brain circuitry replicated in 455 twins

• Ming-Chang Chiang
• Marina Barysheva
• Arthur W. Toga
• Sarah E. Medland
• Narelle K. Hansell
• Michael R. James
• Katie L. McMahon
• Greig I. de Zubicaray

Brain-derived neurotrophic factor (BDNF) plays a key role in learning and memory, but its effects on the fiber architecture of the living brain are unknown. We genotyped 455 healthy adult twins and their non-twin siblings (188 males/267 females; age: 23. 7±2. 1years, mean±SD) and scanned them with high angular resolution diffusion tensor imaging (DTI), to assess how the BDNF Val66Met polymorphism affects white matter microstructure. By applying genetic association analysis to every 3D point in the brain images, we found that the Val-BDNF genetic variant was associated with lower white matter integrity in the splenium of the corpus callosum, left optic radiation, inferior fronto-occipital fasciculus, and superior corona radiata. Normal BDNF variation influenced the association between subjects' performance intellectual ability (as measured by Object Assembly subtest) and fiber integrity (as measured by fractional anisotropy; FA) in the callosal splenium, and pons. BDNF gene may affect the intellectual performance by modulating the white matter development. This combination of genetic association analysis and large-scale diffusion imaging directly relates a specific gene to the fiber microstructure of the living brain and to human intelligence.

AAMAS Conference 2009 Conference Paper

SarsaLandmark: An Algorithm for Learning in POMDPs with Landmarks

• Michael R. James
• Satinder Singh

Reinforcement learning algorithms that use eligibility traces, such as Sarsa(λ), have been empirically shown to be effective in learning good estimated-state-based policies in partially observable Markov decision processes (POMDPs). Nevertheless, one can construct counterexamples, problems in which Sarsa(λ < 1 ) fails to find a good policy even though one exists. Despite this, these algorithms remain of great interest because alternative approaches to learning in POMDPs based on approximating belief-states do not scale. In this paper we present SarsaLandmark, an algorithm for learning in POMDPs with ”landmark” states (most man-made and many natural environments have landmarks). SarsaLandmark simultaneously preserves the advantages offered by eligibility traces and fixes the cause of the failure of Sarsa(λ) on the motivating counterexamples. We present a theoretical analysis of SarsaLandmark for the policy evaluation problem and present empirical results on a few learning control problems.

IJCAI Conference 2007 Conference Paper

• Michael R. James
• Michael E. Samples
• Dmitri A. Dolgov

Planning in partially-observable dynamical systems (such as POMDPs and PSRs) is a computationally challenging task. Popular approximation techniques that have proven successful are point-based planning methods including point-based value iteration (PBVI), which works by approximating the solution at a finite set of points. These point-based methods typically are anytime algorithms, whereby an initial solution is obtained using a small set of points, and the solution may be ncrementally improved by including additional points. We introduce a family of anytime PBVI algorithms that use the information present in the current solution for identifying and adding new points that have the potential to best improve the next solution. We motivate and present two different methods for choosing points and evaluate their performance empirically, demonstrating that high-quality solutions can be obtained with significantly fewer points than previous PBVI approaches.

AAMAS Conference 2007 Conference Paper

Combinatorial Resource Scheduling for Multiagent MDPs

• Dmitri A. Dolgov
• Michael R. James
• Michael E. Samples

Optimal resource scheduling in multiagent systems is a computationally challenging task, particularly when the values of resources are not additive. We consider the combinatorial problem of scheduling the usage of multiple resources among agents that operate in stochastic environments, modeled as Markov decision processes (MDPs). In recent years, efficient resource-allocation algorithms have been developed for agents with resource values induced by MDPs. However, this prior work has focused on static resource-allocation problems where resources are distributed once and then utilized in infinite-horizon MDPs. We extend those existing models to the problem of combinatorial resource scheduling, where agents persist only for finite periods between their (predefined) arrival and departure times, requiring resources only for those time periods. We provide a computationally efficient procedure for computing globally optimal resource assignments to agents over time. We illustrate and empirically analyze the method in the context of a stochastic jobscheduling domain.

AAAI Conference 2006 Conference Paper

Improving Approximate Value Iteration Using Memories and Predictive State Representations

• Michael R. James

Planning in partially-observable dynamical systems is a challenging problem, and recent developments in point-based techniques such as Perseus significantly improve performance as compared to exact techniques. In this paper, we show how to apply these techniques to new models for non- Markovian dynamical systems called Predictive State Representations (PSRs) and Memory-PSRs (mPSRs). PSRs and mPSRs are models of non-Markovian decision processes that differ from latent-variable models (e. g. HMMs, POMDPs) by representing state using only observable quantities. Further, mPSRs explicitly represent certain structural properties of the dynamical system that are also relevant to planning. We show how planning techniques can be adapted to leverage this structure to improve performance both in terms of execution time as well as quality of the resulting policy.

IJCAI Conference 2005 Conference Paper

Combining Memory and Landmarks with Predictive State Representations

• Michael R. James
• Britton Wolfe
• Satinder

It has recently been proposed that it is advantageous to have models of dynamical systems be based solely on observable quantities. Predictive state representations (PSRs) are a type of model that uses predictions about future observations to capture the state of a dynamical system. However, PSRs do not use memory of past observations. We propose a model called memory-PSRs that uses both memories of the past, and predictions of the future. We show that the use of memories provides a number of potential advantages. It can reduce the size of the model (in comparison to a PSR model). In addition many dynamical systems have memories that can serve as landmarks that completely determine the current state. The detection and recognition of landmarks is advantageous because they can serve to reset a model that has gotten off-track, as often happens when the model is learned from samples. This paper develops both memory-PSRs and the use and detection of landmarks.

AAAI Conference 2005 Conference Paper

Planning in Models that Combine Memory with Predictive Representations of State

• Michael R. James

Models of dynamical systems based on predictive state representations (PSRs) use predictions of future observations as their representation of state. A main departure from traditional models such as partially observable Markov decision processes (POMDPs) is that the PSR-model state is composed entirely of observable quantities. PSRs have recently been extended to a class of models called memory-PSRs (mP- SRs) that use both memory of past observations and predictions of future observations in their state representation. Thus, mPSRs preserve the PSR-property of the state being composed of observable quantities while potentially revealing structure in the dynamical system that is not exploited in PSRs. In this paper, we demonstrate that the structure captured by mPSRs can be exploited quite naturally for stochastic planning based on value-iteration algorithms. In particular, we adapt the incremental-pruning (IP) algorithm defined for planning in POMDPs to mPSRs. Our empirical results show that our modified IP on mPSRs outperforms, in most cases, IP on both PSRs and POMDPs.