Learning the value of time

Although single-shot decision tasks have contributed to our knowledge of value based decision making, they do not address the sequential dependence and temporal aspect of most choices. In this study, we investigate these decisions in a patch-foraging context, which requires subjects to consider future out- comes when deciding how to allocate time between harvesting a depleting resource and searching for a new one. The Marginal Value Theorem (MVT) states that in this class of tasks, the complex problem can be optimally summarized by a threshold rule on the average reward rate or opportunity cost of time. In two experiments, we varied the average reward rate and consistently found that subjects adjusted behavior in the direction predicted by the theory. The MVT threshold rule suggests a simple method for learning the quality of an environment (threshold) by averaging rewards in time and can be contrasted with the predominant incremental learning theory in neuroscience, the Temporal-Difference (TD) algorithm. We examined trial- by-trial decisions and found an effect of recently experienced reward sequences that was better explained by an MVT-based learning model than TD. In a subsequent study, we investigated the suggested role of tonic dopamine (DA) as a signal of average reward rate and opportunity cost of time by looking at the foraging decisions of Parkinson’s disease (PD) patients. In particular, we examined the effect of DA depletion (due to PD) and replacement (due to medication) on the subjective opportunity cost of time implied by the for- aging choices and found that it was consistent with a tonic DA signaled average reward rate. These studies unify work in ecology, economics and computational neuroscience that look at time-sensitive, sequential decisions. We expand upon previous work by suggesting that humans may, in certain contexts, implement a simple threshold-based decision rule on the average reward rate and that this quantity may be signaled by tonic DA.

Authors

• Sara Constantino
• Nathaniel Daw

Keywords

No keywords are indexed for this paper.