Author name cluster

Mark Ho

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

1 author row

NeurIPS Conference 2025 Conference Paper

Estimating cognitive biases with attention-aware inverse planning

Sounak Banerjee
Daphne Cornelisse
Deepak Gopinath
Emily Sumner
Jonathan DeCastro
Guy Rosman
Eugene Vinitsky
Mark Ho

People's goal-directed behaviors are influenced by their cognitive biases, and autonomous systems that interact with people should be aware of this. For example, people's attention to objects in their environment will be biased in a way that systematically affects how they perform everyday tasks such as driving to work. Here, building on recent work in computational cognitive science, we formally articulate the \textit{attention-aware inverse planning problem}, in which the goal is to estimate a person's attentional biases from their actions. We demonstrate how attention-aware inverse planning systematically differs from standard inverse reinforcement learning and how cognitive biases can be inferred from behavior. Finally, we present an approach to attention-aware inverse planning that combines deep reinforcement learning with computational cognitive modeling. We use this approach to infer the attentional strategies of RL agents in real-life driving scenarios selected from the Waymo Open Dataset, demonstrating the scalability of estimating cognitive biases with attention-aware inverse planning.

PDF Details

AAAI Conference 2020 Conference Paper

People Do Not Just Plan,They Plan to Plan

Mark Ho
David Abel
Jonathan Cohen
Michael Littman
Thomas Griffiths

Planning is useful. It lets people take actions that have desirable long-term consequences. But, planning is hard. It requires thinking about consequences, which consumes limited computational and cognitive resources. Thus, people should plan their actions, but they should also be smart about how they deploy resources used for planning their actions. Put another way, people should also “plan their plans”. Here, we formulate this aspect of planning as a meta-reasoning problem and formalize it in terms of a recursive Bellman objective that incorporates both task rewards and information-theoretic planning costs. Our account makes quantitative predictions about how people should plan and meta-plan as a function of the overall structure of a task, which we test in two experiments with human participants. We ﬁnd that people’s reaction times reﬂect a planned use of information processing, consistent with our account. This formulation of planning to plan provides new insight into the function of hierarchical planning, state abstraction, and cognitive control in both humans and machines.

PDF Details

NeurIPS Conference 2019 Conference Paper

On the Utility of Learning about Humans for Human-AI Coordination

Micah Carroll
Rohin Shah
Mark Ho
Tom Griffiths
Sanjit Seshia
Pieter Abbeel
Anca Dragan

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be understood by humans. To demonstrate this, we introduce a simple environment that requires challenging coordination, based on the popular game Overcooked, and learn a simple model that mimics human play. We evaluate the performance of agents trained via self-play and population-based training. These agents perform very well when paired with themselves, but when paired with our human model, they are significantly worse than agents designed to play with the human model. An experiment with a planning algorithm yields the same conclusion, though only when the human-aware planner is given the exact human model that it is playing with. A user study with real humans shows this pattern as well, though less strongly. Qualitatively, we find that the gains come from having the agent adapt to the human's gameplay. Given this result, we suggest several approaches for designing agents that learn about humans in order to better coordinate with them. Code is available at https: //github. com/HumanCompatibleAI/overcooked_ai.

PDF Details

NeurIPS Conference 2018 Conference Paper

Learning Task Specifications from Demonstrations

Marcell Vazquez-Chanlatte
Susmit Jha
Ashish Tiwari
Mark Ho
Sanjit Seshia

Real-world applications often naturally decompose into several sub-tasks. In many settings (e. g. , robotics) demonstrations provide a natural way to specify the sub-tasks. However, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the sub-tasks can be safely recombined or limit the types of composition available. Motivated by this deficit, we consider the problem of inferring Boolean non-Markovian rewards (also known as logical trace properties or specifications) from demonstrations provided by an agent operating in an uncertain, stochastic environment. Crucially, specifications admit well-defined composition rules that are typically easy to interpret. In this paper, we formulate the specification inference task as a maximum a posteriori (MAP) probability inference problem, apply the principle of maximum entropy to derive an analytic demonstration likelihood model and give an efficient approach to search for the most likely specification in a large candidate pool of specifications. In our experiments, we demonstrate how learning specifications can help avoid common problems that often arise due to ad-hoc reward composition.

PDF Details

RLDM Conference 2017 Conference Abstract

Learning to Cooperate and Compete*

Max Kleiman-Weiner
Mark Ho
Joseph Austerweil
Michael Littman
Joshua Tenenbaum

Successfully navigating the social world requires reasoning about both high-level strategic goals, such as whether to cooperate or compete, as well as the low-level actions needed to achieve those goals. While previous work in experimental game theory has examined the former and work on multi-agent sys- tems has examined the latter, there has been little work investigating behavior in environments that require simultaneous planning and inference across both levels. We develop a hierarchical model of social agency that infers the intentions of other agents, strategically decides whether to cooperate or compete, and then executes either a cooperative or competitive planning program. The cooperative planning program formal- izes a type of joint intentionality or team reasoning where agents mesh plans to efficiently cooperate. The competitive planning program is a generalization of iterated best-response. These planning programs en- able both strategic action as well as action interpretation – the ability to infer whether others are intending to cooperate or compete from ambiguous actions. We test predictions of this model in multi-agent behav- ioral experiments using rich video-game like environments. Learning occurs across both high-level strategic decisions and low-level actions leading to the emergence of social norms. These rapidly learned norms coor- dinate cooperation and make it more efficient after just a few interactions. By grounding strategic behavior in a formal model of planning, we develop abstract notions of both cooperation and competition and shed light on the computational nature of joint intentionality.

PDF Details

NeurIPS Conference 2016 Conference Paper

Showing versus doing: Teaching by demonstration

Mark Ho
Michael Littman
James MacGlashan
Fiery Cushman
Joseph Austerweil

People often learn from others' demonstrations, and classic inverse reinforcement learning (IRL) algorithms have brought us closer to realizing this capacity in machines. In contrast, teaching by demonstration has been less well studied computationally. Here, we develop a novel Bayesian model for teaching by demonstration. Stark differences arise when demonstrators are intentionally teaching a task versus simply performing a task. In two experiments, we show that human participants systematically modify their teaching behavior consistent with the predictions of our model. Further, we show that even standard IRL algorithms benefit when learning from behaviors that are intentionally pedagogical. We conclude by discussing IRL algorithms that can take advantage of intentional pedagogy.

PDF Details

RLDM Conference 2015 Conference Abstract

Teaching Behavior with Punishments and Rewards

Mark Ho
Michael Littman
Fiery Cushman
Joseph Austerweil

When teaching a complex behavior that achieves a goal, a teacher could simply deliver rewards once a learner has succeeded. For instance, a coach can train a basketball team by showering players with praise when they win a game and lambasting them otherwise. A more reasonable strategy, however, might be to teach subtasks like dribbling, passing, or shooting the ball. Teaching subtasks through reward or punishment, or shaping, can allow a learner to acquire the overarching task more quickly. Effective shaping also requires coordination between a teacher’s training strategy and a learner’s interpretation of rewards and punishments. Specifically, feedback could be treated as a reward to be maximized, as in standard reinforcement learning (RL), or as a signal for the correctness or incorrectness of an action. If feedback is treated as a pure reward signal, giving rewards for subtasks can introduce problematic positive reward cycles, state-action-feedback sequences that return to an initial state but yield a net positive reward. For example, if a player primarily wants to maximize positive teacher feedback, overly praising them for dribbling may lead them to exploit those rewards and not learn how to play the whole game. In contrast, positive cycles do not pose a problem when teaching a learner who interprets feedback as a signal about the correctness of an action. Here, we examine how humans teach with evaluative feedback and whether they teach consistent with a learner who interprets feedback as a reward signal or a signal of action correctness. We first formalize these two models of feedback interpretation as a reward-maximizing model based on standard RL and an action-feedback model based on Bayesian inference. Then, in two experiments in which people trained virtual learners for isolated actions or while learning over time, we show that people shape in a manner more consistent with the action-feedback model.

PDF Details