Learning Constraints on Autonomous Behavior from Proactive Feedback

Learning from feedback is a common paradigm to acquire information that is hard to specify a priori. In this work, we consider an agent with a known nominal reward model that captures its high-level task objective. Furthermore, the agent operates subject to constraints that are unknown a priori and must be inferred from human interventions. Unlike existing methods, our approach does not rely on full or partial demonstration trajectories or assume a fully reactive human. Instead, we assume access only to sparse interventions, which may in fact be generated proactively by the human, and we only make minimal assumptions about the human. We provide both theoretical bounds on performance and empirical validations of our method. We show that our method enables an agent to learn a constraint set with high accuracy that generalizes well to new environments within a domain, whereas methods that only consider reactive feedback learn an incorrect constraint set that does not generalize well, making constraint violations more likely in new environments.

Authors

• Connor Basich
• Saaduddin Mahmud
• Shlomo Zilberstein

Keywords

• Behavioral sciences
• Trajectory
• Calibration
• Task analysis
• Intelligent robots
• Learning Constraints
• Human Intervention
• Empirical Validation
• Constraint Violation
• Part Of Trajectory
• Nominal Model
• Learning Models
• Human Model
• Ensemble Model
• Autonomous Vehicles
• Optimal Policy
• Learning Settings
• Reward Function
• Markov Decision Process
• Training Environment
• Temporal Model
• Behavior Of Agents
• Lexicographic
• Proactivity
• Policy Agencies
• Inverse Reinforcement Learning
• State-action Pair
• Binary Signal
• Reactive Agents
• Understanding Of Agency
• Deployment Phase
• Violation Probability
• Test Environment
• Simulation Domain
• Human Operator