Discovering agents

Causal models of agents have been used to analyse the safety aspects of machine learning systems. But identifying agents is non-trivial – often the causal model is just assumed by the modeller without much justification – and modelling failures can lead to mistakes in the safety analysis. This paper proposes the first formal causal definition of agents – roughly that agents are systems that would adapt their policy if their actions influenced the world in a different way. From this we derive the first causal discovery algorithm for discovering the presence of agents from empirical data, given a set of variables and under certain assumptions. We also provide algorithms for translating between causal models and game-theoretic influence diagrams. We demonstrate our approach by resolving some previous confusions caused by incorrect causal modelling of agents.

Authors

• Zachary Kenton DeepMind, United Kingdom of Great Britain and Northern Ireland
• Ramana Kumar DeepMind, United Kingdom of Great Britain and Northern Ireland
• Sebastian Farquhar DeepMind, United Kingdom of Great Britain and Northern Ireland
• Jonathan Richens DeepMind, United Kingdom of Great Britain and Northern Ireland
• Matt MacDermott Imperial College London, United Kingdom of Great Britain and Northern Ireland
• Tom Everitt DeepMind, United Kingdom of Great Britain and Northern Ireland

Keywords

• Artificial intelligence
• Game theory
• Causality