Alex Pentland

TMLR Journal 2025 Journal Article

Open Problems in Technical AI Governance

• Anka Reuel
• Benjamin Bucknall
• Stephen Casper
• Timothy Fist
• Lisa Soder
• Onni Aarne
• Lewis Hammond
• Lujain Ibrahim

AI progress is creating a growing range of risks and opportunities, but it is often unclear how they should be navigated. In many cases, the barriers and uncertainties faced are at least partly technical. Technical AI governance, referring to technical analysis and tools for supporting the effective governance of AI, seeks to address such challenges. It can help to (a) identify areas where intervention is needed, (b) assess the efficacy of potential governance actions, and (c) enhance governance options by designing mechanisms for enforcement, incentivization, or compliance. In this paper, we explain what technical AI governance is, outline why it is important, and present a taxonomy and incomplete catalog of its open problems. This paper is intended as a resource for technical researchers or research funders looking to contribute to AI governance.

NeurIPS Conference 2025 Conference Paper

ReCAP: Recursive Context-Aware Reasoning and Planning for Large Language Model Agents

• Zhenyu Zhang
• Tianyi Chen
• Weiran Xu
• Alex Pentland
• Jiaxin Pei

Long-horizon tasks requiring multi-step reasoning and dynamic re-planning remain challenging for large language models (LLMs). Sequential prompting methods are prone to context drift, loss of goal information, and recurrent failure cycles, while hierarchical prompting methods often weaken cross-level continuity or incur substantial runtime overhead. We introduce ReCAP (Recursive Context-Aware Reasoning and Planning), a hierarchical framework with shared context for reasoning and planning in LLMs. ReCAP combines three key mechanisms: (i) plan-ahead decomposition, in which the model generates a full subtask list, executes the first item, and refines the remainder; (ii) structured re-injection of parent plans, maintaining consistent multi-level context during recursive return; and (iii) memory-efficient execution, bounding the active prompt so costs scale linearly with task depth. Together these mechanisms align high-level goals with low-level actions, reduce redundant prompting, and preserve coherent context updates across recursion. Experiments demonstrate that ReCAP substantially improves subgoal alignment and success rates on various long-horizon reasoning benchmarks, achieving a 32\% gain on synchronous Robotouille and a 29\% improvement on asynchronous Robotouille under the strict pass@1 protocol.

ICML Conference 2024 Conference Paper

Position: A Safe Harbor for AI Evaluation and Red Teaming

• Shayne Longpre
• Sayash Kapoor
• Kevin Klyman
• Ashwin Ramaswami
• Rishi Bommasani
• Borhane Blili-Hamelin
• Yangsibo Huang
• Aviya Skowron

Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems. However, the terms of service and enforcement strategies used by prominent AI companies to deter model misuse have disincentives on good faith safety evaluations. This causes some researchers to fear that conducting such research or releasing their findings will result in account suspensions or legal reprisal. Although some companies offer researcher access programs, they are an inadequate substitute for independent research access, as they have limited community representation, receive inadequate funding, and lack independence from corporate incentives. We propose that major generative AI developers commit to providing a legal and technical safe harbor, protecting public interest safety research and removing the threat of account suspensions or legal reprisal. These proposals emerged from our collective experience conducting safety, privacy, and trustworthiness research on generative AI systems, where norms and incentives could be better aligned with public interests, without exacerbating model misuse. We believe these commitments are a necessary step towards more inclusive and unimpeded community efforts to tackle the risks of generative AI.

ICML Conference 2024 Conference Paper

Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

• Shayne Longpre
• Robert Mahari
• Naana Obeng-Marnu
• William Brannon
• Tobin South
• Katy Ilonka Gero
• Alex Pentland
• Jad Kabbara

New capabilities in foundation models are owed in large part to massive, widely-sourced, and under-documented training data collections. Existing practices in data collection have led to challenges in tracing authenticity, verifying consent, preserving privacy, addressing representation and bias, respecting copyright, and overall developing ethical and trustworthy foundation models. In response, regulation is emphasizing the need for training data transparency to understand foundation models’ limitations. Based on a large-scale analysis of the foundation model training data landscape and existing solutions, we identify the missing infrastructure to facilitate responsible foundation model development practices. We examine the current shortcomings of common tools for tracing data authenticity, consent, and documentation, and outline how policymakers, developers, and data creators can facilitate responsible foundation model development by adopting universal data provenance standards.

NeurIPS Conference 2021 Conference Paper

One More Step Towards Reality: Cooperative Bandits with Imperfect Communication

• Udari Madhushani
• Abhimanyu Dubey
• Naomi Leonard
• Alex Pentland

The cooperative bandit problem is increasingly becoming relevant due to its applications in large-scale decision-making. However, most research for this problem focuses exclusively on the setting with perfect communication, whereas in most real-world distributed settings, communication is often over stochastic networks, with arbitrary corruptions and delays. In this paper, we study cooperative bandit learning under three typical real-world communication scenarios, namely, (a) message-passing over stochastic time-varying networks, (b) instantaneous reward-sharing over a network with random delays, and (c) message-passing with adversarially corrupted rewards, including byzantine communication. For each of these environments, we propose decentralized algorithms that achieve competitive performance, along with near-optimal guarantees on the incurred group regret as well. Furthermore, in the setting with perfect communication, we present an improved delayed-update algorithm that outperforms the existing state-of-the-art on various network topologies. Finally, we present tight network-dependent minimax lower bounds on the group regret. Our proposed algorithms are straightforward to implement and obtain competitive empirical performance.

ICML Conference 2020 Conference Paper

Cooperative Multi-Agent Bandits with Heavy Tails

• Abhimanyu Dubey
• Alex Pentland

We study the heavy-tailed stochastic bandit problem in the cooperative multi-agent setting, where a group of agents interact with a common bandit problem, while communicating on a network with delays. Existing algorithms for the stochastic bandit in this setting utilize confidence intervals arising from an averaging-based communication protocol known as running consensus, that does not lend itself to robust estimation for heavy-tailed settings. We propose MP-UCB, a decentralized multi-agent algorithm for the cooperative stochastic bandit that incorporates robust estimation with a message-passing protocol. We prove optimal regret bounds for MP-UCB for several problem settings, and also demonstrate its superiority to existing methods. Furthermore, we establish the first lower bounds for the cooperative bandit problem, in addition to providing efficient algorithms for robust bandit estimation of location.

ICML Conference 2020 Conference Paper

Kernel Methods for Cooperative Multi-Agent Contextual Bandits

• Abhimanyu Dubey
• Alex Pentland

Cooperative multi-agent decision making involves a group of agents cooperatively solving learning problems while communicating over a network with delays. In this paper, we consider the kernelised contextual bandit problem, where the reward obtained by an agent is an arbitrary linear function of the contexts’ images in the related reproducing kernel Hilbert space (RKHS), and a group of agents must cooperate to collectively solve their unique decision problems. For this problem, we propose Coop-KernelUCB, an algorithm that provides near-optimal bounds on the per-agent regret, and is both computationally and communicatively efficient. For special cases of the cooperative problem, we also provide variants of Coop-KernelUCB that provides optimal per-agent regret. In addition, our algorithm generalizes several existing results in the multi-agent bandit setting. Finally, on a series of both synthetic and real-world multi-agent network benchmarks, we demonstrate that our algorithm significantly outperforms existing benchmarks.

ICML Conference 2020 Conference Paper

Learning Quadratic Games on Networks

• Yan Leng
• Xiaowen Dong 0001
• Junfeng Wu 0001
• Alex Pentland

Individuals, or organizations, cooperate with or compete against one another in a wide range of practical situations. Such strategic interactions are often modeled as games played on networks, where an individual’s payoff depends not only on her action but also on that of her neighbors. The current literature has largely focused on analyzing the characteristics of network games in the scenario where the structure of the network, which is represented by a graph, is known beforehand. It is often the case, however, that the actions of the players are readily observable while the underlying interaction network remains hidden. In this paper, we propose two novel frameworks for learning, from the observations on individual actions, network games with linear-quadratic payoffs, and in particular, the structure of the interaction network. Our frameworks are based on the Nash equilibrium of such games and involve solving a joint optimization problem for the graph structure and the individual marginal benefits. Both synthetic and real-world experiments demonstrate the effectiveness of the proposed frameworks, which have theoretical as well as practical implications for understanding strategic interactions in a network environment.

ICRA Conference 2019 Conference Paper

Urban Swarms: A new approach for autonomous waste management

• Antonio Luca Alfeo
• Eduardo Castelló Ferrer
• Yago Lizarribar Carrillo
• Arnaud Grignard
• Luis Alonso Pastor
• Dylan T. Sleeper
• Mario G. C. A. Cimino
• Bruno Lepri

Modern cities are growing ecosystems that face new challenges due to the increasing population demands. One of the many problems they face nowadays is waste management, which has become a pressing issue requiring new solutions. Swarm robotics systems have been attracting an increasing amount of attention in the past years and they are expected to become one of the main driving factors for innovation in the field of robotics. The research presented in this paper explores the feasibility of a swarm robotics system in an urban environment. By using bio-inspired foraging methods such as multi-place foraging and stigmergy-based navigation, a swarm of robots is able to improve the efficiency and autonomy of the urban waste management system in a realistic scenario. To achieve this, a diverse set of simulation experiments was conducted using real-world GIS data and implementing different garbage collection scenarios driven by robot swarms. Results presented in this research show that the proposed system outperforms current approaches. Moreover, results not only show the efficiency of our solution, but also give insights about how to design and customize these systems.

AAAI Conference 2016 Conference Paper

Modeling Human Ad Hoc Coordination

• Peter Krafft
• Chris Baker
• Alex Pentland
• Joshua Tenenbaum

UAI Conference 2012 Conference Paper

Graph-Coupled HMMs for Modeling the Spread of Infection

• Wen Dong 0001
• Alex Pentland
• Katherine A. Heller

We develop Graph-Coupled Hidden Markov Models (GCHMMs) for modeling the spread of infectious disease locally within a social network. Unlike most previous research in epidemiology, which typically models the spread of infection at the level of entire populations, we successfully leverage mobile phone data collected from 84 people over an extended period of time to model the spread of infection on an individual level. Our model, the GCHMM, is an extension of widely-used Coupled Hidden Markov Models (CHMMs), which allow dependencies between state transitions across multiple Hidden Markov Models (HMMs), to situations in which those dependencies are captured through the structure of a graph, or to social networks that may change over time. The benefit of making infection predictions on an individual level is enormous, as it allows people to receive more personalized and relevant health advice.

AAAI Conference 2011 Conference Paper

Composite Social Network for Predicting Mobile Apps Installation

• Wei Pan
• Nadav Aharony
• Alex Pentland

We have carefully instrumented a large portion of the population living in a university graduate dormitory by giving participants Android smart phones running our sensing software. In this paper, we propose the novel problem of predicting mobile application (known as “apps”) installation using social networks and explain its challenge. Modern smart phones, like the ones used in our study, are able to collect different social networks using built-in sensors. (e. g. Bluetooth proximity network, call log network, etc) While this information is accessible to app market makers such as the iPhone AppStore, it has not yet been studied how app market makers can use these information for marketing research and strategy development. We develop a simple computational model to better predict app installation by using a composite network computed from the different networks sensed by phones. Our model also captures individual variance and exogenous factors in app adoption. We show the importance of considering all these factors in predicting app installations, and we observe the surprising result that app installation is indeed predictable. We also show that our model achieves the best results compared with generic approaches.

ICML Conference 2001 Conference Paper

Expectation Maximization for Weakly Labeled Data

• Yuri A. Ivanov
• Bruce Blumberg
• Alex Pentland

NeurIPS Conference 2000 Conference Paper

On Reversing Jensen's Inequality

• Tony Jebara
• Alex Pentland

Jensen's inequality is a powerful mathematical tool and one of the workhorses in statistical learning. Its applications therein include the EM algorithm, Bayesian estimation and Bayesian inference. Jensen com(cid: 173) putes simple lower bounds on otherwise intractable quantities such as products of sums and latent log-likelihoods. This simplification then per(cid: 173) mits operations like integration and maximization. Quite often (i. e. in discriminative learning) upper bounds are needed as well. We derive and prove an efficient analytic inequality that provides such variational upper bounds. This inequality holds for latent variable mixtures of exponential family distributions and thus spans a wide range of contemporary statis(cid: 173) tical models. We also discuss applications of the upper bounds including maximum conditional likelihood, large margin discriminative models and conditional Bayesian inference. Convergence, efficiency and prediction results are shown. 1

NeurIPS Conference 1998 Conference Paper

Bayesian Modeling of Facial Similarity

• Baback Moghaddam
• Tony Jebara
• Alex Pentland

In previous work [6, 9, 10], we advanced a new technique for direct visual matching of images for the purposes of face recognition and image retrieval, using a probabilistic measure of similarity based primarily on a Bayesian (MAP) analysis of image differ(cid: 173) ences, leading to a "dual" basis similar to eigenfaces [13]. The performance advantage of this probabilistic matching technique over standard Euclidean nearest-neighbor eigenface matching was recently demonstrated using results from DARPA's 1996 "FERET" face recognition competition, in which this probabilistic matching algorithm was found to be the top performer. We have further developed a simple method of replacing the costly com put ion of nonlinear (online) Bayesian similarity measures by the relatively inexpensive computation of linear (offline) subspace projections and simple (online) Euclidean norms, thus resulting in a significant computational speed-up for implementation with very large image databases as typically encountered in real-world applications.

NeurIPS Conference 1998 Conference Paper

Graphical Models for Recognizing Human Interactions

• Nuria Oliver
• Barbara Rosario
• Alex Pentland

We describe a real-time computer vision and machine learning sys(cid: 173) tem for modeling and recognizing human actions and interactions. Two different domains are explored: recognition of two-handed motions in the martial art 'Tai Chi', and multiple- person interac(cid: 173) tions in a visual surveillance task. Our system combines top-down with bottom-up information using a feedback loop, and is formu(cid: 173) lated with a Bayesian framework. Two different graphical models (HMMs and Coupled HMMs) are used for modeling both individual actions and multiple-agent interactions, and CHMMs are shown to work more efficiently and accurately for a given amount of train(cid: 173) ing. Finally, to overcome the limited amounts of training data, we demonstrate that 'synthetic agents' (Alife-style agents) can be used to develop flexible prior models of the person-to-person inter(cid: 173) actions.

NeurIPS Conference 1998 Conference Paper

Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm

• Tony Jebara
• Alex Pentland

We present the CEM (Conditional Expectation Maximi: :ation) al(cid: 173) gorithm as an extension of the EM (Expectation M aximi: :ation) algorithm to conditional density estimation under missing data. A bounding and maximization process is given to specifically optimize conditional likelihood instead of the usual joint likelihood. We ap(cid: 173) ply the method to conditioned mixture models and use bounding techniques to derive the model's update rules. Monotonic conver(cid: 173) gence, computational efficiency and regression results superior to EM are demonstrated.

NeurIPS Conference 1995 Conference Paper

Active Gesture Recognition using Learned Visual Attention

• Trevor Darrell
• Alex Pentland

We have developed a foveated gesture recognition system that runs in an unconstrained office environment with an active camera. Us(cid: 173) ing vision routines previously implemented for an interactive envi(cid: 173) ronment, we determine the spatial location of salient body parts of a user and guide an active camera to obtain images of gestures or expressions. A hidden-state reinforcement learning paradigm is used to implement visual attention. The attention module selects targets to foveate based on the goal of successful recognition, and uses a new multiple-model Q-Iearning formulation. Given a set of target and distractor gestures, our system can learn where to foveate to maximally discriminate a particular gesture.

NeurIPS Conference 1994 Conference Paper

Correlation and Interpolation Networks for Real-time Expression Analysis/Synthesis

• Trevor Darrell
• Irfan Essa
• Alex Pentland

We describe a framework for real-time tracking of facial expressions that uses neurally-inspired correlation and interpolation methods. A distributed view-based representation is used to characterize facial state, and is computed using a replicated correlation network. The ensemble response of the set of view correlation scores is input to a network based interpolation method, which maps perceptual state to motor control states for a simulated 3-D face model. Activation levels of the motor state correspond to muscle activations in an anatomically derived model. By integrating fast and robust 2-D processing with 3-D models, we obtain a system that is able to quickly track and interpret complex facial motions in real-time.

NeurIPS Conference 1993 Conference Paper

Classifying Hand Gestures with a View-Based Distributed Representation

• Trevor Darrell
• Alex Pentland

We present a method for learning, tracking, and recognizing human hand gestures recorded by a conventional CCD camera without any special gloves or other sensors. A view-based representation is used to model aspects of the hand relevant to the trained gestures, and is found using an unsupervised clustering technique. We use normalized correlation net(cid: 173) works, with dynamic time warping in the temporal domain, as a distance function for unsupervised clustering. Views are computed separably for space and time dimensions; the distributed response of the combination of these units characterizes the input data with a low dimensional repre(cid: 173) sentation. A supervised classification stage uses labeled outputs of the spatio-temporal units as training data. Our system can correctly classify gestures in real time with a low-cost image processing accelerator.

NeurIPS Conference 1991 Conference Paper

Against Edges: Function Approximation with Multiple Support Maps

• Trevor Darrell
• Alex Pentland

Networks for reconstructing a sparse or noisy function often use an edge field to segment the function into homogeneous regions, This approach assumes that these regions do not overlap or have disjoint parts, which is often false. For example, images which contain regions split by an occlud(cid: 173) ing object can't be properly reconstructed using this type of network. We have developed a network that overcomes these limitations, using support maps to represent the segmentation of a signal. In our approach, the sup(cid: 173) port of each region in the signal is explicitly represented. Results from an initial implementation demonstrate that this method can reconstruct images and motion sequences which contain complicated occlusion.

IJCAI Conference 1989 Conference Paper

Perception of Non-Rigid Motion: Inference of Shape, Material and Force

• Alex Pentland
• John Williams

AAAI Conference 1988 Conference Paper

On the Extraction of Shape Information from Shading

• Alex Pentland

ECAI Conference 1982 Conference Paper

Local Computation of Shape

• Alex Pentland