John Laird

AAAI Conference 2018 Conference Paper

Interactively Learning a Blend of Goal-Based and Procedural Tasks

• Aaron Mininger
• John Laird

Agents that can learn new tasks through interactive instruction can utilize goal information to search for and learn flexible policies. This approach can be resilient to variations in initial conditions or issues that arise during execution. However, if a task is not easily formulated as achieving a goal or if the agent lacks sufficient domain knowledge for planning, other methods are required. We present a hybrid approach to interactive task learning that can learn both goal-oriented and procedural tasks, and mixtures of the two, from human natural language instruction. We describe this approach, go through two examples of learning tasks, and outline the space of tasks that the system can learn. We show that our approach can learn a variety of goal-oriented and procedural tasks from a single example and is robust to different amounts of domain knowledge.

AAAI Conference 2018 Conference Paper

Learning Fast and Slow: Levels of Learning in General Autonomous Intelligent Agents

• John Laird
• Shiwali Mohan

We propose two distinct levels of learning for general autonomous intelligent agents. Level 1 consists of fixed architectural learning mechanisms that are innate and automatic. Level 2 consists of deliberate learning strategies that are controlled by the agent’s knowledge. We describe these levels and provide an example of their use in a task-learning agent. We also explore other potential levels and discuss the implications of this view of learning for the design of autonomous agents.

IJCAI Conference 2016 Conference Paper

A Demonstration of Interactive Task Learning

• James Kirk
• Aaron Mininger
• John Laird

We will demonstrate a tabletop robotic agent that learns new tasks through interactive natural language instruction. The tasks to be demonstrated are simple puzzles and games, such as Tower of Hanoi, Eight Puzzle, Tic-Tac-Toe, Three Men's Morris, and the Frog and Toads puzzle. We will include a live, interactive simulation of a mobile robot that learns new tasks using the same system.

AAAI Conference 2015 Conference Paper

Spontaneous Retrieval from Long-Term Memory for a Cognitive Architecture

• Justin Li
• John Laird

This paper presents the first functional evaluation of spontaneous, uncued retrieval from long-term memory in a cognitive architecture. The key insight is that current deliberate cued retrieval mechanisms require the agent to have knowledge of when and what to retrieve — knowledge that may be missing or incorrect. Spontaneous uncued retrieval eliminates these requirements through automatic retrievals that use the agent’s problem solving context as a heuristic for relevance, thus supplementing deliberate cued retrieval. Using constraints derived from this insight, we sketch the space of spontaneous retrieval mechanisms and describe an implementation of spontaneous retrieval in Soar together with an agent that takes advantage of that mechanism. Empirical evidence is provided in the Missing Link word-puzzle domain, where agents using spontaneous retrieval out-perform agents without that capability, leading us to conclude that spontaneous retrieval can be a useful mechanism and is worth further exploration.

RLDM Conference 2015 Conference Abstract

The Carli Architecture–Efficient Value Function Specialization for Relational Reinforcement

• Mitchell Bloch
• John Laird

We introduce Carli–a modular architecture supporting efficient value function specialization for relational reinforcement learning. Using a Rete data structure to support efficient relational representations, it implements an initially general hierarchical tile coding and specializes it over time using a fringe. This hierarchical tile coding constitutes a form of linear function approximation in which conjunctions of re- lational features correspond to weights with non-uniform generality. This relational value function lends itself to learning tasks which can be described by a set of relations over objects. These tasks can have vari- able numbers of both features and possible actions over the course of an episode and goals can vary from episode to episode. We demonstrate these characteristics in a version of Blocks World in which the goal configuration changes between episodes. Using relational features, Carli can solve this Blocks World task, while agents using only propositional features cannot generalize from their experience to solve different goal configurations.

AAAI Conference 2014 Conference Paper

Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction

• Shiwali Mohan
• John Laird

Our research aims at building interactive robots and agents that can expand their knowledge by interacting with human users. In this paper, we focus on learning goal-oriented tasks from situated interactive instructions. Learning the structure of novel tasks and how to execute them is a challenging computational problem requiring the agent to acquire a variety of knowledge including goal definitions and hierarchical control information. We frame acquisition of novel tasks as an explanation-based learning (EBL) problem and propose an interactive learning variant of EBL for a robotic agent. We show that our approach can exploit information in situated instructions along with the domain knowledge to demonstrate fast generalization on several tasks. The knowledge acquired transfers across structurally similar tasks. Finally, we show that our approach seamlessly combines agent-driven exploration with instructions for mixed-initiative learning.

AAAI Conference 2013 Conference Paper

Learning Integrated Symbolic and Continuous Action Models for Continuous Domains

• Joseph Xu
• John Laird

Long-living autonomous agents must be able to learn to perform competently in novel environments. One important aspect of competence is the ability to plan, which entails the ability to learn models of the agent’s own actions and their effects on the environment. In this paper we describe an approach to learn action models of environments with continuous-valued spatial states and realistic physics consisting of multiple interacting rigid objects. In such environments, we hypothesize that objects exhibit multiple qualitatively distinct behaviors we call modes, conditioned on their spatial relationships to each other. We argue that action models that explicitly represent these modes using a combination of symbolic spatial relationships and continuous metric information learn faster, generalize better, and make more accurate predictions than models that only use metric information. We present a method to learn action models with piecewise linear modes conditioned on a combination of first order Horn clauses that test symbolic spatial predicates and continuous classifiers. We empirically demonstrate that our method learns more accurate and more general models of a physics simulation than a method that learns a single function (locally weighted regression).

RLDM Conference 2013 Conference Abstract

Online Value Function Improvement

• Mitchell Bloch
• John Laird

Our goal is to develop broadly competent agents that can dynamically construct an appropri- ate value function for tasks with large state spaces so that they can effectively and efficiently learn using reinforcement learning. We study the case where an agent’s state is determined by a small number of con- tinuous dimensions, so that the problem of determining the relevant features corresponds roughly to that of determining the appropriate level of discretization of the continuous values. We adopt hierarchical tile coding, which applies state aggregation at multiple levels of state abstraction simultaneously. Using our for- mulation, it is possible to capture the advantages of learning with state abstractions ranging from general to specific using linear function approximation. We then develop a novel algorithm for incrementally refining the degree of state abstraction, based on cumulative absolute temporal difference error, which produces a sparse non-uniform tile coding. We empirically evaluate our approach in the Puddle World and Mountain Car environments. The results demonstrate that the static and incremental hierarchical tile codings signif- icantly outperform individual tilings and multilevel tile codings (CMACs) for initial learning. Our results also indicate that the incrementally constructed tilings perform nearly as well as the full hierarchical tile coding while requiring an order of magnitude fewer weights.

AAAI Conference 2013 Conference Paper

Preemptive Strategies for Overcoming the Forgetting of Goals

• Justin Li
• John Laird

Maintaining and pursuing multiple goals over varying time scales is an important ability for artificial agents in many cognitive architectures. Goals that remain suspended for long periods, however, are prone to be forgotten. This paper presents a class of preemptive strategies that allow agents to selectively retain goals in memory and to recover forgotten goals. Preemptive strategies work by retrieving and rehearsing goals at triggers, which are either periodic or are predictive of the opportunity to act. Since cognitive architectures contain common hierarchies of memory systems and share similar forgetting mechanisms, these strategies work across multiple architectures. We evaluate their effectiveness in a simulated mobile robot controlled by Soar, and demonstrate how preemptive strategies can be adapted to different environments and agents.

AAAI Conference 2012 Conference Paper

A Multi-Domain Evaluation of Scaling in a General Episodic Memory

• Nate Derbinsky
• Justin Li
• John Laird

Episodic memory endows agents with numerous general cognitive capabilities, such as action modeling and virtual sensing. However, for long lived agents, there are numerous unexplored computational challenges in supporting useful episodic memory functions while maintaining real time reactivity. In this paper, we review the implementation of episodic memory in Soar and present an expansive evaluation of that system. We demonstrate useful applications of episodic memory across a variety of domains, including games, mobile robotics, planning, and linguistics. In these domains, we characterize properties of environments, tasks, and episodic cues that affect performance, and evaluate the ability of Soar’s episodic memory to support hours to days of real time operation.

AAMAS Conference 2012 Conference Paper

Algorithms for Scaling in a General Episodic Memory

• Nate Derbinsky
• Justin Li
• John Laird

Episodic memory endows autonomous agents with useful cognitive capabilities. However, for long-lived agents, there are numerous unexplored computational challenges in supporting useful episodic-memory functions while maintaining real-time reactivity. This paper presents and summarizes the evaluation of an algorithmic variant to the task-independent episodic memory of Soar that expands the class of tasks and cues the mechanism can support while remaining reactive over long agent lifetimes.

AAAI Conference 2012 Conference Paper

Exploring Mixed-Initiative Interaction for Learning with Situated Instruction in Cognitive Agents

• Shiwali Mohan
• John Laird

AAAI Conference 2012 Conference Paper

Functional Interactions Between Memory and Recognition Judgments

• Justin Li
• Nate Derbinsky
• John Laird

One issue facing agents that accumulate large bodies of knowledge is determining whether they have knowledge that is relevant to its current goals. Performing comprehensive searches of long-term memory in every situation can be computationally expensive and disruptive to task reasoning. In this paper, we demonstrate that the recognition judgment — a heuristic for whether memory structures have been previously perceived — can serve as a low-cost indicator of the existence of potentially relevant knowledge. We present an approach for computing both context-dependent and contextindependent recognition judgments using processes and data shared with declarative memories. We then describe an initial, efficient implementation in the Soar cognitive architecture and evaluate our system in a word sense disambiguation task, showing that it reduces the number of memory searches without degrading agent performance.

AAAI Conference 2011 Conference Paper

A Functional Analysis of Historical Memory Retrieval Bias in the Word Sense Disambiguation Task

• Nate Derbinsky
• John Laird

Effective access to knowledge within large declarative memory stores is one challenge in the development and understanding of long-living, generally intelligent agents. We focus on a sub-component of this problem: given a large store of knowledge, how should an agent's task-independent memory mechanism respond to an ambiguous cue, one that pertains to multiple previously encoded memories. A large body of cognitive modeling work suggests that human memory retrievals are biased in part by the recency and frequency of past memory access. In this paper, we evaluate the functional benefit of a set of memory retrieval heuristics that incorporate these biases, in the context of the word sense disambiguation task, in which an agent must identify the most appropriate word meaning in response to an ambiguous linguistic cue. In addition, we develop methods to integrate these retrieval biases within a task-independent declarative memory system implemented in the Soar cognitive architecture and evaluate their effectiveness and efficiency in three commonly used semantic concordances.

AAAI Conference 2011 Conference Paper

Combining Learned Discrete and Continuous Action Models

• Joseph Xu
• John Laird

AAAI Conference 2010 Conference Paper

Instance-Based Online Learning of Deterministic Relational Action Models

• Joseph Xu
• John Laird

We present an instance-based, online method for learning action models in unanticipated, relational domains. Our algorithm memorizes pre- and post-states of transitions an agent encounters while experiencing the environment, and makes predictions by using analogy to map the recorded transitions to novel situations. Our algorithm is implemented in the Soar cognitive architecture, integrating its task-independent episodic memory module and analogical reasoning implemented in procedural memory. We evaluate this algorithm’s prediction performance in a modified version of the blocks world domain and the taxi domain. We also present a reinforcement learning agent that uses our model learning algorithm to significantly speed up its convergence to an optimal policy in the modified blocks world domain.

AAAI Conference 2010 Conference Paper

Relational Reinforcement Learning in Infinite Mario

• Shiwali Mohan
• John Laird

AAAI Conference 1999 Conference Paper

Intelligent Agents in Computer Games

• Michael van Lent
• John Laird
• Josh Buckman
• Joe Hartford
• Steve Houchard
• Kurt Steinkraus
• Russ Tedrake
• University of Michigan

As computer games become more complex and consumers demand more sophisticated computer controlled opponents, game developers are required to place a greater emphasis on the artificial intelligence aspects of their games. Our experience developing intelligent air combat agents for DARPA has suggested a number of areas of AI research that are applicable to computer games. Research in areas such as intelligent agent architectures, knowledge representation, goal-directed behavior and knowledge reusability are all directly relevant to improving the intelligent agents in computer games. The Soar/Games project at the University of Michigan Artificial Intelligence Lab has developed an interface between Soar (Laird, Newell, and Rosenbloom 1987) and the commercial computer games Quake II and Descent 3. Techniques from each of the research areas mentioned above have been used in developing intelligent opponents in these two games.

AAAI Conference 1988 Conference Paper

Recovery from Incorrect Knowledge in Soar

• John Laird

Incorrect knowledge can be a problem for any intelligent system. Soar is a proposal for the underlying architecture that supports intelligence. It has a single representation of long-term memory and a single learning mechanism called chunking. This paper investigates the problem of recovery from incorrect knowledge in Soar. Recovery is problematic in Soar because of the simplicity of chunking: it does not modify existing productions, nor does it analyze the long-term memory during learning. In spite of these limitations, we demonstrate a domain-independent approach to recovery from incorrect control knowledge and present extensions to this approach for recovering from all types of incorrect knowledge. The key idea is to correct decisions instead of long-term knowledge. Soar’ s architecture allows this corrections to occur in parallel with normal processing. This approach does not require any changes to the Soar architecture and because of Soar’ s uniform representations for tasks and knowledge, this approach can be used for all tasks and subtasks in Soar.