Samuel Spaulding

AAMAS Conference 2021 Conference Paper

Towards Transferrable Personalized Student Models in Educational Games

• Samuel Spaulding
• Jocelyn Shen
• Haewon Park
• Cynthia Breazeal

To help facilitate play and learning, game-based educational activities often feature a computational agent as a co-player. Personalizing this agent’s behavior to the student player is an active area of research, and prior work has demonstrated the benefits of personalized educational interaction across a variety of domains. A critical research challenge for personalized educational agents is real-time student modeling. Most student models are designed for and trained on only a single task, which limits the variety, flexibility, and efficiency of student player model learning. In this paper we present a research project applying transfer learning methods to student player models over different educational tasks, studying the effects of an algorithmic “multi-task personalization” approach on the accuracy and data efficiency of student model learning. We describe a unified robotic game system for studying multi-task personalization over two different educational games, each emphasizing early language and literacy skills such as rhyming and spelling. We present a flexible Gaussian Process-based approach for rapidly learning student models from interactive play in each game, and a method for transferring each game’s learned student model to the other via a novel instance-weighting protocol based on task similarity. We present results from a simulation-based investigation of the impact of multi-task personalization, establishing the core viability and benefits of transferrable student models and outlining new questions for future in-person research.

AAAI Conference 2019 Conference Paper

A Model-Free Affective Reinforcement Learning Approach to Personalization of an Autonomous Social Robot Companion for Early Literacy Education

• Hae Won Park
• Ishaan Grover
• Samuel Spaulding
• Louis Gomez
• Cynthia Breazeal

Personalized education technologies capable of delivering adaptive interventions could play an important role in addressing the needs of diverse young learners at a critical time of school readiness. We present an innovative personalized social robot learning companion system that utilizes children’s verbal and nonverbal affective cues to modulate their engagement and maximize their long-term learning gains. We propose an affective reinforcement learning approach to train a personalized policy for each student during an educational activity where a child and a robot tell stories to each other. Using the personalized policy, the robot selects stories that are optimized for each child’s engagement and linguistic skill progression. We recruited 67 bilingual and English language learners between the ages of 4–6 years old to participate in a between-subjects study to evaluate our system. Over a three-month deployment in schools, a unique storytelling policy was trained to deliver a personalized story curriculum for each child in the Personalized group. We compared their engagement and learning outcomes to a Non-personalized group with a fixed curriculum robot, and a baseline group that had no robot intervention. In the Personalization condition, our results show that the affective policy successfully personalized to each child to boost their engagement and outcomes with respect to learning and retaining more target words as well as using more target syntax structures as compared to children in the other groups.

AAMAS Conference 2018 Conference Paper

A Social Robot System for Modeling Children's Word Pronunciation

• Samuel Spaulding
• Huili Chen
• Safinah Ali
• Michael Kulinski
• Cynthia Breazeal

Autonomous educational social robots can be used to help promote literacy skills in young children. Such robots, which emulate the emotive, perceptual, and empathic abilities of human teachers, are capable of replicating some of the benefits of one-on-one tutoring from human teachers, in part by leveraging individual student’s behavior and task performance data to infer sophisticated models of their knowledge. These student models are then used to provide personalized educational experiences by, for example, determining the optimal sequencing of curricular material. In this paper we introduce an integrated system for autonomously analyzing and assessing children’s speech and pronunciation in the context of an interactive word game between a social robot and a child. We present a novel game environment and its computational formulation, an integrated pipeline for capturing and analyzing children’s speech in real-time, and an autonomous robot that models children’s word pronunciation via Gaussian Process Regression (GPR), augmented with an Active Learning protocol that informs the robot’s behavior. We show that the system is capable of autonomously assessing children’s pronunciation ability, with ground truth determined by a post-experiment evaluation by human raters. We also compare phoneme- and word-level GPR models and discuss trade-offs of each approach in modeling children’s pronunciation. Finally, we describe and analyze a pipeline for automatic analysis of children’s speech and pronunciation, including an evaluation of SpeechAce as a tool for future development of autonomous, speech-based language tutors.

AAMAS Conference 2018 Conference Paper

Personalized Robot Tutors that Learn from Multimodal Data

• Samuel Spaulding

As the cost of sensors decreases and ability to model and learn from multi-modal data increases, researchers are exploring how to use the unique qualities of physically embodied robots to help engage students and promote learning. These robots are designed to emulate the emotive, perceptual, and empathic abilities of human teachers, and are capable of replicating some of the benefits of one-on-one tutoring from human teachers. My thesis research focuses on developing methods for robots to analyze and integrate multimodal data including speech, facial expressions, and task performance to build rich models of the user’s knowledge and preferences. These student models are then used to provide personalized educational experiences, such as optimal curricular sequencing, or leaning preferences for educational style. In this abstract, we summarize past projects in this area and discuss applications such as learning from affective signals and model transfer across tasks.

AAMAS Conference 2016 Conference Paper

Affect-Aware Student Models for Robot Tutors

• Samuel Spaulding
• Goren Gordon
• Cynthia Breazeal

Computational tutoring systems, such as educational software or interactive robots, have the potential for great societal benefit. Such systems track and assess students’ knowledge via inferential methods, such as the popular Bayesian Knowledge Tracing (BKT) algorithm. However, these methods do not typically draw on the affective signals that human teachers use to assess knowledge, such as indications of discomfort, engagement, or frustration. In this paper we present a novel extension to the BKT model that uses affective data, derived autonomously from video records of children playing an interactive story-telling game with a robot, to infer student knowledge of reading skills. We find that, compared to a control group of children who played the game with only a tablet, children who interacted with an embodied social robot generated stronger affective data signals of engagement and enjoyment during the interaction. We then show that incorporating this affective data into model training improves the quality of the learned knowledge inference models. These results suggest that physically embodied, affect-aware robot tutors can provide more effective and empathic educational experiences for children, and advance both algorithmic and humancentered motivations for further development of systems that tightly integrate affect understanding and complex models of inference with interactive, educational robots.

AAAI Conference 2016 Conference Paper

Affective Personalization of a Social Robot Tutor for Children’s Second Language Skills

• Goren Gordon
• Samuel Spaulding
• Jacqueline Kory Westlund
• Jin Joo Lee
• Luke Plummer
• Marayna Martinez
• Madhurima Das
• Cynthia Breazeal

Though substantial research has been dedicated towards using technology to improve education, no current methods are as effective as one-on-one tutoring. A critical, though relatively understudied, aspect of effective tutoring is modulating the student’s affective state throughout the tutoring session in order to maximize long-term learning gains. We developed an integrated experimental paradigm in which children play a second-language learning game on a tablet, in collaboration with a fully autonomous social robotic learning companion. As part of the system, we measured children’s valence and engagement via an automatic facial expression analysis system. These signals were combined into a reward signal that fed into the robot’s affective reinforcement learning algorithm. Over several sessions, the robot played the game and personalized its motivational strategies (using verbal and non-verbal actions) to each student. We evaluated this system with 34 children in preschool classrooms for a duration of two months. We saw that (1) children learned new words from the repeated tutoring sessions, (2) the affective policy personalized to students over the duration of the study, and (3) students who interacted with a robot that personalized its affective feedback strategy showed a significant increase in valence, as compared to students who interacted with a nonpersonalizing robot. This integrated system of tablet-based educational content, affective sensing, affective policy learning, and an autonomous social robot holds great promise for a more comprehensive approach to personalized tutoring.

AAMAS Conference 2016 Conference Paper

Learning From the Wizard: Programming Social Interaction through Teleoperated Demonstrations (Extended Abstract)

• W. Bradley Knox
• Samuel Spaulding
• Cynthia Breazeal