Fred Charles

AAMAS Conference 2023 Conference Paper

Communicating Agent Intentions for Human-Agent Decision Making under Uncertainty

• Julie Porteous
• Alan Lindsay
• Fred Charles

Recent advances in visualisation technologies have opened up new possibilities for human-agent communication. For systems where agents use automated planning, visualisation of agent intentions, i. e. , agent planned actions, can assist human understanding and decision making (e. g. , deciding when human control is required or when it can be delegated to an agent). We are working in an application area, shipbuilding, where branched plans are often essential, due to the typical uncertainty experienced. Our focus is how best to communicate, using visualisation, the key information content of branched plans. It is important that such visualisations communicate the complexity and variety of the possible agent intentions i. e. , executions, captured in a branched plan, whilst also connecting to the practitioner’s understanding of the problem. Thus we utilise an approach to generate the complete branched plan, to be able to provide a full picture of its complexity, and a mechanism to select a subset of diverse traces that characterise the possible agent intentions. We have developed an interface which uses 3D visualisation to communicate details of these characterising execution traces. Using this interface, we conducted a study evaluating the impact of different modes of presentation on user understanding. Our results support our expectation that visualisation of branched plan characterising execution traces increases user understanding of agent intention and plan execution possibilities.

AAMAS Conference 2023 Conference Paper

Generating Stylistic and Personalized Dialogues for Virtual Agents in Narratives

• Weilai Xu
• Fred Charles
• Charlie Hargood

Virtual agents interact with each other through dialogues in various types of narratives (e. g. films). In this paper, we propose an approach on the basis of DialoGPT pre-trained language model, which explores the impact of dialogue generation with different levels of agents’ personalities derived from narrative films based on the Big-Five model, as well as with three different embedding methods. From the experimental results using automatic metrics and human user evaluation, we investigate and analyze the impact of different settings on narrative dialogue generation. We demonstrate that our approach is able to generate dialogues with increased variety that correctly reflect the corresponding target personality.

ECAI Conference 2020 Conference Paper

Influence of Personality-Based Features for Dialogue Generation in Computational Narratives

• Weilai Xu
• Fred Charles
• Charlie Hargood
• Feng Tian 0006
• Wen Tang 0004

In this paper, we present an approach for generating dialogues for characters within the context of computational narratives using personality-based features for deep neural networks. The approach integrates the requirements of both narrative genres and personality traits for the definition of character-based stylistic models. The modelling of characters’ features from existing datasets of complete stories permits the generation of personality-rich character dialogues. We present early results from an evaluation based on a sample of characters’ personality traits across different narrative genres, demonstrating variability in the resulting dialogues.

YNIMG Journal 2019 Journal Article

Volitional limbic neuromodulation exerts a beneficial clinical effect on Fibromyalgia

• Noam Goldway
• Jacob Ablin
• Omer Lubin
• Yoav Zamir
• Jackob Nimrod Keynan
• Ayelet Or-Borichev
• Marc Cavazza
• Fred Charles

Volitional neural modulation using neurofeedback has been indicated as a potential treatment for chronic conditions that involve peripheral and central neural dysregulation. Here we utilized neurofeedback in patients suffering from Fibromyalgia - a chronic pain syndrome that involves sleep disturbance and emotion dysregulation. These ancillary symptoms, which have an amplificating effect on pain, are known to be mediated by heightened limbic activity. In order to reliably probe limbic activity in a scalable manner fit for EEG-neurofeedback training, we utilized an Electrical Finger Print (EFP) model of amygdala-BOLD signal (termed Amyg-EFP), that has been successfully validated in our lab in the context of volitional neuromodulation. We anticipated that Amyg-EFP-neurofeedback training aimed at limbic down modulation would improve chronic pain in patients suffering from Fibromyalgia, by reducing sleep disorder improving emotion regulation. We further expected that improved clinical status would correspond with successful training as indicated by improved down modulation of the Amygdala-EFP signal. Thirty-Four Fibromyalgia patients (31F; age 35. 6 ± 11. 82) participated in a randomized placebo-controlled trial with biweekly Amyg-EFP-neurofeedback sessions or sham neurofeedback (n = 9) for a total duration of five consecutive weeks. Following training, participants in the real-neurofeedback group were divided into good (n = 13) or poor (n = 12) modulators according to their success in the neurofeedback training. Before and after treatment, self-reports on pain, depression, anxiety, fatigue and sleep quality were obtained, as well as objective sleep indices. Long-term clinical follow-up was made available, within up to three years of the neurofeedback training completion. REM latency and objective sleep quality index were robustly improved following the treatment course only in the real-neurofeedback group (time × group p < 0. 05) and to a greater extent among good modulators (time × sub-group p < 0. 05). In contrast, self-report measures did not reveal a treatment-specific response at the end of the neurofeedback training. However, the follow-up assessment revealed a delayed improvement in chronic pain and subjective sleep experience, evident only in the real-neurofeedback group (time × group p < 0. 05). Moderation analysis showed that the enduring clinical effects on pain evident in the follow-up assessment were predicted by the immediate improvements following training in objective sleep and subjective affect measures. Our findings suggest that Amyg-EFP-neurofeedback that specifically targets limbic activity down modulation offers a successful principled approach for volitional EEG based neuromodulation treatment in Fibromyalgia patients. Importantly, it seems that via its immediate sleep improving effect, the neurofeedback training induced a delayed reduction in the target subjective symptom of chronic pain, far and beyond the immediate placebo effect. This indirect approach to chronic pain management reflects the substantial link between somatic and affective dysregulation that can be successfully targeted using neurofeedback.

AAMAS Conference 2018 Conference Paper

Benchmark Framework for Virtual Students' Behaviours

• Jean-Luc Lugrin
• Fred Charles
• Michael Habel
• Jamie Matthews
• Henrik Dudaczy
• Sebastian Oberd�rfer
• Alice Wittmann
• Christian Seufert

This paper demonstrates the integration and evaluation of different atmosphere models into Virtual Reality (VR) training for teacher education. We developed three behaviour models to simulate different levels of class discipline. We evaluated their performances using a combination of objective and subjective measurements. Our initial results suggest that the more believable and distinguishable classroom atmospheres are produced by creating more consistent behaviours across virtual students. Our results confirm the importance of similar behaviours to elicit a particular atmosphere.

AAMAS Conference 2017 Conference Paper

An Interactive Narrative Platform for Story Understanding Experiments

• Julie Porteous
• Fred Charles
• Cameron Smith
• Marc Cavazza
• Jolien Mouw
• Paul van den Broek

Interactive Narratives are systems that use automated narrative generation techniques to create multiple story variants which can be shown to an audience, as virtual narratives, using cinematic staging techniques. The focus of previous research has included aspects such as the quality of automatically generated narratives and the way in which audiences respond to them. However in this work we have developed a mechanism for control of interactive narratives that supports their use in experiments to assess story understanding. This is implemented in our demonstration system, which features two parts: an interface that allows high-level specification of criteria for story understanding experiments; and a participant interface in which virtual narratives, conforming to the experimental design, are presented as 3D visualizations. The virtual narrative is based on a pre-existing children’s story, and features a cast of virtual characters.

AAMAS Conference 2017 Conference Paper

Mise-En-Sc&egrave; ne of Narrative Action in Interactive Storytelling

• Jamie Matthews
• Fred Charles
• Julie Porteous
• Alexandra Mendes

In this Interactive Storytelling (IS) work, we provide a framework for automating several aspects of staging the activities of a population of narrative agents and their interactions. Narrative agents can have differing levels of narrative relevance within the situated narrative actions which are visualised within a 3D real-time virtual world. The solution we propose here offers a framework for integrating the use of multiple dynamic regions within the 3D story world defined through a semantic representation that is able to support the staging of narrative actions through the behaviours of the primary and background agents’ involved. This includes both the mechanics of dealing with the narrative discourse level as well as the interaction with the narrative generation layer to account for any dynamic modifications of the 3D story world. We refer to this approach as MISE-en-scène Region support (MISER) and are providing here details of the demonstration application which users will be able to interact with.

AAMAS Conference 2017 Conference Paper

MISER: Mise-En-Sc&egrave; ne Region Support for Staging Narrative Actions in Interactive Storytelling

• Jamie Matthews
• Fred Charles
• Julie Porteous
• Alexandra Mendes

The recent increase in interest in Interactive Storytelling systems, spurred on by the emergence of affordable virtual reality technology, has brought with it a need to address the way in which narrative content is visualized through the complex staging of multiple narrative agents’ behaviors within virtual story worlds. In this work we address the challenge of automating several aspects of staging the activities of a population of narrative agents and their interactions, where agents can have differing levels of narrative relevance within the situated narrative actions. Our solution defines an approach that integrates the use of multiple dynamic regions within a virtual story world, specified via a semantic representation that is able to support the staging of narrative actions through the behaviors of the primary and background agents’ that are involved. This encompasses both the mechanics of dealing with the narrative discourse level as well as the interaction with the narrative generation layer to account for any dynamic modifications of the virtual story world. We refer to this approach as mise-en-scène region (miser) support. In this paper, we describe our approach and its integration as part of a fully implemented Interactive Storytelling system. We illustrate the work through detailed examples of short narrative instantiations. We present the results of our evaluation which clearly demonstrate the potential of the miser approach, as well as its scalability.

AAMAS Conference 2017 Conference Paper

Using Virtual Narratives to Explore Children's Story Understanding

• Julie Porteous
• Fred Charles
• Cameron Smith
• Marc Cavazza
• Jolien Mouw
• Paul van den Broek

Interactive Narratives are systems that use automated narrative generation techniques to create multiple story variants which can be shown to an audience, as virtual narratives, using cinematic staging techniques. Previous research in this area has focused on assessment of aspects such as the quality of the automatically generated narratives and their acceptance by the audience. However in our work we deviate from this to explore the use of interactive narratives to support cognitive psychology experiments in story understanding. We hypothesized that the use of virtual narratives would enable narrative comprehension to be studied independently of linguistic phenomena. To assess this we developed a demonstration interactive narrative featuring a virtual environment (Unity3D engine) based on a pre-existing children’s story which allows for the generation of variants of the original story that can be "told" via visualization in the 3D world. In the paper we introduce a narrative generation mechanism that provides control over insertion of cues facilitating story understanding, whilst also ensuring that the plot itself is unaffected. An intuitive user interface allows experimenters to insert and order cues and specific events while the narrative generation techniques ensure these requests are effected in a consistent fashion. We also report the results of a field experiment with children (age 9-10) that demonstrates the potential for the use of virtual narratives in story understanding experiments. Our results demonstrated acceptance of virtual narratives, the usability of the system and the impact of cue insertion on inference and story understanding.

ECAI Conference 2016 Conference Paper

Plan-Based Narrative Generation with Coordinated Subplots

• Julie Porteous
• Fred Charles
• Marc Cavazza

Despite recent progress in plan-based narrative generation, one major limitation is that systems tend to produce a single plotline whose progression entirely determines the narrative experience. However, for certain narrative genres such as serial dramas and soaps, multiple interleaved subplots are expected by the audience, as this tends to be the norm in real-world, human-authored narratives. Current narrative generation techniques have overlooked this important requirement, something which could improve the perceived quality of generated stories. To this end, we have developed a flexible plan-based approach to multiplot narrative generation, that successfully generates narratives conforming to different subplot profiles, in terms of the number of subplots interleaved and the relative time spent on each presentation. We have identified specific challenges such as: distribution of virtual characters across subplots; length of each subplot presentation; and transitioning between subplots.

AAAI Conference 2015 Conference Paper

Using Social Relationships to Control Narrative Generation

• Julie Porteous
• Fred Charles
• Marc Cavazza

Narrative generation represents an application domain for AI planning where plan quality is related to properties such as shape of plan trajectory. In our work we have developed a plan-based approach to narrative generation that uses character relationships as a key determinant in controlling plan shape (relationships are key in genres such as serial dramas and soaps). Our approach is implemented in a demonstration Interactive Narrative, called NETWORKING, set in the medical drama genre. The system features a user-friendly mechanism for specifying relationships between virtual characters, via a social network and real-time visualisation of generated narratives on a 3D stage.

AAMAS Conference 2013 Conference Paper

A Social Network Interface to an Interactive Narrative

• Julie Porteous
• Fred Charles
• Marc Cavazza

Regular viewers of serial dramas tend to construct a model of the social relationships between characters as a main determinant for the narrative events that constitute each episode. We have developed a novel approach to interactive narrative in which this dependency is made explicit and can be used to control the generation of various episodes for a baseline drama. This is implemented in our demonstration system, NetworkING (social Network for Interactive Narrative Generation), which provides an interface that enables users to set the social relationships between virtual characters in order to create an episode of their choosing which they can watch as it is visualised as a 3D animation. The domain for the interactive narrative is a medical drama with a cast of virtual characters, such as doctors, nurses and patients. The use of the social network makes the relationships between characters visible and hence leads to the generation of narratives featuring shenanigans within the context of medical story lines.

AAMAS Conference 2011 Conference Paper

Controlling Narrative Time in Interactive Storytelling

• Julie Porteous
• Jonathan Teutenberg
• Fred Charles
• Marc Cavazza

Narrative time has an important role to play in Interactive Storytelling (IS). The prevailing approach to controlling narrative time has been to use implicit models that allow only limited temporal reasoning about virtual agent behaviour. In contrast, this paper proposes the use of an explicit model of narrative time which provides a control mechanism that enhances narrative generation, orchestration of virtual agents and number of possibilities for the staging of agent actions. This approach can help address a number of problems experienced in IS systems both at the level of execution staging and at the level of narrative generation. Consequently it has a number of advantages: it is more flexible with respect to the staging of virtual agent actions; it reduces the possibility of timing problems in the coordination of virtual agents; and it enables more expressive representation of narrative worlds and narrative generative power. Overall it provides a uniform, consistent, principled and rigorous approach to the problem of time in agent-based storytelling. In the paper we demonstrate how this approach to controlling narrative time can be implemented within an IS system and illustrate this using our fully implemented IS system that features virtual agents inspired by Shakespeare's The Merchant of Venice. The paper presents results of an experimental evaluation with the system that demonstrates the use of this approach to co-ordinate the actions of virtual agents and to increase narrative generative power.

AAMAS Conference 2011 Conference Paper

Interactive Storytelling with Temporal Planning

• Julie Porteous
• Jonathan Teutenberg
• Fred Charles
• Marc Cavazza

Narrative time has an important role to play in Interactive Storytelling (IS) systems. In contrast to prevailing IS approaches which use implicit models of time, in our work we have used an explicit model of narrative time. The goal of the demonstration IS system is to show how this explicit temporal representation and reasoning can help overcome certain problems experienced in IS systems such as the coordination of virtual agents and system inflexibility with respect to the staging of virtual agent actions. The fully implemented system features virtual agents and situations inspired by Shakespeare's play The Merchant of Venice.

TIST Journal 2010 Journal Article

Applying planning to interactive storytelling

• Julie Porteous
• Marc Cavazza
• Fred Charles

We have seen ten years of the application of AI planning to the problem of narrative generation in Interactive Storytelling (IS). In that time planning has emerged as the dominant technology and has featured in a number of prototype systems. Nevertheless key issues remain, such as how best to control the shape of the narrative that is generated (e.g., by using narrative control knowledge, i.e., knowledge about narrative features that enhance user experience) and also how best to provide support for real-time interactive performance in order to scale up to more realistic sized systems. Recent progress in planning technology has opened up new avenues for IS and we have developed a novel approach to narrative generation that builds on this. Our approach is to specify narrative control knowledge for a given story world using state trajectory constraints and then to treat these state constraints as landmarks and to use them to decompose narrative generation in order to address scalability issues and the goal of real-time performance in larger story domains. This approach to narrative generation is fully implemented in an interactive narrative based on the “Merchant of Venice.” The contribution of the work lies both in our novel use of state constraints to specify narrative control knowledge for interactive storytelling and also our development of an approach to narrative generation that exploits such constraints. In the article we show how the use of state constraints can provide a unified perspective on important problems faced in IS.

AAMAS Conference 2010 Conference Paper

Multimodal Interaction with a Virtual Character in Interactive Storytelling

• Nikolaus Bee
• Johannes Wagner
• Elisabeth Andre
• Fred Charles
• David Pizzi
• Marc Cavazza

A number of interactive storytelling (IS) systems offer the user the possibility to input natural language input which determines how a story progresses. In this paper, we present an approach where the user's affective and attentive state mainly drives the evolution of the narrative. We introduce a framework for real-time signal processing (SSI) to analyze the users' state which then influences the feelings of the story characters and their actions in the story. SSI is not only used to enable more natural character responses that are sensitive to the user's state at runtime. In addition SSI offers the possibility to collect a large variety of synchronized user data which can be used to analyze the user's experience in offline mode. The underlying narrative in which the approach was tested is based on a classical XIX$^\mathrm{th}$ century psychological novel: Madame Bovary, by Flaubert.

AAMAS Conference 2010 Conference Paper

Narrative Generation through Characters' Point of View

• Julie Porteous
• Marc Cavazza
• Fred Charles

Virtual Actors are at the heart of Interactive Storytellingsystems and in recent years multiple approaches have beendescribed to specify their autonomous behaviour. One wellknown problem is how to achieve a balance between thecharacters' autonomy, defined in terms of their individualroles and motivations, and the global structure of the plot, which tends to emphasise narrative phenomena and the coordination of multiple characters. In this paper we report anew approach to the definition of virtual characters aimed atachieving a balance between character autonomy and globalplot structure. Where previous approaches have tended tofocus on individual actions our objective is to reincorporatehigher-level narrative elements in the behaviour of individual actors and address the relation between character andplot at the level of behaviour representation. To this endwe introduce the notion of a characters' Point of View andshow how it enables a story to be described from the perspective of a number of different characters: it is not merelya presentation effect it is also a different way to tell a story. As an illustration, we have developed an Interactive Narrative based on Shakespeare's Merchant of Venice. The system, which features a novel planning approach to story generation, can generate very different stories depending on thePoint of View adopted and support dynamic modification ofthe story world which results in different story consequences. In the paper, we illustrate this approach using example narratives generated using our fully implemented prototype.

AAMAS Conference 2009 Conference Paper

Emotional Input for Character-based Interactive Storytelling

• Marc Cavazza
• David Pizzi
• Fred Charles
• Thurid Vogt
• Elisabeth André

In most Interactive Storytelling systems, user interaction is based on natural language communication with virtual agents, either through isolated utterances or through dialogue. Natural language communication is also an essential element of interactive narratives in which the user is supposed to impersonate one of the story’s characters. Whilst techniques for narrative generation and agent behaviour have made significant progress in recent years, natural language processing remains a bottleneck hampering the scalability of Interactive Storytelling systems. In this paper, we introduce a novel interaction technique based solely on emotional speech recognition. It allows the user to take part in dialogue with virtual actors without any constraints on style or expressivity, by mapping the recognised emotional categories to narrative situations and virtual characters feelings. Our Interactive Storytelling system uses an emotional planner to drive characters’ behaviours. The main feature of this approach is that characters’ feelings are part of the planning domain and are at the heart of narrative representations. The emotional speech recogniser analyses the speech signal to produce a variety of features which can be used to define ad-hoc categories on which to train the system. The content of our interactive narrative is an adaptation of one chapter of the XIXth century classic novel, Madame Bovary, which is well suited to a formalisation in terms of characters’ feelings. At various stages of the narrative, the user can address the main character or respond to her, impersonating her lover. The emotional category extracted from the user utterance can be analysed in terms of the current narrative context, which includes characters’ beliefs, feelings and expectations, to produce a specific influence on the target character, which will become visible through a change in its behaviour, achieving a high level of realism for the interaction. A limited number of emotional categories is sufficient to drive the narrative across multiple courses of actions, since it comprises over thirty narrative functions. We report results from a fully implemented prototype, both in terms of proof of concept and of usability through a preliminary user study.

AAMAS Conference 2004 Conference Paper

Exploring the Scalability of Character-based Storytelling

• Fred Charles
• Marc Cavazza

AAMAS Conference 2002 Conference Paper

Interacting with Virtual Characters in Interactive Storytelling

• Marc Cavazza
• Fred Charles
• Steven J. Mead