Ralph Weischedel

AAAI Conference 2020 Conference Paper

Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems

• Sarik Ghazarian
• Ralph Weischedel
• Aram Galstyan
• Nanyun Peng

User engagement is a critical metric for evaluating the quality of open-domain dialogue systems. Prior work has focused on conversation-level engagement by using heuristically constructed features such as the number of turns and the total time of the conversation. In this paper, we investigate the possibility and efficacy of estimating utterance-level engagement and define a novel metric, predictive engagement, for automatic evaluation of open-domain dialogue systems. Our experiments demonstrate that (1) human annotators have high agreement on assessing utterance-level engagement scores; (2) conversation-level engagement scores can be predicted from properly aggregated utterance-level engagement scores. Furthermore, we show that the utterance-level engagement scores can be learned from data. These scores can be incorporated into automatic evaluation metrics for open-domain dialogue systems to improve the correlation with human judgements. This suggests that predictive engagement can be used as a real-time feedback for training better dialogue models.

AAAI Conference 2019 Conference Paper

Plan-and-Write: Towards Better Automatic Storytelling

• Lili Yao
• Nanyun Peng
• Ralph Weischedel
• Kevin Knight
• Dongyan Zhao
• Rui Yan

Automatic storytelling is challenging since it requires generating long, coherent natural language to describes a sensible sequence of events. Despite considerable efforts on automatic story generation in the past, prior work either is restricted in plot planning, or can only generate stories in a narrow domain. In this paper, we explore open-domain story generation that writes stories given a title (topic) as input. We propose a plan-and-write hierarchical generation framework that first plans a storyline, and then generates a story based on the storyline. We compare two planning strategies. The dynamic schema interweaves story planning and its surface realization in text, while the static schema plans out the entire storyline before generating stories. Experiments show that with explicit storyline planning, the generated stories are more diverse, coherent, and on topic than those generated without creating a full plan, according to both automatic and human evaluations.

IJCAI Conference 1991 Conference Paper

POST: Using Probabilities in Language Processing

• Marie Me leer
• Richard Schwartz
• Ralph Weischedel