Giuseppe Carenini

AAAI Conference 2025 Conference Paper

Evaluating LLM Reasoning in the Operations Research Domain with ORQA

• Mahdi Mostajabdaveh
• Timothy Tin Long Yu
• Samarendra Chandan Bindu Dash
• Rindra Ramamonjison
• Jabo Serge Byusa
• Giuseppe Carenini
• Zirui Zhou
• Yong Zhang

In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark, to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark is designed to evaluate whether LLMs can emulate the knowledge and reasoning skills of OR experts when given diverse and complex optimization problems. The dataset, crafted by OR experts, presents real-world optimization problems that require multistep reasoning to build their mathematical models. Our evaluations of various open-source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral reveal their modest performance, indicating a gap in their aptitude to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs’ generalization capabilities, providing insights for future research in this area. The dataset and evaluation code are publicly available.

AAAI Conference 2022 Conference Paper

Predicting Above-Sentence Discourse Structure Using Distant Supervision from Topic Segmentation

• Patrick Huber
• Linzi Xing
• Giuseppe Carenini

RST-style discourse parsing plays a vital role in many NLP tasks, revealing the underlying semantic/pragmatic structure of potentially complex and diverse documents. Despite its importance, one of the most prevailing limitations in modern day discourse parsing is the lack of large-scale datasets. To overcome the data sparsity issue, distantly supervised approaches from tasks like sentiment analysis and summarization have been recently proposed. Here, we extend this line of research by exploiting distant supervision from topic segmentation, which can arguably provide a strong and oftentimes complementary signal for high-level discourse structures. Experiments on two human-annotated discourse treebanks confirm that our proposal generates accurate tree structures on sentence and paragraph level, consistently outperforming previous distantly supervised models on the sentence-todocument task and occasionally reaching even higher scores on the sentence-to-paragraph level.

AAAI Conference 2021 Conference Paper

Unsupervised Learning of Discourse Structures using a Tree Autoencoder

• Patrick Huber
• Giuseppe Carenini

Discourse information, as postulated by popular discourse theories, such as RST and PDTB, has been shown to improve an increasing number of downstream NLP tasks, showing positive effects and synergies of discourse with important real-world applications. While methods for incorporating discourse become more and more sophisticated, the growing need for robust and general discourse structures has not been sufficiently met by current discourse parsers, usually trained on small scale datasets in a strictly limited number of domains. This makes the prediction for arbitrary tasks noisy and unreliable. The overall resulting lack of high-quality, highquantity discourse trees poses a severe limitation to further progress. In order the alleviate this shortcoming, we propose a new strategy to generate tree structures in a task-agnostic, unsupervised fashion by extending a latent tree induction framework with an auto-encoding objective. The proposed approach can be applied to any tree-structured objective, such as syntactic parsing, discourse parsing and others. However, due to the especially difficult annotation process to generate discourse trees, we initially develop a method to generate larger and more diverse discourse treebanks. In this paper we are inferring general tree structures of natural text in multiple domains, showing promising results on a diverse set of tasks.

IJCAI Conference 2016 Conference Paper

Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data

• S
• eacute; bastien Lall
• eacute;
• Cristina Conati
• Giuseppe Carenini

Confusion has been found to hinder user experience with visualizations. If confusion could be predicted and resolved in real time, user experience and satisfaction would greatly improve. In this paper, we focus on predicting occurrences of confusion during the interaction with a visualization using eye tracking and mouse data. The data was collected during a user study with ValueChart, an interactive visualization to support preferential choices. We report very promising results based on Random Forest classifiers.

AAAI Conference 2016 Conference Paper

What’s Hot in Intelligent User Interfaces

• Shimei Pan
• Oliver Brdiczka
• Giuseppe Carenini
• Duen Chau
• Per Ola Kristensson

AAAI Conference 2015 Conference Paper

Towards User-Adaptive Information Visualization

• Cristina Conati
• Giuseppe Carenini
• Dereck Toker
• Sébastien Lallé

This paper summarizes an ongoing multi-year project aiming to uncover knowledge and techniques for devising intelligent environments for user-adaptive visualizations. We ran three studies designed to investigate the impact of user and task characteristics on user performance and satisfaction in different visualization contexts. Eye-tracking data collected in each study was analyzed to uncover possible interactions between user/task characteristics and gaze behavior during visualization processing. Finally, we investigated user models that can assess user characteristics relevant for adaptation from eye tracking data.

IJCAI Conference 2011 Conference Paper

Unsupervised Modeling of Dialog Acts in Asynchronous Conversations

• Shafiq Joty
• Giuseppe Carenini
• Chin-Yew Lin

We present unsupervised approaches to the problem of modeling dialog acts in asynchronous conversations; i. e. , conversations where participants collaborate with each other at different times. In particular, we investigate a graph-theoretic deterministic framework and two probabilistic conversation models (i. e. , HMM and HMM+Mix) for modeling dialog acts in emails and forums. We train and test our conversation models on (a) temporal order and (b) graph-structural order of the datasets. Empirical evaluation suggests (i) the graph-theoretic framework that relies on lexical and structural similarity metrics is not the right model for this task, (ii) conversation models perform better on the graph-structural order than the temporal order of the datasets and (iii) HMM+Mix is a better conversation model than the simple HMM model.

AAAI Conference 2010 Conference Paper

Model AI Assignments

• Todd Neller
• John DeNero
• Dan Klein
• Sven Koenig
• William Yeoh
• Xiaoming Zheng
• Kenny Daniel
• Alex Nash

AIJ Journal 2006 Journal Article

Generating and evaluating evaluative arguments

• Giuseppe Carenini
• Johanna D. Moore

Evaluative arguments are pervasive in natural human communication. In countless situations people attempt to advise or persuade their interlocutors that something is desirable (vs. undesirable) or right (vs. wrong). With the proliferation of on-line systems serving as personal advisors and assistants, there is a pressing need to develop general and testable computational models for generating and presenting evaluative arguments. Previous research on generating evaluative arguments has been characterized by two major limitations. First, researchers have tended to focus only on specific aspects of the generation process. Second, the proposed approaches were not empirically tested. The research presented in this paper addresses both limitations. We have designed and implemented a complete computational model for generating evaluative arguments. For content selection and organization, we devised an argumentation strategy based on guidelines from argumentation theory. For expressing the content in natural language, we extended and integrated previous work in computational linguistics on generating evaluative arguments. The key knowledge source for both tasks is a quantitative model of user preferences. To empirically test critical aspects of our generation model, we have devised and implemented an evaluation framework in which the effectiveness of evaluative arguments can be measured with real users. Within the framework, we have performed an experiment to test two basic hypotheses on which the design of the computational model is based; namely, that our proposal for tailoring an evaluative argument to the addressee's preferences increases its effectiveness, and that differences in conciseness significantly influence argument effectiveness. The second hypothesis was confirmed in the experiment. In contrast, the first hypothesis was only marginally confirmed. However, independent testing by other researchers has recently provided further support for this hypothesis.

AAAI Conference 2004 Conference Paper

Exploring More Realistic Evaluation Measures for Collaborative Filtering

• Giuseppe Carenini
• Rita Sharma

Collaborative filtering is a popular technique for recommending items to people. Several methods for collaborative filtering have been proposed in the literature and the quality of their predictions compared in empirical studies. In this paper, we argue that the measures of quality used in these studies are based on rather simple assumptions. We propose and apply additional measures for comparing the effectiveness of collaborative filtering methods which are grounded in decisiontheory. [keywords: information agents, human-computer interaction, recommender systems, evaluation]

AIIM Journal 1998 Journal Article

Designing computer-based frameworks that facilitate doctor–patient collaboration

• Bruce G Buchanan
• Giuseppe Carenini
• Vibhu O Mittal
• Johanna D Moore

A current trend in medicine involves establishing collaborative problem solving between patients and physicians in order to involve patients more in their own care. Neither diagnosis nor therapy can be completely successful unless the patient and the doctor understand each other and collaborate with each other in an effort to gauge the other's requests, needs and concerns. This is made even more difficult by the fact that there is often a big difference between the doctors and patients in terms of expectations, vocabulary used, and other factors. For diagnosis of many disorders, a detailed description of the problem and of the patient's history are required. For therapy, patients must understand how and when to take prescribed drugs, what changes to make in diet, exercise, or lifestyle—and why they are important. This paper describes a model of asynchronous collaboration between people with very different knowledge of medicine in which a computer framework attempts to mediate between patients and physicians and reduce some of the differences in communication. It allows patients to pace themselves in familiarizing themselves with the relevant domain terms, some of the medical factors underlying the conditions under question, and the justifications and implications of the prescribed treatment plan. It also allows physicians to request more information of patients and gives patients contextual information to help them understand the underlying reasons for the questions. This framework has been partially implemented in the domain of migraines. As described in the paper, not only is the system designed to cooperate with the patient, but using the system also results in better mutual understanding between the doctor and the patient, thus leading to better collaboration between them.

AIIM Journal 1995 Journal Article

An intelligent interactive system for delivering individualized information to patients

• Bruce G. Buchanan
• Johanna D. Moore
• Diana E. Forsythe
• Giuseppe Carenini
• Stellan Ohlsson
• Gordon Banks

This paper is a report on the first phase of a long-term, interdisciplinary project whose goal is to increase the overall effectiveness of physicians' time, and thus the quality of health care, by improving the information exchange between physicians and patients in clinical settings. We are focusing on patients with long-term and chronic conditions, initially on migraine patients, who require periodic interaction with their physicians for effective management of their condition. We are using medical informatics to focus on the information needs of patients, as well as of physicians, and to address problems of information exchange. This requires understanding patients' concerns to design an appropriate system, and using state-of-the-art artificial intelligence techniques to build an interactive explanation system. In contrast to many other knowledge-based systems, our system's design is based on empirical data on actual information needs. We used ethnographic techniques to observe explanations actually given in clinic settings, and to conduct interviews with migraine sufferers and physicians. Our system has an extensive knowledge base that contains both general medical terminology and specific knowledge about migraine, such as common trigger factors and symptoms of migraine, the common therapies, and the most common effects and side effects of those therapies. The system consists of two main components: (a) an interactive history-taking module that collects information from patients prior to each visit, builds a patient model, and summarizes the patients' status for their physicians; and (b) an intelligent explanation module that produces an interactive information sheet containing explanations in everyday language that are tailored to individual patients, and responds intelligently to follow-up questions about topics covered in the information sheet.