Hai Leong Chieu

ICLR Conference 2025 Conference Paper

Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse

• Maojia Song
• Shang Hong Sim
• Rishabh Bhardwaj
• Hai Leong Chieu
• Navonil Majumder
• Soujanya Poria

LLMs are an integral component of retrieval-augmented generation (RAG) systems. While many studies focus on evaluating the overall quality of end-to-end RAG systems, there is a gap in understanding the appropriateness of LLMs for the RAG task. To address this, we introduce Trust-Score, a holistic metric that evaluates the trustworthiness of LLMs within the RAG framework. Our results show that various prompting methods, such as in-context learning, fail to effectively adapt LLMs to the RAG task as measured by Trust-Score. Consequently, we propose Trust-Align, a method to align LLMs for improved Trust-Score performance. 26 out of 27 models aligned using Trust-Align substantially outperform competitive baselines on ASQA, QAMPARI, and ELI5. Specifically, in LLaMA-3-8b, Trust-Align outperforms FRONT on ASQA (↑12.56), QAMPARI (↑36.04), and ELI5 (↑17.69). Trust-Align also significantly enhances models’ ability to correctly refuse and provide quality citations. We also demonstrate the effectiveness of Trust-Align across different open-weight models, including the LLaMA series (1b to 8b), Qwen-2.5 series (0.5b to 7b), and Phi3.5 (3.8b). We release our code at https://github.com/declare-lab/trust-align.

AAAI Conference 2020 Conference Paper

Interpretable Rumor Detection in Microblogs by Attending to User Interactions

• Ling Min Serena Khoo
• Hai Leong Chieu
• Zhong Qian
• Jing Jiang

We address rumor detection by learning to differentiate between the community’s response to real and fake claims in microblogs. Existing state-of-the-art models are based on tree models that model conversational trees. However, in social media, a user posting a reply might be replying to the entire thread rather than to a specific user. We propose a post-level attention model (PLAN) to model long distance interactions between tweets with the multi-head attention mechanism in a transformer network. We investigated variants of this model: (1) a structure aware self-attention model (StA-PLAN) that incorporates tree structure information in the transformer network, and (2) a hierarchical token and post-level attention model (StA-HiTPLAN) that learns a sentence representation with token-level self-attention. To the best of our knowledge, we are the first to evaluate our models on two rumor detection data sets: the PHEME data set as well as the Twitter15 and Twitter16 data sets. We show that our best models outperform current state-of-the-art models for both data sets. Moreover, the attention mechanism allows us to explain rumor detection predictions at both token-level and post-level.

JMLR Journal 2014 Journal Article

Conditional Random Field with High-order Dependencies for Sequence Labeling and Segmentation

• Nguyen Viet Cuong
• Nan Ye
• Wee Sun Lee
• Hai Leong Chieu

Dependencies among neighboring labels in a sequence are important sources of information for sequence labeling and segmentation. However, only first-order dependencies, which are dependencies between adjacent labels or segments, are commonly exploited in practice because of the high computational complexity of typical inference algorithms when longer distance dependencies are taken into account. In this paper, we give efficient inference algorithms to handle high-order dependencies between labels or segments in conditional random fields, under the assumption that the number of distinct label patterns used in the features is small. This leads to efficient learning algorithms for these conditional random fields. We show experimentally that exploiting high-order dependencies can lead to substantial performance improvements for some problems, and we discuss conditions under which high-order features can be effective. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2014. ( edit, beta )

NeurIPS Conference 2013 Conference Paper

Active Learning for Probabilistic Hypotheses Using the Maximum Gibbs Error Criterion

• Nguyen Viet Cuong
• Wee Sun Lee
• Nan Ye
• Kian Ming Chai
• Hai Leong Chieu

We introduce a new objective function for pool-based Bayesian active learning with probabilistic hypotheses. This objective function, called the policy Gibbs error, is the expected error rate of a random classifier drawn from the prior distribution on the examples adaptively selected by the active learning policy. Exact maximization of the policy Gibbs error is hard, so we propose a greedy strategy that maximizes the Gibbs error at each iteration, where the Gibbs error on an instance is the expected error of a random classifier selected from the posterior label distribution on that instance. We apply this maximum Gibbs error criterion to three active learning scenarios: non-adaptive, adaptive, and batch active learning. In each scenario, we prove that the criterion achieves near-maximal policy Gibbs error when constrained to a fixed budget. For practical implementations, we provide approximations to the maximum Gibbs error criterion for Bayesian conditional random fields and transductive Naive Bayes. Our experimental results on a named entity recognition task and a text classification task show that the maximum Gibbs error criterion is an effective active learning criterion for noisy models.

ICML Conference 2012 Conference Paper

A Split-Merge Framework for Comparing Clusterings

• Qiaoliang Xiang
• Qi Mao 0001
• Kian Ming Adam Chai
• Hai Leong Chieu
• Ivor W. Tsang
• Zhendong Zhao

ICML Conference 2012 Conference Paper

Optimizing F-measure: A Tale of Two Approaches

• Nan Ye
• Kian Ming Adam Chai
• Wee Sun Lee
• Hai Leong Chieu

AAAI Conference 2008 Conference Paper

Relaxed Survey Propagation: A Sum-Product Algorithm for Max-SAT

• Hai Leong Chieu

The survey propagation (SP) algorithm has been shown to work well on large instances of the random 3-SAT problem near its phase transition. It was shown that SP estimates marginals over covers, using joker states to represent clusters of configurations. The SP-y algorithm generalizes SP to work on the Max-SAT problem, but the cover interpretation of SP does not generalize to SP-y. Recently, a relaxed survey propagation (RSP) algorithm has been proposed for inference in Markov random fields (MRF). RSP for MRFs assigns zero probability to joker states, and hence the cover interpretation is also inapplicable. We adapt RSP to solve Max-SAT problems, and show that it has an interpretation of estimating marginals over covers violating a minimum number of clauses. This naturally generalizes the cover interpretation of SP. Empirically, we show that RSP outperforms SP-y and other state-of-the-art solvers on random as well as benchmark instances of Max-SAT.

AAAI Conference 2002 Conference Paper

A Maximum Entropy Approach to Information Extraction from Semi-Structured and Free Text

• Hai Leong Chieu

In this paper, we present a classification-based approach towards single-slot as well as multi-slot information extraction (IE). For single-slot IE, we worked on the domain of Seminar Announcements, where each document contains information on only one seminar. For multi-slot IE, we worked on the domain of Management Succession. For this domain, we restrict ourselves to extracting information sentence by sentence, in the same way as (Soderland 1999). Each sentence can contain information on several management succession events. By using a classification approach based on a maximum entropy framework, our system achieves higher accuracy than the best previously published results in both domains.