Author name cluster

Sumit Bhatia

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

10 papers

2 author rows

NAI Journal 2025 Journal Article

Benchmarking Neurosymbolic Description Logic Reasoners: Existing Challenges and a Way Forward

Gunjan Singh
Riccardo Tommasini
Sumit Bhatia
Raghava Mutharaju

Recently, there has been significant progress in the development of robust and highly scalable neurosymbolic description logic reasoners. However, the field faces challenges arising from diverse design strategies and evaluation methods. We address the latter challenge by emphasizing the critical requirement for a comprehensive benchmark framework tailored to the unique evaluation needs of neurosymbolic description logic reasoners. In this paper, we address barriers that must be overcome to facilitate the effective evaluation of these reasoners and outline a potential methodology for designing the benchmark framework. This work contributes toward a more systematic and principled evaluation framework for neurosymbolic reasoning, highlighting the broader role of benchmarks in advancing the field.

Details DOI

ICLR Conference 2025 Conference Paper

It Helps to Take a Second Opinion: Teaching Smaller LLMs To Deliberate Mutually via Selective Rationale Optimisation

Sohan Patnaik
Milan Aggarwal
Sumit Bhatia
Balaji Krishnamurthy

Very large language models (LLMs) such as GPT-4 have shown the ability to handle complex tasks by generating and self-refining step-by-step rationales. Smaller language models (SLMs), typically with < 13B parameters, have been improved by using the data generated from very-large LMs through knowledge distillation. However, various practical constraints such as API costs, copyright, legal and ethical policies restrict using large (often opaque) models to train smaller models for commercial use. Limited success has been achieved at improving the ability of an SLM to explore the space of possible rationales and evaluate them by itself through self-deliberation. To address this, we propose COALITION, a trainable framework that facilitates interaction between two variants of the same SLM and trains them to generate and refine rationales optimized for the end-task. The variants exhibit different behaviors to produce a set of diverse candidate rationales during the generation and refinement steps. The model is then trained via Selective Rationale Optimization (SRO) to prefer generating rationale candidates that maximize the likelihood of producing the ground-truth answer. During inference, COALITION employs a controller to select the suitable variant for generating and refining the rationales. On five different datasets covering mathematical problems, commonsense reasoning, and natural language inference, COALITION outperforms several baselines by up to 5%. Our ablation studies reveal that cross-communication between the two variants performs better than using the single model to self-refine the rationales. We also demonstrate the applicability of COALITION for LMs of varying scales (4B to 14B parameters) and model families (Mistral, Llama, Qwen, Phi). We release the code for this work here.

Details

AAAI Conference 2024 Conference Paper

All Should Be Equal in the Eyes of LMs: Counterfactually Aware Fair Text Generation

Pragyan Banerjee
Abhinav Java
Surgan Jandial
Simra Shahid
Shaz Furniturewala
Balaji Krishnamurthy
Sumit Bhatia

Fairness in Language Models (LMs) remains a long-standing challenge, given the inherent biases in training data that can be perpetuated by models and affect the downstream tasks. Recent methods employ expensive retraining or attempt debiasing during inference by constraining model outputs to contrast from a reference set of biased templates/exemplars. Regardless, they don’t address the primary goal of fairness to maintain equitability across different demographic groups. In this work, we posit that inferencing LMs to generate unbiased output for one demographic under a context ensues from being aware of outputs for other demographics under the same context. To this end, we propose Counterfactually Aware Fair InferencE (CAFIE), a framework that dynamically compares the model’s understanding of diverse demographics to generate more equitable sentences. We conduct an extensive empirical evaluation using base LMs of varying sizes and across three diverse datasets and found that CAFIE outperforms strong baselines. CAFIE produces fairer text and strikes the best balance between fairness and language modeling capability.

PDF Details DOI

ICLR Conference 2024 Conference Paper

CABINET: Content Relevance-based Noise Reduction for Table Question Answering

Sohan Patnaik
Heril Changwal
Milan Aggarwal
Sumit Bhatia
Yaman Kumar Singla
Balaji Krishnamurthy

Table understanding capability of Large Language Models (LLMs) has been extensively studied through the task of question-answering (QA) over tables. Typically, only a small part of the whole table is relevant to derive the answer for a given question. The irrelevant parts act as noise and are distracting information, resulting in sub-optimal performance due to the vulnerability of LLMs to noise. To mitigate this, we propose CABINET (Content RelevAnce-Based NoIse ReductioN for TablE QuesTion-Answering) – a framework to enable LLMs to focus on relevant tabular data by suppressing extraneous information. CABINET comprises an Unsupervised Relevance Scorer (URS), trained differentially with the QA LLM, that weighs the table content based on its relevance to the input question before feeding it to the question answering LLM (QA LLM). To further aid the relevance scorer, CABINET employs a weakly supervised module that generates a parsing statement describing the criteria of rows and columns relevant to the question and highlights the content of corresponding table cells. CABINET significantly outperforms various tabular LLM baselines, as well as GPT3-based in-context learning methods, is more robust to noise, maintains outperformance on tables of varying sizes, and establishes new SoTA performance on WikiTQ, FeTaQA, and WikiSQL datasets. We release our code and datasets here.

Details

AAAI Conference 2024 Short Paper

Evaluating the Efficacy of Prompting Techniques for Debiasing Language Model Outputs (Student Abstract)

Shaz Furniturewala
Surgan Jandial
Abhinav Java
Simra Shahid
Pragyan Banerjee
Balaji Krishnamurthy
Sumit Bhatia
Kokil Jaidka

Achieving fairness in Large Language Models (LLMs) continues to pose a persistent challenge, as these models are prone to inheriting biases from their training data, which can subsequently impact their performance in various applications. There is a need to systematically explore whether structured prompting techniques can offer opportunities for debiased text generation by LLMs. In this work, we designed an evaluative framework to test the efficacy of different prompting techniques for debiasing text along different dimensions. We aim to devise a general structured prompting approach to achieve fairness that generalizes well to different texts and LLMs.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

CyCLIP: Cyclic Contrastive Language-Image Pretraining

Shashank Goel
Hritik Bansal
Sumit Bhatia
Ryan Rossi
Vishwa Vinay
Aditya Grover

Recent advances in contrastive representation learning over paired image-text data have led to models such as CLIP that achieve state-of-the-art performance for zero-shot classification and distributional robustness. Such models typically require joint reasoning in the image and text representation spaces for downstream inference tasks. Contrary to prior beliefs, we demonstrate that the image and text representations learned via a standard contrastive objective are not interchangeable and can lead to inconsistent downstream predictions. To mitigate this issue, we formalize consistency and propose CyCLIP, a framework for contrastive representation learning that explicitly optimizes for the learned representations to be geometrically consistent in the image and text space. In particular, we show that consistent representations can be learned by explicitly symmetrizing (a) the similarity between the two mismatched image-text pairs (cross-modal consistency); and (b) the similarity between the image-image pair and the text-text pair (in-modal consistency). Empirically, we show that the improved consistency in CyCLIP translates to significant gains over CLIP, with gains ranging from 10%-24% for zero-shot classification on standard benchmarks (CIFAR-10, CIFAR-100, ImageNet1K) and 10%-27% for robustness to various natural distribution shifts.

PDF Details

AAAI Conference 2021 Short Paper

Neuro-Symbolic Techniques for Description Logic Reasoning (Student Abstract)

Gunjan Singh
Sutapa Mondal
Sumit Bhatia
Raghava Mutharaju

With the goal to find scalable reasoning approaches, neurosymbolic techniques have gained significant attention. However, the existing approaches do not take into account the inference capabilities of ontology languages that are based on expressive description logic (such as OWL 2). To fill this gap, we propose two approaches: an ontology-based embedding model for theories in EL++ description logic and a reinforcement learning-based solution for efficient tableau-based reasoning on description logic. We describe promising initial results of our efforts towards these directions and lay down the direction for future work.

PDF Details

AAAI Conference 2018 Conference Paper

Bernoulli Embeddings for Graphs

Vinith Misra
Sumit Bhatia

Just as semantic hashing (Salakhutdinov and Hinton 2009) can accelerate information retrieval, binary valued embeddings can signiﬁcantly reduce latency in the retrieval of graphical data. We introduce a simple but effective model for learning such binary vectors for nodes in a graph. By imagining the embeddings as independent coin ﬂips of varying bias, continuous optimization techniques can be applied to the approximate expected loss. Embeddings optimized in this fashion consistently outperform the quantization of both spectral graph embeddings and various learned real-valued embeddings, on both ranking and pre-ranking tasks for a variety of datasets.

PDF Details

AAAI Conference 2010 Conference Paper

Adopting Inference Networks for Online Thread Retrieval

Sumit Bhatia
Prasenjit Mitra

Online forums contain valuable human-generated information. End-users looking for information would like to find only those threads in forums where relevant information is present. Due to the distinctive characteristics of forum pages from generic web pages, special techniques are required to organize and search for information in these forums. Threads and pages in forums are different from other webpages in their hyperlinking patterns. Forum posts also have associated social and non-textual metadata. In this paper, we propose a model for online thread retrieval based on inference networks that utilizes the structural properties of forum threads. We also investigate the effects of incorporating various relevance indicators in our model. We empirically show the effectiveness of our proposed model using real-world data.

PDF Details

AAAI Conference 2010 Conference Paper

Utilizing Context in Generative Bayesian Models for Linked Corpus

Saurabh Kataria
Prasenjit Mitra
Sumit Bhatia

In an interlinked corpus of documents, the context in which a citation appears provides extra information about the cited document. However, associating terms in the context to the cited document remains an open problem. We propose a novel document generation approach that statistically incorporates the context in which a document links to another document. We quantitatively show that the proposed generation scheme explains the linking phenomenon better than previous approaches. The context information along with the actual content of the document provides signicant improvements over the previous approaches for various real world evaluation tasks such as link prediction and log-likelihood estimation on unseen content. The proposed method is more scalable to large collection of documents compared to the previous approaches.

PDF Details