Author name cluster

Pradeep Shenoy

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

15 papers

2 author rows

ICML Conference 2025 Conference Paper

Masked Generative Nested Transformers with Decode Time Scaling

Sahil Goyal
Debapriya Tula
Gagan Jain
Pradeep Shenoy
Prateek Jain 0002
Sujoy Paul

Recent advances in visual generation have made significant strides in producing content of exceptional quality. However, most methods suffer from a fundamental problem - a bottleneck of inference computational efficiency. Most of these algorithms involve multiple passes over a transformer model to generate tokens or denoise inputs. However, the model size is kept consistent throughout all iterations, which makes it computationally expensive. In this work, we aim to address this issue primarily through two key ideas - (a) not all parts of the generation process need equal compute, and we design a decode time model scaling schedule to utilize compute effectively, and (b) we can cache and reuse some of the intermediate computation. Combining these two ideas leads to using smaller models to process more tokens while large models process fewer tokens. These different-sized models do not increase the parameter size, as they share parameters. We rigorously experiment with ImageNet256$\times$256, UCF101, and Kinetics600 to showcase the efficacy of the proposed method for image/video generation and frame prediction. Our experiments show that with almost $3\times$ less compute than baseline, our model obtains competitive performance.

Details

TMLR Journal 2025 Journal Article

Unified Wisdom: Harnessing Collaborative Learning to Improve Efficacy of Knowledge Distillation

Atharva Abhijit Tambat
Durga S
Ganesh Ramakrishnan
Pradeep Shenoy

Knowledge distillation (KD), which involves training a smaller student model to approximate the predictions of a larger teacher model is useful in striking a balance between model accuracy and computational constraints. However, KD has been found to be ineffective when the teacher and student models have a significant capacity gap. In this work, we address this issue via “meta-collaborative distillation” (MC-Distil), where students of varying capacities collaborate during distillation. Using a “coordinator” network (C-Net), MC-Distil enables mutual learning among students as a meta-learning task. Our insight is that C-Net learns from each student’s performance and training instance characteristics, allowing students of different capacities to improve together. Our method enhances student accuracy for all students, surpassing state-of-the-art baselines, including multi-step distillation, consensus enforcement, and teacher re-training. We achieve average gains of 2.5% on CIFAR-100 and 2% on Tiny ImageNet datasets, consistently across diverse student sizes, teacher sizes, and architectures. Notably, larger students benefiting through meta-collaboration with smaller students is a novel idea. MC-Distil excels in training superior student models under real-world conditions such as label noise and domain adaptation. Our approach also yields consistent improvements on the MS COCO object detection benchmark and introduces only a modest 5% computational overhead during training, with no additional cost at inference.

PDF Details

AAAI Conference 2024 Conference Paper

Instance-Conditional Timescales of Decay for Non-Stationary Learning

Nishant Jain
Pradeep Shenoy

Slow concept drift is a ubiquitous, yet under-studied problem in practical machine learning systems. In such settings, although recent data is more indicative of future data, naively prioritizing recent instances runs the risk of losing valuable information from the past. We propose an optimization-driven approach towards balancing instance importance over large training windows. First, we model instance relevance using a mixture of multiple timescales of decay, allowing us to capture rich temporal trends. Second, we learn an auxiliary scorer model that recovers the appropriate mixture of timescales as a function of the instance itself. Finally, we propose a nested optimization objective for learning the scorer, by which it maximizes forward transfer for the learned model. Experiments on a large real-world dataset of 39M photos over a 9 year period show upto 15% relative gains in accuracy compared to other robust learning baselines. We replicate our gains on two collections of real-world datasets for non-stationary learning, and extend our work to continual learning settings where, too, we beat SOTA methods by large margins.

PDF Details DOI

ICLR Conference 2024 Conference Paper

Learning model uncertainty as variance-minimizing instance weights

Nishant Jain
Karthikeyan Shanmugam 0001
Pradeep Shenoy

Predictive uncertainty--a model’s self-awareness regarding its accuracy on an input--is key for both building robust models via training interventions and for test-time applications such as selective classification. We propose a novel instance-conditional reweighting approach that captures predictive uncertainty using an auxiliary network, and unifies these train- and test-time applications. The auxiliary network is trained using a meta-objective in a bilevel optimization framework. A key contribution of our proposal is the meta-objective of minimizing dropout variance, an approximation of Bayesian predictive uncertainty, We show in controlled experiments that we effectively capture diverse specific notions of uncertainty through this meta-objective, while previous approaches only capture certain aspects. These results translate to significant gains in real-world settings–selective classification, label noise, domain adaptation, calibration–and across datasets–Imagenet, Cifar100, diabetic retinopathy, Camelyon, WILDs, Imagenet-C,-A,-R, Clothing-1.6M, etc. For Diabetic Retinopathy, we see upto 3.4\%/3.3\% accuracy & AUC gains over SOTA in selective classification. We also improve upon large-scale pretrained models such as PLEX.

Details

AAAI Conference 2023 Conference Paper

Adaptive Mixing of Auxiliary Losses in Supervised Learning

Durga Sivasubramanian
Ayush Maheshwari
Prathosh AP
Pradeep Shenoy
Ganesh Ramakrishnan

In many supervised learning scenarios, auxiliary losses are used in order to introduce additional information or constraints into the supervised learning objective. For instance, knowledge distillation aims to mimic outputs of a powerful teacher model; similarly, in rule-based approaches, weak labeling information is provided by labeling functions which may be noisy rule-based approximations to true labels. We tackle the problem of learning to combine these losses in a principled manner. Our proposal, AMAL, uses a bi-level optimization criterion on validation data to learn optimal mixing weights, at an instance-level, over the training data. We describe a meta-learning approach towards solving this bi-level objective, and show how it can be applied to different scenarios in supervised learning. Experiments in a number of knowledge distillation and rule denoising domains show that AMAL provides noticeable gains over competitive baselines in those domains. We empirically analyze our method and share insights into the mechanisms through which it provides performance gains. The code for AMAL is at: https://github.com/durgas16/AMAL.git.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Interactive Concept Bottleneck Models

Kushal Chauhan
Rishabh Tiwari
Jan Freyberg
Pradeep Shenoy
Krishnamurthy Dvijotham

Concept bottleneck models (CBMs) are interpretable neural networks that first predict labels for human-interpretable concepts relevant to the prediction task, and then predict the final label based on the concept label predictions. We extend CBMs to interactive prediction settings where the model can query a human collaborator for the label to some concepts. We develop an interaction policy that, at prediction time, chooses which concepts to request a label for so as to maximally improve the final prediction. We demonstrate that a simple policy combining concept prediction uncertainty and influence of the concept on the final prediction achieves strong performance and outperforms static approaches as well as active feature acquisition methods proposed in the literature. We show that the interactive CBM can achieve accuracy gains of 5-10% with only 5 interactions over competitive baselines on the Caltech-UCSD Birds, CheXpert and OAI datasets.

PDF Details DOI

AAMAS Conference 2023 Conference Paper

Matching Options to Tasks using Option-Indexed Hierarchical Reinforcement Learning

Kushal Chauhan
Soumya Chatterjee
Akash Reddy
Aniruddha S
Balaraman Ravindran
Pradeep Shenoy

The options framework in Hierarchical Reinforcement Learning breaks down overall goals into a combination of simpler tasks (options) and their policies, allowing for abstraction in the action space. Ideally, options can be reused across different goals; indeed, this is necessary to build a continual learning agent that can effectively leverage its prior experience. Previous approaches allow limited transfer of pre-learned options to new task settings. We propose a novel option indexing approach to hierarchical learning (OI-HRL), where we learn an affinity function between options and items present in the environment. With OI-HRL, we effectively reuse a large library of pre-trained options in zero-shot generalization at test time by restricting goal-directed learning to relevant options alone. We develop a meta-training loop that learns the representations of options and environments over a series of HRL problems by incorporating feedback about the relevance of retrieved options to the higher-level goal. Our model is competitive with oracular baselines and substantially better than a baseline with the entire option pool available for learning the hierarchical policy.

PDF

ICML Conference 2023 Conference Paper

Overcoming Simplicity Bias in Deep Networks using a Feature Sieve

Rishabh Tiwari
Pradeep Shenoy

Simplicity bias is the concerning tendency of deep networks to over-depend on simple, weakly predictive features, to the exclusion of stronger, more complex features. This causes biased, incorrect model predictions in many real-world applications, exacerbated by incomplete training data containing spurious feature-label correlations. We propose a direct, interventional method for addressing simplicity bias in DNNs, which we call the feature sieve. We aim to automatically identify and suppress easily-computable spurious features in lower layers of the network, thereby allowing the higher network levels to extract and utilize richer, more meaningful representations. We provide concrete evidence of this differential suppression & enhancement of relevant features on both controlled datasets and real-world images, and report substantial gains on many real-world debiasing benchmarks (11. 4% relative gain on Imagenet-A; 3. 2% on BAR, etc). Crucially, we outperform many baselines that incorporate knowledge about known spurious or biased attributes, despite our method not using any such information. We believe that our feature sieve work opens up exciting new research directions in automated adversarial feature extraction & representation learning for deep networks.

Details

IJCAI Conference 2023 Conference Paper

Shaken, and Stirred: Long-Range Dependencies Enable Robust Outlier Detection with PixelCNN++

Barath Mohan Umapathi
Kushal Chauhan
Pradeep Shenoy
Devarajan Sridharan

Reliable outlier detection is critical for real-world deployment of deep learning models. Although extensively studied, likelihoods produced by deep generative models have been largely dismissed as being impractical for outlier detection. First, deep generative model likelihoods are readily biased by low-level input statistics. Second, many recent solutions for correcting these biases are computationally expensive, or do not generalize well to complex, natural datasets. Here, we explore outlier detection with a state-of-the-art deep autoregressive model: PixelCNN++. We show that biases in PixelCNN++ likelihoods arise primarily from predictions based on local dependencies. We propose two families of bijective transformations -- ``stirring'' and ``shaking'' -- which ameliorate low-level biases and isolate the contribution of long-range dependencies to PixelCNN++ likelihoods. These transformations are inexpensive and readily computed at evaluation time. We test our approaches extensively with five grayscale and six natural image datasets and show that they achieve or exceed state-of-the-art outlier detection, particularly on datasets with complex, natural images. We also show that our solutions work well with other types of generative models (generative flows and variational autoencoders) and that their efficacy is governed by each model's reliance on local dependencies. In sum, lightweight remedies suffice to achieve robust outlier detection on image data with deep generative models.

PDF Details DOI

AAAI Conference 2021 Conference Paper

Model-Agnostic Fits for Understanding Information Seeking Patterns in Humans

Soumya Chatterjee
Pradeep Shenoy

In decision making tasks under uncertainty, humans display characteristic biases in seeking, integrating, and acting upon information relevant to the task. Here, we reexamine data from previous carefully designed experiments, collected at scale, that measured and catalogued these biases in aggregate form. We design deep learning models that replicate these biases in aggregate, while also capturing individual variation in behavior. A key finding of our work is that paucity of data collected from each individual subject can be overcome by sampling large numbers of subjects from the population, while still capturing individual differences. We predict human behavior with high accuracy without making any assumptions about task goals, reward structure, or individual biases, thus providing a model-agnostic fit to human behavior in the task. Such an approach can sidestep potential limitations in modeler-specified inductive biases, and has implications for computational modeling of human cognitive function in general, and of human-AI interfaces in particular.

PDF Details

NeurIPS Conference 2012 Conference Paper

Strategic Impatience in Go/NoGo versus Forced-Choice Decision-Making

Pradeep Shenoy
Angela Yu

Two-alternative forced choice (2AFC) and Go/NoGo (GNG) tasks are behavioral choice paradigms commonly used to study sensory and cognitive processing in choice behavior. While GNG is thought to isolate the sensory/decisional component by removing the need for response selection, a consistent bias towards the Go response (higher hits and false alarm rates) in the GNG task suggests possible fundamental differences in the sensory or cognitive processes engaged in the two tasks. Existing mechanistic models of these choice tasks, mostly variants of the drift-diffusion model (DDM; [1, 2]) and the related leaky competing accumulator models [3, 4] capture various aspects of behavior but do not address the provenance of the Go bias. We postulate that this ``impatience'' to go is a strategic adjustment in response to the implicit asymmetry in the cost structure of GNG: the NoGo response requires waiting until the response deadline, while a Go response immediately terminates the current trial. We show that a Bayes-risk minimizing decision policy that minimizes both error rate and average decision delay naturally exhibits the experimentally observed bias. The optimal decision policy is formally equivalent to a DDM with a time-varying threshold that initially rises after stimulus onset, and collapses again near the response deadline. The initial rise is due to the fading temporal advantage of choosing the Go response over the fixed-delay NoGo response. We show that fitting a simpler, fixed-threshold DDM to the optimal model reproduces the counterintuitive result of a higher threshold in GNG than 2AFC decision-making, previously observed in direct DDM fit to behavioral data [2], although such approximations cannot reproduce the Go bias. Thus, observed discrepancies between GNG and 2AFC decision-making may arise from rational strategic adjustments to the cost structure, and need not imply additional differences in the underlying sensory and cognitive processes.

PDF Details

NeurIPS Conference 2010 Conference Paper

A rational decision making framework for inhibitory control

Pradeep Shenoy
Angela Yu
Rajesh Rao

Intelligent agents are often faced with the need to choose actions with uncertain consequences, and to modify those actions according to ongoing sensory processing and changing task demands. The requisite ability to dynamically modify or cancel planned actions is known as inhibitory control in psychology. We formalize inhibitory control as a rational decision-making problem, and apply to it to the classical stop-signal task. Using Bayesian inference and stochastic control tools, we show that the optimal policy systematically depends on various parameters of the problem, such as the relative costs of different action choices, the noise level of sensory inputs, and the dynamics of changing environmental demands. Our normative model accounts for a range of behavioral data in humans and animals in the stop-signal task, suggesting that the brain implements statistically optimal, dynamically adaptive, and reward-sensitive decision-making in the context of inhibitory control problems.

PDF Details

YNIMG Journal 2007 Journal Article

Real-time functional brain mapping using electrocorticography

Kai J. Miller
Marcel denNijs
Pradeep Shenoy
John W. Miller
Rajesh P.N. Rao
Jeffrey G. Ojemann

We demonstrate the feasibility of real-time cortical mapping from arrays of subdural electrodes using the electrocorticographic signal power in the higher spectral frequencies (76–200 Hz, or “χ-index”). Hand area was mapped offline in eight individuals using brief baseline and hand-movement measurements. In one patient, hand sensorimotor cortex was identified online during a handshake. We propose that this high-frequency component of the electrocorticogram provides a generic, reliable, clinically useful correlate of local cortical function.

Details DOI

AAAI Conference 2005 Conference Paper

Real-Time Classification of Electromyographic Signals for Robotic Control

Beau Crawford
Pradeep Shenoy

Advances in bioengineering have led to increasingly sophisticated prosthetic devices for amputees and paralyzed individuals. Control of such devices necessitates real-time classification of biosignals, e. g. , electromyographic (EMG) signals recorded from intact muscles. In this paper, we show that a 4-degrees-of-freedom robotic arm can be controlled in real-time using non-invasive surface EMG signals recorded from the forearm. The innovative features of our system include a physiologically-informed selection of forearm muscles for recording EMG signals, intelligent choice of hand gestures for easy classification, and fast, simple feature extraction from EMG signals. Our selection of gestures is meant to intuitively map to appropriate degrees of freedom in the robotic arm. These design decisions allow us to build fast accurate classifiers online, and control a 4-DOF robotic arm in real-time. In a study involving 3 subjects, we achieved accuracies of 92-98% on an 8-class classification problem using linear SVMs. These classifiers can be learned on-line in under 10 minutes, including data collection and training. Our study also analyzes the issues and tradeoffs involved in designing schemes for robotic control using EMG. Finally, we present details of online experiments where subjects successfully solved tasks of varying complexity using EMG to control the robotic arm.

PDF Details

NeurIPS Conference 2004 Conference Paper

Dynamic Bayesian Networks for Brain-Computer Interfaces

Pradeep Shenoy
Rajesh Rao

We describe an approach to building brain-computer interfaces (BCI) based on graphical models for probabilistic inference and learning. We show how a dynamic Bayesian network (DBN) can be used to infer probability distributions over brain- and body-states during planning and execution of actions. The DBN is learned directly from observed data and allows measured signals such as EEG and EMG to be interpreted in terms of internal states such as intent to move, preparatory activity, and movement execution. Unlike traditional classification-based approaches to BCI, the proposed approach (1) allows continuous tracking and predic- tion of internal states over time, and (2) generates control signals based on an entire probability distribution over states rather than binary yes/no decisions. We present preliminary results of brain- and body-state es- timation using simultaneous EEG and EMG signals recorded during a self-paced left/right hand movement task.

PDF Details