Author name cluster

Nori Jacoby

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

2 author rows

TMLR Journal 2025 Journal Article

Getting aligned on representational alignment

Ilia Sucholutsky
Lukas Muttenthaler
Adrian Weller
Andi Peng
Andreea Bobu
Been Kim
Bradley C. Love
Christopher J Cueva

Biological and artificial information processing systems form representations of the world that they can use to categorize, reason, plan, navigate, and make decisions. How can we measure the similarity between the representations formed by these diverse systems? Do similarities in representations then translate into similar behavior? If so, then how can a system's representations be modified to better match those of another system? These questions pertaining to the study of \emph{representational alignment} are at the heart of some of the most promising research areas in contemporary cognitive science, neuroscience, and machine learning. In this Perspective, we survey the exciting recent developments in representational alignment research in the fields of cognitive science, neuroscience, and machine learning. Despite their overlapping interests, there is limited knowledge transfer between these fields, so work in one field ends up duplicated in another, and useful innovations are not shared effectively. To improve communication, we propose a unifying framework that can serve as a common language for research on representational alignment, and map several streams of existing work across fields within our framework. We also lay out open problems in representational alignment where progress can benefit all three of these fields. We hope that this paper will catalyze cross-disciplinary collaboration and accelerate progress for all communities studying and developing information processing systems.

PDF Details

ICML Conference 2023 Conference Paper

Analyzing Diffusion as Serial Reproduction

Raja Marjieh
Ilia Sucholutsky
Thomas A. Langlois
Nori Jacoby
Thomas L. Griffiths 0001

Diffusion models are a class of generative models that learn to synthesize samples by inverting a diffusion process that gradually maps data into noise. While these models have enjoyed great success recently, a full theoretical understanding of their observed properties is still lacking, in particular, their weak sensitivity to the choice of noise family and the role of adequate scheduling of noise levels for good synthesis. By identifying a correspondence between diffusion models and a well-known paradigm in cognitive science known as serial reproduction, whereby human agents iteratively observe and reproduce stimuli from memory, we show how the aforementioned properties of diffusion models can be explained as a natural consequence of this correspondence. We then complement our theoretical analysis with simulations that exhibit these key features. Our work highlights how classic paradigms in cognitive science can shed light on state-of-the-art machine learning problems.

Details

UAI Conference 2023 Conference Paper

On the informativeness of supervision signals

Ilia Sucholutsky
Ruairidh M. Battleday
Katherine M. Collins
Raja Marjieh
Joshua C. Peterson
Pulkit Singh
Umang Bhatt
Nori Jacoby

Supervised learning typically focuses on learning transferable representations from training examples annotated by humans. While rich annotations (like soft labels) carry more information than sparse annotations (like hard labels), they are also more expensive to collect. For example, while hard labels only provide information about the closest class an object belongs to (e. g. , “this is a dog”), soft labels provide information about the object’s relationship with multiple classes (e. g. , “this is most likely a dog, but it could also be a wolf or a coyote”). We use information theory to compare how a number of commonly-used supervision signals contribute to representation-learning performance, as well as how their capacity is affected by factors such as the number of labels, classes, dimensions, and noise. Our framework provides theoretical justification for using hard labels in the big-data regime, but richer supervision signals for few-shot learning and out-of-distribution generalization. We validate these results empirically in a series of experiments with over 1 million crowdsourced image annotations and conduct a cost-benefit analysis to establish a tradeoff curve that enables users to optimize the cost of supervising representation learning on their own datasets.

Details

ICLR Conference 2023 Conference Paper

Words are all you need? Language as an approximation for human similarity judgments

Raja Marjieh
Pol van Rijn
Ilia Sucholutsky
Theodore R. Sumers
Harin Lee
Thomas L. Griffiths 0001
Nori Jacoby

Human similarity judgments are a powerful supervision signal for machine learning applications based on techniques such as contrastive learning, information retrieval, and model alignment, but classical methods for collecting human similarity judgments are too expensive to be used at scale. Recent methods propose using pre-trained deep neural networks (DNNs) to approximate human similarity, but pre-trained DNNs may not be available for certain domains (e.g., medical images, low-resource languages) and their performance in approximating human similarity has not been extensively tested. We conducted an evaluation of 611 pre-trained models across three domains -- images, audio, video -- and found that there is a large gap in performance between human similarity judgments and pre-trained DNNs. To address this gap, we propose a new class of similarity approximation methods based on language. To collect the language data required by these new methods, we also developed and validated a novel adaptive tag collection pipeline. We find that our proposed language-based methods are significantly cheaper, in the number of human judgments, than classical methods, but still improve performance over the DNN-based methods. Finally, we also develop `stacked' methods that combine language embeddings with DNN embeddings, and find that these consistently provide the best approximations for human similarity across all three of our modalities. Based on the results of this comprehensive study, we provide a concise guide for researchers interested in collecting or approximating human similarity data. To accompany this guide, we also release all of the similarity and language data, a total of 206,339 human judgments, that we collected in our experiments, along with a detailed breakdown of all modeling results.

Details

NeurIPS Conference 2021 Conference Paper

Passive attention in artificial neural networks predicts human visual selectivity

Thomas Langlois
Haicheng Zhao
Erin Grant
Ishita Dasgupta
Tom Griffiths
Nori Jacoby

Developments in machine learning interpretability techniques over the past decade have provided new tools to observe the image regions that are most informative for classification and localization in artificial neural networks (ANNs). Are the same regions similarly informative to human observers? Using data from 79 new experiments and 7, 810 participants, we show that passive attention techniques reveal a significant overlap with human visual selectivity estimates derived from 6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search, and saliency search fixations. We find that input visualizations derived from relatively simple ANN architectures probed using guided backpropagation methods are the best predictors of a shared component in the joint variability of the human measures. We validate these correlational results with causal manipulations using recognition experiments. We show that images masked with ANN attention maps were easier for humans to classify than control masks in a speeded recognition experiment. Similarly, we find that recognition performance in the same ANN models was likewise influenced by masking input images using human visual selectivity maps. This work contributes a new approach to evaluating the biological and psychological validity of leading ANNs as models of human vision: by examining their similarities and differences in terms of their visual selectivity to the information contained in images.

PDF Details

NeurIPS Conference 2020 Conference Paper

Gibbs Sampling with People

Peter Harrison
Raja Marjieh
Federico Adolfi
Pol van Rijn
Manuel Anglada-Tort
Ofer Tchernichovski
Pauline Larrouy-Maestri
Nori Jacoby

A core problem in cognitive science and machine learning is to understand how humans derive semantic representations from perceptual objects, such as color from an apple, pleasantness from a musical chord, or seriousness from a face. Markov Chain Monte Carlo with People (MCMCP) is a prominent method for studying such representations, in which participants are presented with binary choice trials constructed such that the decisions follow a Markov Chain Monte Carlo acceptance rule. However, while MCMCP has strong asymptotic properties, its binary choice paradigm generates relatively little information per trial, and its local proposal function makes it slow to explore the parameter space and find the modes of the distribution. Here we therefore generalize MCMCP to a continuous-sampling paradigm, where in each iteration the participant uses a slider to continuously manipulate a single stimulus dimension to optimize a given criterion such as ‘pleasantness’. We formulate both methods from a utility-theory perspective, and show that the new method can be interpreted as ‘Gibbs Sampling with People’ (GSP). Further, we introduce an aggregation parameter to the transition step, and show that this parameter can be manipulated to flexibly shift between Gibbs sampling and deterministic optimization. In an initial study, we show GSP clearly outperforming MCMCP; we then show that GSP provides novel and interpretable results in three other domains, namely musical chords, vocal emotions, and faces. We validate these results through large-scale perceptual rating experiments. The final experiments use GSP to navigate the latent space of a state-of-the-art image synthesis network (StyleGAN), a promising approach for applying GSP to high-dimensional perceptual spaces. We conclude by discussing future cognitive applications and ethical implications.

PDF Details

YNIMG Journal 2016 Journal Article

Common modulation of limbic network activation underlies musical emotions as they unfold

Neomi Singer
Nori Jacoby
Tamar Lin
Gal Raz
Lavi Shpigelman
Gadi Gilam
Roni Y. Granot
Talma Hendler

Music is a powerful means for communicating emotions among individuals. Here we reveal that this continuous stream of affective information is commonly represented in the brains of different listeners and that particular musical attributes mediate this link. We examined participants' brain responses to two naturalistic musical pieces using functional Magnetic Resonance imaging (fMRI). Following scanning, as participants listened to the musical pieces for a second time, they continuously indicated their emotional experience on scales of valence and arousal. These continuous reports were used along with a detailed annotation of the musical features, to predict a novel index of Dynamic Common Activation (DCA) derived from ten large-scale data-driven functional networks. We found an association between the unfolding music-induced emotionality and the DCA modulation within a vast network of limbic regions. The limbic-DCA modulation further corresponded with continuous changes in two temporal musical features: beat-strength and tempo. Remarkably, this “collective limbic sensitivity” to temporal features was found to mediate the link between limbic-DCA and the reported emotionality. An additional association with the emotional experience was found in a left fronto-parietal network, but only among a sub-group of participants with a high level of musical experience (>5years). These findings may indicate two processing-levels underlying the unfolding of common music emotionality; (1) a widely shared core-affective process that is confined to a limbic network and mediated by temporal regularities in music and (2) an experience based process that is rooted in a left fronto-parietal network that may involve functioning of the ‘mirror-neuron system’.

Details DOI