Author name cluster

Christopher Summerfield

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

1 author row

NeurIPS Conference 2025 Conference Paper

Measuring what Matters: Construct Validity in Large Language Model Benchmarks

Andrew M. Bean
Ryan Othniel Kearns
Angelika Romanou
Franziska Sofia Hafner
Harry Mayne
Jan Batzner
Negar Foroutan Eghlidi
Chris Schmitz

Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as safety' and robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.

PDF Details

NeurIPS Conference 2024 Conference Paper

Flexible task abstractions emerge in linear networks with fast and bounded units

Kai Sandbrink
Jan P. Bauer
Alexandra M. Proca
Andrew M. Saxe
Christopher Summerfield
Ali Hummos

Animals survive in dynamic environments changing at arbitrary timescales, but such data distribution shifts are a challenge to neural networks. To adapt to change, neural systems may change a large number of parameters, which is a slow process involving forgetting past information. In contrast, animals leverage distribution changes to segment their stream of experience into tasks and associate them with internal task abstracts. Animals can then respond flexibly by selecting the appropriate task abstraction. However, how such flexible task abstractions may arise in neural systems remains unknown. Here, we analyze a linear gated network where the weights and gates are jointly optimized via gradient descent, but with neuron-like constraints on the gates including a faster timescale, non-negativity, and bounded activity. We observe that the weights self-organize into modules specialized for tasks or sub-tasks encountered, while the gates layer forms unique representations that switch the appropriate weight modules (task abstractions). We analytically reduce the learning dynamics to an effective eigenspace, revealing a virtuous cycle: fast adapting gates drive weight specialization by protecting previous knowledge, while weight specialization in turn increases the update rate of the gating layer. Task switching in the gating layer accelerates as a function of curriculum block size and task training, mirroring key findings in cognitive neuroscience. We show that the discovered task abstractions support generalization through both task and subtask composition, and we extend our findings to a non-linear network switching between two tasks. Overall, our work offers a theory of cognitive flexibility in animals as arising from joint gradient descent on synaptic and neural gating in a neural network architecture.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

Fine-tuning language models to find agreement among humans with diverse preferences

Michiel Bakker
Martin Chadwick
Hannah Sheahan
Michael Tessler
Lucy Campbell-Gillingham
Jan Balaguer
Nat McAleese
Amelia Glaese

Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user. This work assumes that human preferences are static and homogeneous across individuals, so that aligning to a single "generic" user will confer more general alignment. Here, we embrace the heterogeneity of human preferences to consider a different challenge: how might a machine help people with diverse views find agreement? We fine-tune a 70 billion parameter LLM to generate statements that maximize the expected approval for a group of people with potentially diverse opinions. Human participants provide written opinions on thousands of questions touching on moral and political issues (e. g. , "should we raise taxes on the rich? "), and rate the LLM's generated candidate consensus statements for agreement and quality. A reward model is then trained to predict individual preferences, enabling it to quantify and rank consensus statements in terms of their appeal to the overall group, defined according to different aggregation (social welfare) functions. The model produces consensus statements that are preferred by human users over those from prompted LLMs ($>70\%$) and significantly outperforms a tight fine-tuned baseline that lacks the final ranking step. Further, our best model's consensus statements are preferred over the best human-generated opinions ($>65\%$). We find that when we silently constructed consensus statements from only a subset of group members, those who were excluded were more likely to dissent, revealing the sensitivity of the consensus to individual contributions. These results highlight the potential to use LLMs to help groups of humans align their values with one another.

PDF Details

NeurIPS Conference 2020 Conference Paper

Characterizing emergent representations in a space of candidate learning rules for deep networks

Yinan Cao
Christopher Summerfield
Andrew Saxe

How are sensory representations learned via experience? Deep learning offers a theoretical toolkit for studying how neural codes emerge under different learning rules. Studies suggesting that representations in deep networks resemble those in biological brains have mostly relied on one specific learning rule: gradient descent, the workhorse behind modern deep learning. However, it remains unclear how robust these emergent representations in deep networks are to this specific choice of learning algorithm. Here we present a continuous two-dimensional space of candidate learning rules, parameterized by levels of top-down feedback and Hebbian learning. We show that this space contains five important candidate learning algorithms as specific points--Gradient Descent, Contrastive Hebbian, quasi-Predictive Coding, Hebbian & Anti-Hebbian. Next, we exhaustively characterize the properties of each rule during learning about hierarchically structured data, and identify zones within this space where deep networks exhibit qualitative signatures of biological learning. We find that while a large set of algorithms achieve zero training error at convergence, only a subset show hallmarks of human semantic development like progressive differentiation and illusory correlations. Further, only a subset adjust intermediate neural representations toward task-relevant representations, indicative of backpropagation-like behavior. Finally, we show that algorithms can dramatically differ in their learned neural representations and dynamics, providing experimentally testable hallmarks of different learning principles. Our findings provide a framework linking diverse neural representational geometries to learning principles which can guide future experiments, and offer evidence about the learning rules likely to be at work in biology.

PDF Details

YNIMG Journal 2016 Journal Article

Rhythmic gain control during supramodal integration of approximate number

Bernhard Spitzer
Felix Blankenburg
Christopher Summerfield

According to one view, neural oscillations structure information processing in time, determining whether sensory inputs have a strong or weak impact on behavior. Recent work showed that during sequential integration of visual inputs, stimuli that fall in the preferred phase of slow (1–3Hz), endogenous EEG activity carry greater weight in subsequent judgment. Here, we asked two questions. Firstly, is this phenomenon modality-specific, or is it supramodal? Secondly, does this effect occur at the level of sequential encoding, or only during decision formation? We analyzed scalp EEG recordings from healthy human participants while they compared the approximate number of visual, auditory or somatosensory pulses in two successive intervals (N1 and N2). Despite differences in activity evoked in different domains, a common, slowly-oscillating (~3Hz) choice-predictive signal was observed in all three modalities with a maximum coincident with pulse onset. Critically, this signal was present during N2 (when a decision was being formed) but absent during N1 (when perceptual information was encoded, but no decision could be made). In other words, rhythmic gain control during sequential processing is a supramodal phenomenon that occurs while information is integrated towards a categorical decision.

Details DOI

YNIMG Journal 2005 Journal Article

Coherent theta-band EEG activity predicts item-context binding during encoding

Christopher Summerfield
Jennifer A. Mangels

Episodic memories consist of semantic information coupled with a rich array of contextual detail. Here, we investigate the neural processes by which information about the sensory context of a learning event is “bound” to the semantic representation of the to-be-encoded item. We present evidence that item-context binding during encoding is mediated by frontoposterior electroencephalographic (EEG) phase locking within and between hemispheres in the theta (4–8 Hz) band. During a task in which subjects encoded words in different font colors, later memory for the word was associated with sustained frontal theta activity and frontoposterior theta-band coherence, primarily within the left hemisphere. When the word-color association was later successfully retrieved, however, neurons synchronized their theta-band responses bilaterally in a more sustained fashion, particularly during the latter part of the stimulus epoch (>800 ms). Our results confirm the importance of functional coupling between frontal and posterior regions for successful encoding. One interpretation of these data is hemispheric contributions to item and context encoding may be asymmetric, with left hemisphere coherence facilitating semantic processing of an item and right hemisphere coherence facilitating processing of sensory context. Theta-band coherence may be an important mechanism by which brain networks exchange information during learning.

Details DOI