Author name cluster

Ferenc Huszár

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

3 papers

1 author row

NeurIPS Conference 2024 Conference Paper

Do Finetti: On Causal Effects for Exchangeable Data

Siyuan Guo
Chi Zhang
Karthika Mohan
Ferenc Huszár
Bernhard Schölkopf

We study causal effect estimation in a setting where the data are not i. i. d. $\ $(independent and identically distributed). We focus on exchangeable data satisfying an assumption of independent causal mechanisms. Traditional causal effect estimation frameworks, e. g. , relying on structural causal models and do-calculus, are typically limited to i. i. d. data and do not extend to more general exchangeable generative processes, which naturally arise in multi-environment data. To address this gap, we develop a generalized framework for exchangeable data and introduce a truncated factorization formula that facilitates both the identification and estimation of causal effects in our setting. To illustrate potential applications, we introduce a causal Pólya urn model and demonstrate how intervention propagates effects in exchangeable data settings. Finally, we develop an algorithm that performs simultaneous causal discovery and effect estimation given multi-environment data.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Rule Extrapolation in Language Modeling: A Study of Compositional Generalization on OOD Prompts

Anna Mészáros
Szilvia Ujváry
Wieland Brendel
Patrik Reizinger
Ferenc Huszár

LLMs show remarkable emergent abilities, such as inferring concepts from presumably out-of-distribution prompts, known as in-context learning. Though this success is often attributed to the Transformer architecture, our systematic understanding is limited. In complex real-world data sets, even defining what is out-of-distribution is not obvious. To better understand the OOD behaviour of autoregressive LLMs, we focus on formal languages, which are defined by the intersection of rules. We define a new scenario of OOD compositional generalization, termed \textit{rule extrapolation}. Rule extrapolation describes OOD scenarios, where the prompt violates at least one rule. We evaluate rule extrapolation in formal languages with varying complexity in linear and recurrent architectures, the Transformer, and state space models to understand the architectures' influence on rule extrapolation. We also lay the first stones of a normative theory of rule extrapolation, inspired by the Solomonoff prior in algorithmic information theory.

PDF Details DOI

TMLR Journal 2023 Journal Article

Jacobian-based Causal Discovery with Nonlinear ICA

Patrik Reizinger
Yash Sharma
Matthias Bethge
Bernhard Schölkopf
Ferenc Huszár
Wieland Brendel

Today's methods for uncovering causal relationships from observational data either constrain functional assignments (linearity/additive noise assumptions) or the data generating process (e.g., non-i.i.d. assumptions). Unlike previous works, which use conditional independence tests, we rely on the inference function's Jacobian to determine nonlinear cause-effect relationships. We prove that, under strong identifiability, the inference function's Jacobian captures the sparsity structure of the causal graph; thus, generalizing the classic LiNGAM method to the nonlinear case. We use nonlinear Independent Component Analysis (ICA) to infer the underlying sources from the observed variables and show how nonlinear ICA is compatible with causal discovery via non-i.i.d data. Our approach avoids the cost of exponentially many independence tests and makes our method end-to-end differentiable. We demonstrate that the proposed method can infer the causal graph on multiple synthetic data sets, and in most scenarios outperforms previous work.

PDF Details