Author name cluster

Manfred Opper

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

42 papers

2 author rows

NeurIPS Conference 2025 Conference Paper

Fractional Diffusion Bridge Models

Gabriel Nobis
Maximilian Springenberg
Arina Belova
Rembert Daems
Christoph Knochenhauer
Manfred Opper
Tolga Birdal
Wojciech Samek

We present *Fractional Diffusion Bridge Models* (FDBM), a novel generative diffusion bridge framework driven by an approximation of the rich and non-Markovian fractional Brownian motion (fBM). Real stochastic processes exhibit a degree of memory effects (correlations in time), long-range dependencies, roughness and anomalous diffusion phenomena that are not captured in standard diffusion or bridge modeling due to the use of Brownian motion (BM). As a remedy, leveraging a recent Markovian approximation of fBM (MA-fBM), we construct FDBM that enable tractable inference while preserving the non-Markovian nature of fBM. We prove the existence of a coupling-preserving generative diffusion bridge and leverage it for future state prediction from paired training data. We then extend our formulation to the Schrödinger bridge problem and derive a principled loss function to learn the unpaired data translation. We evaluate FDBM on both tasks: predicting future protein conformations from aligned data, and unpaired image translation. In both settings, FDBM achieves superior performance compared to the Brownian baselines, yielding lower root mean squared deviation (RMSD) of C$_\alpha$ atomic positions in protein structure prediction and lower Fréchet Inception Distance (FID) in unpaired image translation.

ICML Conference 2024 Conference Paper

Bridging discrete and continuous state spaces: Exploring the Ehrenfest process in time-continuous diffusion models

Ludwig Winkler
Lorenz Richter
Manfred Opper

Generative modeling via stochastic processes has led to remarkable empirical results as well as to recent advances in their theoretical understanding. In principle, both space and time of the processes can be discrete or continuous. In this work, we study time-continuous Markov jump processes on discrete state spaces and investigate their correspondence to state-continuous diffusion processes given by SDEs. In particular, we revisit the $\textit{Ehrenfest process}$, which converges to an Ornstein-Uhlenbeck process in the infinite state space limit. Likewise, we can show that the time-reversal of the Ehrenfest process converges to the time-reversed Ornstein-Uhlenbeck process. This observation bridges discrete and continuous state spaces and allows to carry over methods from one to the respective other setting, such as for instance loss functions that lead to improved convergence. Additionally, we suggest an algorithm for training the time-reversal of Markov jump processes which relies on conditional expectations and can thus be directly related to denoising score matching. We demonstrate our methods in multiple convincing numerical experiments.

NeurIPS Conference 2024 Conference Paper

Generative Fractional Diffusion Models

Gabriel Nobis
Maximilian Springenberg
Marco Aversa
Michael Detzel
Rembert Daems
Roderick Murray-Smith
Shinichi Nakajima
Sebastian Lapuschkin

We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. In this paper, we replace BM with an approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index $H \in (0, 1)$, where $H=0. 5$ recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-fBM) and derive its reverse-time model, resulting in *generative fractional diffusion models* (GFDM). We characterize the forward dynamics using a continuous reparameterization trick and propose *augmented score matching* to efficiently learn the score function, which is partly known in closed form, at minimal added cost. The ability to drive our diffusion model via MA-fBM offers flexibility and control. $H \leq 0. 5$ enters the regime of *rough paths* whereas $H>0. 5$ regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering a promising alternative to traditional diffusion models

PDF Details DOI

ICLR Conference 2024 Conference Paper

Variational Inference for SDEs Driven by Fractional Noise

Rembert Daems
Manfred Opper
Guillaume Crevecoeur
Tolga Birdal

We present a novel variational framework for performing inference in (neural) stochastic differential equations (SDEs) driven by Markov-approximate fractional Brownian motion (fBM). SDEs offer a versatile tool for modeling real-world continuous-time dynamic systems with inherent noise and randomness. Combining SDEs with the powerful inference capabilities of variational methods, enables the learning of representative distributions through stochastic gradient descent. However, conventional SDEs typically assume the underlying noise to follow a Brownian motion (BM), which hinders their ability to capture long-term dependencies. In contrast, fractional Brownian motion (fBM) extends BM to encompass non-Markovian dynamics, but existing methods for inferring fBM parameters are either computationally demanding or statistically inefficient. In this paper, building upon the Markov approximation of fBM, we derive the evidence lower bound essential for efficient variational inference of posterior path measures, drawing from the well-established field of stochastic analysis. Additionally, we provide a closed-form expression for optimal approximation coefficients and propose to use neural networks to learn the drift, diffusion and control terms within our variational posterior, leading to the variational training of neural-SDEs. In this framework, we also optimize the Hurst index, governing the nature of our fractional noise. Beyond validation on synthetic data, we contribute a novel architecture for variational latent video prediction,—an approach that, to the best of our knowledge, enables the first variational neural-SDE application to video perception.

AAAI Conference 2019 Conference Paper

Efficient Gaussian Process Classification Using Pólya-Gamma Data Augmentation

Florian Wenzel
Théo Galy-Fajou
Christan Donner
Marius Kloft
Manfred Opper

We propose a scalable stochastic variational approach to GP classification building on Pólya-Gamma data augmentation and inducing points. Unlike former approaches, we obtain closed-form updates based on natural gradients that lead to efficient optimization. We evaluate the algorithm on real-world datasets containing up to 11 million data points and demonstrate that it is up to two orders of magnitude faster than the state-of-the-art while being competitive in terms of prediction performance.

UAI Conference 2019 Conference Paper

Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation

Théo Galy-Fajou
Florian Wenzel
Christian Donner
Manfred Opper

We propose a new scalable multi-class Gaussian process classification approach building on a novel modified softmax likelihood function. The new likelihood has two benefits: it leads to well-calibrated uncertainty estimates and allows for an efficient latent variable augmentation. The augmented model has the advantage that it is conditionally conjugate leading to a fast variational inference method via block coordinate ascent updates. Previous approaches suffered from a trade-off between uncertainty calibration and speed. Our experiments show that our method leads to well-calibrated uncertainty estimates and competitive predictive performance while being up to two orders faster than the state of the art.

UAI Conference 2018 Conference Paper

Efficient Bayesian Inference for a Gaussian Process Density Model

Christian Donner
Manfred Opper

We reconsider a nonparametric density model based on Gaussian processes. By augmenting the model with latent Pólya–Gamma random variables and a latent marked Poisson process we obtain a new likelihood which is conjugate to the model’s Gaussian process prior. The augmented posterior allows for efficient inference by Gibbs sampling and an approximate variational mean field approach. For the latter we utilise sparse GP approximations to tackle the infinite dimensionality of the problem. The performance of both algorithms and comparisons with other density estimators are demonstrated on artificial and real datasets with up to several thousand data points.

JMLR Journal 2018 Journal Article

Efficient Bayesian Inference of Sigmoidal Gaussian Cox Processes

Christian Donner
Manfred Opper

We present an approximate Bayesian inference approach for estimating the intensity of a inhomogeneous Poisson process, where the intensity function is modelled using a Gaussian process (GP) prior via a sigmoid link function. Augmenting the model using a latent marked Poisson process and Polya--Gamma random variables we obtain a representation of the likelihood which is conjugate to the GP prior. We estimate the posterior using a variational free--form mean field optimisation together with the framework of sparse GPs. Furthermore, as alternative approximation we suggest a sparse Laplace's method for the posterior, for which an efficient expectation--maximisation algorithm is derived to find the posterior's mode. Both algorithms compare well against exact inference obtained by a Markov Chain Monte Carlo sampler and standard variational Gauss approach solving the same model, while being one order of magnitude faster. Furthermore, the performance and speed of our method is competitive with that of another recently proposed Poisson process model based on a quadratic link function, while not being limited to GPs with squared exponential kernels and rectangular domains. [abs] [ pdf ][ bib ] &copy JMLR 2018. ( edit, beta )

NeurIPS Conference 2017 Conference Paper

Perturbative Black Box Variational Inference

Robert Bamler
Cheng Zhang
Manfred Opper
Stephan Mandt

Black box variational inference (BBVI) with reparameterization gradients triggered the exploration of divergence measures other than the Kullback-Leibler (KL) divergence, such as alpha divergences. In this paper, we view BBVI with generalized divergences as a form of estimating the marginal likelihood via biased importance sampling. The choice of divergence determines a bias-variance trade-off between the tightness of a bound on the marginal likelihood (low bias) and the variance of its gradient estimators. Drawing on variational perturbation theory of statistical physics, we use these insights to construct a family of new variational bounds. Enumerated by an odd integer order $K$, this family captures the standard KL bound for $K=1$, and converges to the exact marginal likelihood as $K\to\infty$. Compared to alpha-divergences, our reparameterization gradients have a lower variance. We show in experiments on Gaussian Processes and Variational Autoencoders that the new bounds are more mass covering, and that the resulting posterior covariances are closer to the true posterior and lead to higher likelihoods on held-out data.

NeurIPS Conference 2015 Conference Paper

A Tractable Approximation to Optimal Point Process Filtering: Application to Neural Encoding

Yuval Harel
Ron Meir
Manfred Opper

The process of dynamic state estimation (filtering) based on point process observations is in general intractable. Numerical sampling techniques are often practically useful, but lead to limited conceptual insight about optimal encoding/decoding strategies, which are of significant relevance to Computational Neuroscience. We develop an analytically tractable Bayesian approximation to optimal filtering based on point process observations, which allows us to introduce distributional assumptions about sensory cell properties, that greatly facilitates the analysis of optimal encoding in situations deviating from common assumptions of uniform coding. The analytic framework leads to insights which are difficult to obtain from numerical algorithms, and is consistent with experiments about the distribution of tuning curve centers. Interestingly, we find that the information gained from the absence of spikes may be crucial to performance.

NeurIPS Conference 2014 Conference Paper

Optimal Neural Codes for Control and Estimation

Alex Susemihl
Ron Meir
Manfred Opper

Agents acting in the natural world aim at selecting appropriate actions based on noisy and partial sensory observations. Many behaviors leading to decision making and action selection in a closed loop setting are naturally phrased within a control theoretic framework. Within the framework of optimal Control Theory, one is usually given a cost function which is minimized by selecting a control law based on the observations. While in standard control settings the sensors are assumed fixed, biological systems often gain from the extra flexibility of optimizing the sensors themselves. However, this sensory adaptation is geared towards control rather than perception, as is often assumed. In this work we show that sensory adaptation for control differs from sensory adaptation for perception, even for simple control setups. This implies, consistently with recent experimental results, that when studying sensory adaptation, it is essential to account for the task being performed.

NeurIPS Conference 2014 Conference Paper

Poisson Process Jumping between an Unknown Number of Rates: Application to Neural Spike Data

Florian Stimberg
Andreas Ruttor
Manfred Opper

We introduce a model where the rate of an inhomogeneous Poisson process is modified by a Chinese restaurant process. Applying a MCMC sampler to this model allows us to do posterior Bayesian inference about the number of states in Poisson-like data. Our sampler is shown to get accurate results for synthetic data and we apply it to V1 neuron spike data to find discrete firing rate states depending on the orientation of a stimulus.

NeurIPS Conference 2013 Conference Paper

Approximate Gaussian process inference for the drift function in stochastic differential equations

Andreas Ruttor
Philipp Batz
Manfred Opper

We introduce a nonparametric approach for estimating drift functions in systems of stochastic differential equations from incomplete observations of the state vector. Using a Gaussian process prior over the drift as a function of the state vector, we develop an approximate EM algorithm to deal with the unobserved, latent dynamics between observations. The posterior over states is approximated by a piecewise linearized process and the MAP estimation of the drift is facilitated by a sparse Gaussian process regression.

NeurIPS Conference 2013 Conference Paper

Approximate inference in latent Gaussian-Markov models from continuous time observations

Botond Cseke
Manfred Opper
Guido Sanguinetti

We propose an approximate inference algorithm for continuous time Gaussian-Markov process models with both discrete and continuous time likelihoods. We show that the continuous time limit of the expectation propagation algorithm exists and results in a hybrid fixed point iteration consisting of (1) expectation propagation updates for the discrete time terms and (2) variational updates for the continuous time term. We introduce corrections methods that improve on the marginals of the approximation. This approach extends the classical Kalman-Bucy smoothing procedure to non-Gaussian observations, enabling continuous-time inference in a variety of models, including spiking neuronal models (state-space models with point process observations) and box likelihood models. Experimental results on real and simulated data demonstrate high distributional accuracy and significant computational savings compared to discrete-time approaches in a neural application.

ICAPS Conference 2013 Conference Paper

Optimal Control as a Graphical Model Inference Problem

Hilbert J. Kappen
Vicenç Gómez
Manfred Opper

In this paper we show the identification between stochastic optimal control computation and probabilistic inference on a graphical model for certain class of control problems. We refer to these problems as Kullback-Leibler (KL) control problems. We illustrate how KL control can be used to model a multi-agent cooperative game for which optimal control can be approximated using belief propagation when exact inference is unfeasible.

JMLR Journal 2013 Journal Article

Perturbative Corrections for Approximate Inference in Gaussian Latent Variable Models

Manfred Opper
Ulrich Paquet
Ole Winther

Expectation Propagation (EP) provides a framework for approximate inference. When the model under consideration is over a latent Gaussian field, with the approximation being Gaussian, we show how these approximations can systematically be corrected. A perturbative expansion is made of the exact but intractable correction, and can be applied to the model's partition function and other moments of interest. The correction is expressed over the higher-order cumulants which are neglected by EP's local matching of moments. Through the expansion, we see that EP is correct to first order. By considering higher orders, corrections of increasing polynomial complexity can be applied to the approximation. The second order provides a correction in quadratic time, which we apply to an array of Gaussian process and Ising models. The corrections generalize to arbitrarily complex approximating families, which we illustrate on tree- structured Ising model approximations. Furthermore, they provide a polynomial-time assessment of the approximation error. We also provide both theoretical and practical insights on the exactness of the EP solution. [abs] [ pdf ][ bib ] &copy JMLR 2013. ( edit, beta )

NeurIPS Conference 2011 Conference Paper

Analytical Results for the Error in Filtering of Gaussian Processes

Alex Susemihl
Ron Meir
Manfred Opper

Bayesian filtering of stochastic stimuli has received a great deal of attention re- cently. It has been applied to describe the way in which biological systems dy- namically represent and make decisions about the environment. There have been no exact results for the error in the biologically plausible setting of inference on point process, however. We present an exact analysis of the evolution of the mean- squared error in a state estimation task using Gaussian-tuned point processes as sensors. This allows us to study the dynamics of the error of an optimal Bayesian decoder, providing insights into the limits obtainable in this task. This is done for Markovian and a class of non-Markovian Gaussian processes. We find that there is an optimal tuning width for which the error is minimized. This leads to a char- acterization of the optimal encoding for the setting as a function of the statistics of the stimulus, providing a mathematically sound primer for an ecological theory of sensory processing.

NeurIPS Conference 2011 Conference Paper

Inference in continuous-time change-point models

Florian Stimberg
Manfred Opper
Guido Sanguinetti
Andreas Ruttor

We consider the problem of Bayesian inference for continuous time multi-stable stochastic systems which can change both their diffusion and drift parameters at discrete times. We propose exact inference and sampling methodologies for two specific cases where the discontinuous dynamics is given by a Poisson process and a two-state Markovian switch. We test the methodology on simulated data, and apply it to two real data sets in finance and systems biology. Our experimental results show that the approach leads to valid inferences and non-trivial insights.

NeurIPS Conference 2010 Conference Paper

Approximate inference in continuous time Gaussian-Jump processes

Manfred Opper
Andreas Ruttor
Guido Sanguinetti

We present a novel approach to inference in conditionally Gaussian continuous time stochastic processes, where the latent process is a Markovian jump process. We first consider the case of jump-diffusion processes, where the drift of a linear stochastic differential equation can jump at arbitrary time points. We derive partial differential equations for exact inference and present a very efficient mean field approximation. By introducing a novel lower bound on the free energy, we then generalise our approach to Gaussian processes with arbitrary covariance, such as the non-Markovian RBF covariance. We present results on both simulated and real data, showing that the approach is very accurate in capturing latent dynamics and can be useful in a number of real data modelling tasks.

JMLR Journal 2009 Journal Article

Perturbation Corrections in Approximate Inference: Mixture Modelling Applications

Ulrich Paquet
Ole Winther
Manfred Opper

Bayesian inference is intractable for many interesting models, making deterministic algorithms for approximate inference highly desirable. Unlike stochastic methods, which are exact in the limit, the accuracy of these approaches cannot be reasonably judged. In this paper we show how low order perturbation corrections to an expectation-consistent (EC) approximation can provide the necessary tools to ameliorate inference accuracy, and to give an indication of the quality of approximation without having to resort to Monte Carlo methods. Further comparisons are given with variational Bayes and parallel tempering (PT) combined with thermodynamic integration on a Gaussian mixture model. To obtain practical results we further generalize PT to temper from arbitrary distributions rather than a prior in Bayesian inference. [abs] [ pdf ][ bib ] &copy JMLR 2009. ( edit, beta )

NeurIPS Conference 2008 Conference Paper

Improving on Expectation Propagation

Manfred Opper
Ulrich Paquet
Ole Winther

We develop as series of corrections to Expectation Propagation (EP), which is one of the most popular methods for approximate probabilistic inference. These corrections can lead to improvements of the inference approximation or serve as a sanity check, indicating when EP yields unrealiable results.

NeurIPS Conference 2007 Conference Paper

Variational Inference for Diffusion Processes

Cédric Archambeau
Manfred Opper
Yuan Shen
Dan Cornford
John Shawe-Taylor

Diffusion processes are a family of continuous-time continuous-state stochastic processes that are in general only partially observed. The joint estimation of the forcing parameters and the system noise (volatility) in these dynamical systems is a crucial, but non-trivial task, especially when the system is nonlinear and multi-modal. We propose a variational treatment of diffusion processes, which allows us to estimate these parameters by simple gradient techniques and which is computationally less demanding than most MCMC approaches. Furthermore, our parameter inference scheme does not break down when the time step gets smaller, unlike most current approaches. Finally, we show how a cheap estimate of the posterior over the parameters can be constructed based on the variational free energy.

NeurIPS Conference 2007 Conference Paper

Variational inference for Markov jump processes

Manfred Opper
Guido Sanguinetti

Markov jump processes play an important role in a large number of application domains. However, realistic systems are analytically intractable and they have traditionally been analysed using simulation based techniques, which do not provide a framework for statistical inference. We propose a mean field approximation to perform posterior inference and parameter estimation. The approximation allows a practical solution to the inference problem, {while still retaining a good degree of accuracy. } We illustrate our approach on two biologically motivated systems.

NeurIPS Conference 2005 Conference Paper

An Approximate Inference Approach for the PCA Reconstruction Error

Manfred Opper

The problem of computing a resample estimate for the reconstruction error in PCA is reformulated as an inference problem with the help of the replica method. Using the expectation consistent (EC) approximation, the intractable inference problem can be solved efficiently using only two variational parameters. A perturbative correction to the result is computed and an alternative simplified derivation is also presented.

JMLR Journal 2005 Journal Article

Expectation Consistent Approximate Inference

Manfred Opper
Ole Winther

We propose a novel framework for approximations to intractable probabilistic models which is based on a free energy formulation. The approximation can be understood as replacing an average over the original intractable distribution with a tractable one. It requires two tractable probability distributions which are made consistent on a set of moments and encode different features of the original intractable distribution. In this way we are able to use Gaussian approximations for models with discrete or bounded variables which allow us to include non-trivial correlations. These are neglected in many other methods. We test the framework on toy benchmark problems for binary variables on fully connected graphs and 2D grids and compare with other methods, such as loopy belief propagation. Good performance is already achieved by using single nodes as tractable substructures. Significant improvements are obtained when a spanning tree is used instead. [abs] [ pdf ][ bib ] &copy JMLR 2005. ( edit, beta )

NeurIPS Conference 2004 Conference Paper

Expectation Consistent Free Energies for Approximate Inference

Manfred Opper
Ole Winther

We propose a novel a framework for deriving approximations for in- tractable probabilistic models. This framework is based on a free energy (negative log marginal likelihood) and can be seen as a generalization of adaptive TAP [1, 2, 3] and expectation propagation (EP) [4, 5]. The free energy is constructed from two approximating distributions which encode different aspects of the intractable model such a single node con- straints and couplings and are by construction consistent on a chosen set of moments. We test the framework on a difficult benchmark problem with binary variables on fully connected graphs and 2D grid graphs. We find good performance using sets of moments which either specify fac- torized nodes or a spanning tree on the nodes (structured approximation). Surprisingly, the Bethe approximation gives very inferior results even on grids.

JMLR Journal 2003 Journal Article

An Approximate Analytical Approach to Resampling Averages (Kernel Machines Section)

Dörthe Malzahn
Manfred Opper

Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our approach uses a combination of the replica "trick" of statistical physics and the TAP approach for approximate Bayesian inference. We demonstrate our approach on regression with Gaussian processes. A comparison with averages obtained by Monte-Carlo sampling shows that our method achieves good accuracy. [abs] [ pdf ][ ps.gz ][ ps ]

NeurIPS Conference 2003 Conference Paper

Approximate Analytical Bootstrap Averages for Support Vector Classifiers

Dörthe Malzahn
Manfred Opper

We compute approximate analytical bootstrap averages for support vec- tor classiﬁcation using a combination of the replica method of statistical physics and the TAP approach for approximate inference. We test our method on a few datasets and compare it with exact averages obtained by extensive Monte-Carlo sampling.

NeurIPS Conference 2003 Conference Paper

Variational Linear Response

Manfred Opper
Ole Winther

A general linear response method for deriving improved estimates of cor- relations in the variational Bayes framework is presented. Three applica- tions are given and it is discussed how to use linear response as a general principle for improving mean ﬁeld approximations.

NeurIPS Conference 2002 Conference Paper

A Statistical Mechanics Approach to Approximate Analytical Bootstrap Averages

Dörthe Malzahn
Manfred Opper

We apply the replica method of Statistical Physics combined with a vari- ational method to the approximate analytical computation of bootstrap averages for estimating the generalization error. We demonstrate our ap- proach on regression with Gaussian processes and compare our results with averages obtained by Monte-Carlo sampling.

NeurIPS Conference 2001 Conference Paper

A Variational Approach to Learning Curves

Dörthe Malzahn
Manfred Opper

We combine the replica approach from statistical physics with a varia- tional approach to analyze learning curves analytically. We apply the method to Gaussian process regression. As a main result we derive ap- proximative relations between empirical error measures, the generaliza- tion error and the posterior variance.

NeurIPS Conference 2001 Conference Paper

Asymptotic Universality for Learning Curves of Support Vector Machines

Manfred Opper
Robert Urbanczik

Using methods of Statistical Physics, we investigate the rOle of model complexity in learning with support vector machines (SVMs). We show the advantages of using SVMs with kernels of infinite complexity on noisy target rules, which, in contrast to common theoretical beliefs, are found to achieve optimal general(cid: 173) ization error although the training error does not converge to the generalization error. Moreover, we find a universal asymptotics of the learning curves which only depend on the target rule but not on the SVM kernel.

NeurIPS Conference 2001 Conference Paper

TAP Gibbs Free Energy, Belief Propagation and Sparsity

Lehel Csató
Manfred Opper
Ole Winther

The adaptive TAP Gibbs free energy for a general densely connected probabilistic model with quadratic interactions and arbritary single site constraints is derived. We show how a speciﬁc sequential minimization of the free energy leads to a generalization of Minka’s expectation propa- gation. Lastly, we derive a sparse representation version of the sequential algorithm. The usefulness of the approach is demonstrated on classiﬁca- tion and density estimation with Gaussian processes and on an indepen- dent component analysis problem.

NeurIPS Conference 2000 Conference Paper

Learning Curves for Gaussian Processes Regression: A Framework for Good Approximations

Dörthe Malzahn
Manfred Opper

Based on a statistical mechanics approach, we develop a method for approximately computing average case learning curves for Gaus(cid: 173) sian process regression models. The approximation works well in the large sample size limit and for arbitrary dimensionality of the input space. We explain how the approximation can be systemati(cid: 173) cally improved and argue that similar techniques can be applied to general likelihood models.

NeurIPS Conference 2000 Conference Paper

Sparse Representation for Gaussian Process Models

Lehel Csató
Manfred Opper

We develop an approach for a sparse representation for Gaussian Process (GP) models in order to overcome the limitations of GPs caused by large data sets. The method is based on a combination of a Bayesian online al(cid: 173) gorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the model. Experi(cid: 173) mental results on toy examples and large real-world data sets indicate the efficiency of the approach.

NeurIPS Conference 1999 Conference Paper

Efficient Approaches to Gaussian Process Classification

Lehel Csató
Ernest Fokoué
Manfred Opper
Bernhard Schottky
Ole Winther

We present three simple approximations for the calculation of the posterior mean in Gaussian Process classification. The first two methods are related to mean field ideas known in Statistical Physics. The third approach is based on Bayesian online approach which was motivated by recent results in the Statistical Mechanics of Neural Networks. We present simulation results showing: 1. that the mean field Bayesian evidence may be used for hyperparameter tuning and 2. that the online approach may achieve a low training error fast.

NeurIPS Conference 1998 Conference Paper

Finite-Dimensional Approximation of Gaussian Processes

Giancarlo Ferrari-Trecate
Christopher Williams
Manfred Opper

Gaussian process (GP) prediction suffers from O(n3) scaling with the data set size n. By using a finite-dimensional basis to approximate the GP predictor, the computational complexity can be reduced. We de(cid: 173) rive optimal finite-dimensional predictors under a number of assump(cid: 173) tions, and show the superiority of these predictors over the Projected Bayes Regression method (which is asymptotically optimal). We also show how to calculate the minimal model size for a given n. The calculations are backed up by numerical experiments.

NeurIPS Conference 1998 Conference Paper

General Bounds on Bayes Errors for Regression with Gaussian Processes

Manfred Opper
Francesco Vivarelli

Based on a simple convexity lemma, we develop bounds for differ(cid: 173) ent types of Bayesian prediction errors for regression with Gaussian processes. The basic bounds are formulated for a fixed training set. Simpler expressions are obtained for sampling from an input distri(cid: 173) bution which equals the weight function of the covariance kernel, yielding asymptotically tight results. The results are compared with numerical experiments.

NeurIPS Conference 1998 Conference Paper

Mean Field Methods for Classification with Gaussian Processes

Manfred Opper
Ole Winther

We discuss the application of TAP mean field methods known from the Statistical Mechanics of disordered systems to Bayesian classifi(cid: 173) cation models with Gaussian processes. In contrast to previous ap(cid: 173) proaches, no knowledge about the distribution of inputs is needed. Simulation results for the Sonar data set are given. 1 Modeling with Gaussian Processes Bayesian models which are based on Gaussian prior distributions on function spaces are promising non-parametric statistical tools. They have been recently introduced into the Neural Computation community (Neal 1996, Williams & Rasmussen 1996, Mackay 1997). To give their basic definition, we assume that the likelihood of the output or target variable T for a given input s E RN can be written in the form p(Tlh(s)) where h: RN --+ R is a priori assumed to be a Gaussian random field. If we assume fields with zero prior mean, the statistics of h is entirely defined by the second order correlations C(s, S') == E[h(s)h(S')], where E denotes expectations 310 M Opper and 0. Winther with respect to the prior. Interesting examples are

NeurIPS Conference 1996 Conference Paper

A Mean Field Algorithm for Bayes Learning in Large Feed-forward Neural Networks

Manfred Opper
Ole Winther

We present an algorithm which is expected to realise Bayes optimal predictions in large feed-forward networks. It is based on mean field methods developed within statistical mechanics of disordered sys(cid: 173) tems. We give a derivation for the single layer perceptron and show that the algorithm also provides a leave-one-out cross-validation test of the predictions. Simulations show excellent agreement with theoretical results of statistical mechanics.

NeurIPS Conference 1996 Conference Paper

Dynamics of Training

Siegfried Bös
Manfred Opper

A new method to calculate the full training process of a neural net(cid: 173) work is introduced. No sophisticated methods like the replica trick are used. The results are directly related to the actual number of training steps. Some results are presented here, like the maximal learning rate, an exact description of early stopping, and the neces(cid: 173) sary number of training steps. Further problems can be addressed with this approach.

NeurIPS Conference 1991 Conference Paper

Estimating Average-Case Learning Curves Using Bayesian, Statistical Physics and VC Dimension Methods

David Haussler
Michael Kearns
Manfred Opper
Robert Schapire

In this paper we investigate an average-case model of concept learning, and give results that place the popular statistical physics and VC dimension theories of learning curve behavior in a common framework.