Author name cluster

Aki Vehtari

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

31 papers

2 author rows

TMLR Journal 2026 Journal Article

Amortized Bayesian Workflow

Chengkun Li
Aki Vehtari
Paul-Christian Bürkner
Stefan T. Radev
Luigi Acerbi
Marvin Schmitt

Bayesian inference often faces a trade-off between computational speed and sampling accuracy. We propose an adaptive workflow that integrates rapid amortized inference with gold-standard MCMC techniques to achieve a favorable combination of both speed and accuracy when performing inference on many observed datasets. Our approach uses principled diagnostics to guide the choice of inference method for each dataset, moving along the Pareto front from fast amortized sampling via generative neural networks to slower but guaranteed-accurate MCMC when needed. By reusing computations across steps, our workflow synergizes amortized and MCMC-based inference. We demonstrate the effectiveness of this integrated approach on several synthetic and real-world problems with tens of thousands of datasets, showing efficiency gains while maintaining high posterior quality.

PDF Details

JMLR Journal 2024 Journal Article

A Framework for Improving the Reliability of Black-box Variational Inference

Manushi Welandawe
Michael Riis Andersen
Aki Vehtari
Jonathan H. Huggins

Black-box variational inference (BBVI) now sees widespread use in machine learning and statistics as a fast yet flexible alternative to Markov chain Monte Carlo methods for approximate Bayesian inference. However, stochastic optimization methods for BBVI remain unreliable and require substantial expertise and hand-tuning to apply effectively. In this paper, we propose robust and automated black-box VI (RABVI), a framework for improving the reliability of BBVI optimization. RABVI is based on rigorously justified automation techniques, includes just a small number of intuitive tuning parameters, and detects inaccurate estimates of the optimal variational approximation. RABVI adaptively decreases the learning rate by detecting convergence of the fixed--learning-rate iterates, then estimates the symmetrized Kullback--Leibler (KL) divergence between the current variational approximation and the optimal one. It also employs a novel optimization termination criterion that enables the user to balance desired accuracy against computational cost by comparing (i) the predicted relative decrease in the symmetrized KL divergence if a smaller learning were used and (ii) the predicted computation required to converge with the smaller learning rate. We validate the robustness and accuracy of RABVI through carefully designed simulation studies and on a diverse set of real-world model and data examples. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2024. ( edit, beta )

PDF Details

JMLR Journal 2024 Journal Article

Pareto Smoothed Importance Sampling

Aki Vehtari
Daniel Simpson
Andrew Gelman
Yuling Yao
Jonah Gabry

Importance weighting is a general way to adjust Monte Carlo integration to account for draws from the wrong distribution, but the resulting estimate can be highly variable when the importance ratios have a heavy right tail. This routinely occurs when there are aspects of the target distribution that are not well captured by the approximating distribution, in which case more stable estimates can be obtained by modifying extreme importance ratios. We present a new method for stabilizing importance weights using a generalized Pareto distribution fit to the upper tail of the distribution of the simulated importance ratios. The method, which empirically performs better than existing methods for stabilizing importance sampling estimates, includes stabilized effective sample size estimates, Monte Carlo error estimates, and convergence diagnostics. The presented Pareto $\hat{k}$ finite sample convergence rate diagnostic is useful for any Monte Carlo estimator. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2024. ( edit, beta )

PDF Details

YNIMG Journal 2022 Journal Article

Atlas of type 2 dopamine receptors in the human brain: Age and sex dependent variability in a large PET cohort

Tuulia Malén
Tomi Karjalainen
Janne Isojärvi
Aki Vehtari
Paul-Christian Bürkner
Vesa Putkinen
Valtteri Kaasinen
Jarmo Hietala

BACKGROUND: The dopamine system contributes to a multitude of functions ranging from reward and motivation to learning and movement control, making it a key component in goal-directed behavior. Altered dopaminergic function is observed in neurological and psychiatric conditions. Numerous factors have been proposed to influence dopamine function, but due to small sample sizes and heterogeneous data analysis methods in previous studies their specific and joint contributions remain unresolved. METHODS: R) availability in the human brain. We analyzed a large historical dataset (n=156, 120 males and 36 females) of [11C]raclopride PET scans performed between 2004 and 2018. RESULTS: R. The observed effects were independent of regional volumes. These results were validated using two different spatial normalization methods, and the age and sex effects also replicated in an independent sample (n=135). CONCLUSIONS: R expression.

Details DOI

JMLR Journal 2022 Journal Article

Pathfinder: Parallel quasi-Newton variational inference

Lu Zhang
Bob Carpenter
Andrew Gelman
Aki Vehtari

We propose Pathfinder, a variational method for approximately sampling from differentiable probability densities. Starting from a random initialization, Pathfinder locates normal approximations to the target density along a quasi-Newton optimization path, with local covariance estimated using the inverse Hessian estimates produced by the optimizer. Pathfinder returns draws from the approximation with the lowest estimated Kullback-Leibler (KL) divergence to the target distribution. We evaluate Pathfinder on a wide range of posterior distributions, demonstrating that its approximate draws are better than those from automatic differentiation variational inference (ADVI) and comparable to those produced by short chains of dynamic Hamiltonian Monte Carlo (HMC), as measured by 1-Wasserstein distance. Compared to ADVI and short dynamic HMC runs, Pathfinder requires one to two orders of magnitude fewer log density and gradient evaluations, with greater reductions for more challenging posteriors. Importance resampling over multiple runs of Pathfinder improves the diversity of approximate draws, reducing 1-Wasserstein distance further and providing a measure of robustness to optimization failures on plateaus, saddle points, or in minor modes. The Monte Carlo KL divergence estimates are embarrassingly parallelizable in the core Pathfinder algorithm, as are multiple runs in the resampling version, further increasing Pathfinder's speed advantage with multiple cores. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2022. ( edit, beta )

PDF Details

JMLR Journal 2022 Journal Article

Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors

Yuling Yao
Aki Vehtari
Andrew Gelman

When working with multimodal Bayesian posterior distributions, Markov chain Monte Carlo (MCMC) algorithms have difficulty moving between modes, and default variational or mode-based approximate inferences will understate posterior uncertainty. And, even if the most important modes can be found, it is difficult to evaluate their relative weights in the posterior. Here we propose an approach using parallel runs of MCMC, variational, or mode-based inference to hit as many modes or separated regions as possible and then combine these using Bayesian stacking, a scalable method for constructing a weighted average of distributions. The result from stacking efficiently samples from multimodal posterior distribution, minimizes cross validation prediction error, and represents the posterior uncertainty better than variational inference, but it is not necessarily equivalent, even asymptotically, to fully Bayesian inference. We present theoretical consistency with an example where the stacked inference approximates the true data generating process from the misspecified model and a non-mixing sampler, from which the predictive performance is better than full Bayesian inference, hence the multimodality can be considered a blessing rather than a curse under model misspecification. We demonstrate practical implementation in several model families: latent Dirichlet allocation, Gaussian process regression, hierarchical regression, horseshoe variable selection, and neural networks. [abs] [ pdf ][ bib ] &copy JMLR 2022. ( edit, beta )

PDF Details

NeurIPS Conference 2021 Conference Paper

Challenges and Opportunities in High Dimensional Variational Inference

Akash Kumar Dhaka
Alejandro Catalina
Manushi Welandawe
Michael R. Andersen
Jonathan Huggins
Aki Vehtari

Current black-box variational inference (BBVI) methods require the user to make numerous design choices – such as the selection of variational objective and approximating family – yet there is little principled guidance on how to do so. We develop a conceptual framework and set of experimental tools to understand the effects of these choices, which we leverage to propose best practices for maximizing posterior approximation accuracy. Our approach is based on studying the pre-asymptotic tail behavior of the density ratios between the joint distribution and the variational approximation, then exploiting insights and tools from the importance sampling literature. Our framework and supporting experiments help to distinguish between the behavior of BBVI methods for approximating low-dimensional versus moderate-to-high-dimensional posteriors. In the latter case, we show that mass-covering variational objectives are difficult to optimize and do not improve accuracy, but flexible variational families can improve accuracy and the effectiveness of importance sampling – at the cost of additional optimization challenges. Therefore, for moderate-to-high-dimensional posteriors we recommend using the (mode-seeking) exclusive KL divergence since it is the easiest to optimize, and improving the variational family or using model parameter transformations to make the posterior and optimal variational approximation more similar. On the other hand, in low-dimensional settings, we show that heavy-tailed variational families and mass-covering divergences are effective and can increase the chances that the approximation can be improved by importance sampling.

PDF Details

UAI Conference 2021 Conference Paper

Uncertainty-aware sensitivity analysis using Rényi divergences

Topi Paananen
Michael Riis Andersen
Aki Vehtari

For nonlinear supervised learning models, assessing the importance of predictor variables or their interactions is not straightforward because importance can vary in the domain of the variables. Importance can be assessed locally with sensitivity analysis using general methods that rely on the model’s predictions or their derivatives. In this work, we extend derivative based sensitivity analysis to a Bayesian setting by differentiating the R\’{e}nyi divergence of a model’s predictive distribution. By utilising the predictive distribution instead of a point prediction, the model uncertainty is taken into account in a principled way. Our empirical results on simulated and real data sets demonstrate accurate and reliable identification of important variables and interaction effects compared to alternative methods.

Details

UAI Conference 2020 Conference Paper

Batch simulations and uncertainty quantification in Gaussian process surrogate approximate Bayesian computation

Marko Järvenpää
Aki Vehtari
Pekka Marttinen

The computational efficiency of approximate Bayesian computation (ABC) has been improved by using surrogate models such as Gaussian processes (GP). In one such promising framework the discrepancy between the simulated and observed data is modelled with a GP which is further used to form a model-based estimator for the intractable posterior. In this article we improve this approach in several ways. We develop batch-sequential Bayesian experimental design strategies to parallellise the expensive simulations. In earlier work only sequential strategies have been used. Current surrogate-based ABC methods also do not fully account the uncertainty due to the limited budget of simulations as they output only a point estimate of the ABC posterior. We propose a numerical method to fully quantify the uncertainty in, for example, ABC posterior moments. We also provide some new analysis on the GP modelling assumptions in the resulting improved framework called Bayesian ABC and discuss its connection to Bayesian quadrature (BQ) and Bayesian optimisation (BO). Experiments with toy and real-world simulation models demonstrate advantages of the proposed techniques.

Details

JMLR Journal 2020 Journal Article

Expectation Propagation as a Way of Life: A Framework for Bayesian Inference on Partitioned Data

Aki Vehtari
Andrew Gelman
Tuomas Sivula
Pasi Jylänki
Dustin Tran
Swupnil Sahai
Paul Blomstedt
John P. Cunningham

A common divide-and-conquer approach for Bayesian computation with big data is to partition the data, perform local inference for each piece separately, and combine the results to obtain a global posterior approximation. While being conceptually and computationally appealing, this method involves the problematic need to also split the prior for the local inferences; these weakened priors may not provide enough regularization for each separate computation, thus eliminating one of the key advantages of Bayesian methods. To resolve this dilemma while still retaining the generalizability of the underlying local inference method, we apply the idea of expectation propagation (EP) as a framework for distributed Bayesian inference. The central idea is to iteratively update approximations to the local likelihoods given the state of the other approximations and the prior. The present paper has two roles: we review the steps that are needed to keep EP algorithms numerically stable, and we suggest a general approach, inspired by EP, for approaching data partitioning problems in a way that achieves the computational benefits of parallelism while allowing each local update to make use of relevant information from the other sites. In addition, we demonstrate how the method can be applied in a hierarchical context to make use of partitioning of both data and parameters. The paper describes a general algorithmic framework, rather than a specific algorithm, and presents an example implementation for it. [abs] [ pdf ][ bib ] &copy JMLR 2020. ( edit, beta )

PDF Details

NeurIPS Conference 2020 Conference Paper

Hamiltonian Monte Carlo using an adjoint-differentiated Laplace approximation: Bayesian inference for latent Gaussian models and beyond

Charles Margossian
Aki Vehtari
Daniel Simpson
Raj Agrawal

Gaussian latent variable models are a key class of Bayesian hierarchical models with applications in many fields. Performing Bayesian inference on such models can be challenging as Markov chain Monte Carlo algorithms struggle with the geometry of the resulting posterior distribution and can be prohibitively slow. An alternative is to use a Laplace approximation to marginalize out the latent Gaussian variables and then integrate out the remaining hyperparameters using dynamic Hamiltonian Monte Carlo, a gradient-based Markov chain Monte Carlo sampler. To implement this scheme efficiently, we derive a novel adjoint method that propagates the minimal information needed to construct the gradient of the approximate marginal likelihood. This strategy yields a scalable differentiation method that is orders of magnitude faster than state of the art differentiation techniques when the hyperparameters are high dimensional. We prototype the method in the probabilistic programming framework Stan and test the utility of the embedded Laplace approximation on several models, including one where the dimension of the hyperparameter is ∼6, 000. Depending on the cases, the benefits can include an alleviation of the geometric pathologies that frustrate Hamiltonian Monte Carlo and a dramatic speed-up.

PDF Details

YNIMG Journal 2020 Journal Article

Interindividual variability and lateralization of μ-opioid receptors in the human brain

Tatu Kantonen
Tomi Karjalainen
Janne Isojärvi
Pirjo Nuutila
Jouni Tuisku
Juha Rinne
Jarmo Hietala
Valtteri Kaasinen

Alterations in the brain’s μ-opioid receptor (MOR) system have been associated with several neuropsychiatric disorders. Central MOR availability also varies considerably in healthy individuals. Multiple epidemiological factors have been proposed to influence the MOR system, but due to small sample sizes the magnitude of their influence remains inconclusive. We compiled [11C]carfentanil positron emission tomography scans from 204 individuals with no neurologic or psychiatric disorders, and estimated the effects of sex, age, body mass index (BMI) and smoking on [11C]carfentanil binding potential using between-subject regression analysis. We also examined hemispheric differences in MOR availability. Older age was associated with increase in MOR availability in frontotemporal areas but decrease in amygdala, thalamus, and nucleus accumbens. The age-dependent increase was stronger in males. MOR availability was globally lowered in smokers but independent of BMI. Finally, MOR availability was higher in the right versus the left hemisphere. The presently observed variation in MOR availability may explain why some individuals are prone to develop MOR-linked pathological states, such as chronic pain or psychiatric disorders. Lateralized MOR system may reflect hemispheric work specialization in central emotion and pain processes.

Details DOI

NeurIPS Conference 2020 Conference Paper

Robust, Accurate Stochastic Optimization for Variational Inference

Akash Kumar Dhaka
Alejandro Catalina
Michael R. Andersen
Måns Magnusson
Jonathan Huggins
Aki Vehtari

We examine the accuracy of black box variational posterior approximations for parametric models in a probabilistic programming context. The performance of these approximations depends on (1) how well the variational family approximates the true posterior distribution, (2) the choice of divergence, and (3) the optimization of the variational objective. We show that even when the true variational family is used, high-dimensional posteriors can be very poorly approximated using common stochastic gradient descent (SGD) optimizers. Motivated by recent theory, we propose a simple and parallel way to improve SGD estimates for variational inference. The approach is theoretically motivated and comes with a diagnostic for convergence and a novel stopping rule, which is robust to noisy objective functions evaluations. We show empirically, the new workflow works well on a diverse set of models and datasets, or warns if the stochastic optimization fails or if the used variational distribution is not good.

PDF Details

ICML Conference 2019 Conference Paper

Active Learning for Decision-Making from Imbalanced Observational Data

Iiris Sundin
Peter Schulam 0001
Eero Siivola
Aki Vehtari
Suchi Saria
Samuel Kaski

Machine learning can help personalized decision support by learning models to predict individual treatment effects (ITE). This work studies the reliability of prediction-based decision-making in a task of deciding which action $a$ to take for a target unit after observing its covariates $\tilde{x}$ and predicted outcomes $\hat{p}(\tilde{y} \mid \tilde{x}, a)$. An example case is personalized medicine and the decision of which treatment to give to a patient. A common problem when learning these models from observational data is imbalance, that is, difference in treated/control covariate distributions, which is known to increase the upper bound of the expected ITE estimation error. We propose to assess the decision-making reliability by estimating the ITE model’s Type S error rate, which is the probability of the model inferring the sign of the treatment effect wrong. Furthermore, we use the estimated reliability as a criterion for active learning, in order to collect new (possibly expensive) observations, instead of making a forced choice based on unreliable predictions. We demonstrate the effectiveness of this decision-making aware active learning in two decision-making tasks: in simulated data with binary outcomes and in a medical dataset with synthetic and continuous treatment outcomes.

Details

ICML Conference 2019 Conference Paper

Bayesian leave-one-out cross-validation for large data

Måns Magnusson
Michael Riis Andersen
Johan Jonasson
Aki Vehtari

Model inference, such as model comparison, model checking, and model selection, is an important part of model development. Leave-one-out cross-validation (LOO) is a general approach for assessing the generalizability of a model, but unfortunately, LOO does not scale well to large datasets. We propose a combination of using approximate inference techniques and probability-proportional-to-size-sampling (PPS) for fast LOO model evaluation for large datasets. We provide both theoretical and empirical results showing good properties for large data.

Details

JMLR Journal 2018 Journal Article

ELFI: Engine for Likelihood-Free Inference

Jarno Lintusaari
Henri Vuollekoski
Antti Kangasrääsiö
Kusti Skytén
Marko Järvenpää
Pekka Marttinen
Michael U. Gutmann
Aki Vehtari

Engine for Likelihood-Free Inference (ELFI) is a Python software library for performing likelihood-free inference (LFI). ELFI provides a convenient syntax for arranging components in LFI, such as priors, simulators, summaries or distances, to a network called ELFI graph. The components can be implemented in a wide variety of languages. The stand-alone ELFI graph can be used with any of the available inference methods without modifications. A central method implemented in ELFI is Bayesian Optimization for Likelihood-Free Inference (BOLFI), which has recently been shown to accelerate likelihood-free inference up to several orders of magnitude by surrogate-modelling the distance. ELFI also has an inbuilt support for output data storing for reuse and analysis, and supports parallelization of computation from multiple cores up to a cluster environment. ELFI is designed to be extensible and provides interfaces for widening its functionality. This makes the adding of new inference methods to ELFI straightforward and automatically compatible with the inbuilt features. [abs] [ pdf ][ bib ] [ webpage ] [ code ] &copy JMLR 2018. ( edit, beta )

PDF Details

ICML Conference 2018 Conference Paper

Yes, but Did It Work? : Evaluating Variational Inference

Yuling Yao
Aki Vehtari
Daniel Simpson
Andrew Gelman

While it’s always possible to compute a variational approximation to a posterior distribution, it can be difficult to discover problems with this approximation. We propose two diagnostic algorithms to alleviate this problem. The Pareto-smoothed importance sampling (PSIS) diagnostic gives a goodness of fit measurement for joint distributions, while simultaneously improving the error in the estimate. The variational simulation-based calibration (VSBC) assesses the average performance of point estimates.

Details

JMLR Journal 2017 Journal Article

Bayesian Inference for Spatio-temporal Spike-and-Slab Priors

Michael Riis Andersen
Aki Vehtari
Ole Winther
Lars Kai Hansen

In this work, we address the problem of solving a series of underdetermined linear inverse problemblems subject to a sparsity constraint. We generalize the spike-and-slab prior distribution to encode a priori correlation of the support of the solution in both space and time by imposing a transformed Gaussian process on the spike-and-slab probabilities. An expectation propagation (EP) algorithm for posterior inference under the proposed model is derived. For large scale problems, the standard EP algorithm can be prohibitively slow. We therefore introduce three different approximation schemes to reduce the computational complexity. Finally, we demonstrate the proposed model using numerical experiments based on both synthetic and real data sets. [abs] [ pdf ][ bib ] &copy JMLR 2017. ( edit, beta )

PDF Details

YNIMG Journal 2017 Journal Article

Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex

Juha Salmi
Olli-Pekka Koistinen
Enrico Glerean
Pasi Jylänki
Aki Vehtari
Iiro P. Jääskeläinen
Sasu Mäkelä
Lauri Nummenmaa

During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.

Details DOI

JMLR Journal 2016 Journal Article

Bayesian Leave-One-Out Cross-Validation Approximations for Gaussian Latent Variable Models

Aki Vehtari
Tommi Mononen
Ville Tolvanen
Tuomas Sivula
Ole Winther

The future predictive performance of a Bayesian model can be estimated using Bayesian cross-validation. In this article, we consider Gaussian latent variable models where the integration over the latent values is approximated using the Laplace method or expectation propagation (EP). We study the properties of several Bayesian leave-one-out (LOO) cross-validation approximations that in most cases can be computed with a small additional cost after forming the posterior approximation given the full data. Our main objective is to assess the accuracy of the approximative LOO cross-validation estimators. That is, for each method (Laplace and EP) we compare the approximate fast computation with the exact brute force LOO computation. Secondarily, we evaluate the accuracy of the Laplace and EP approximations themselves against a ground truth established through extensive Markov chain Monte Carlo simulation. Our empirical results show that the approach based upon a Gaussian approximation to the LOO marginal distribution (the so-called cavity distribution) gives the most accurate and reliable results among the fast methods. [abs] [ pdf ][ bib ] &copy JMLR 2016. ( edit, beta )

PDF Details

JMLR Journal 2014 Journal Article

Expectation Propagation for Neural Networks with Sparsity-Promoting Priors

Pasi Jylänki
Aapo Nummenmaa
Aki Vehtari

We propose a novel approach for nonlinear regression using a two-layer neural network (NN) model structure with sparsity- favoring hierarchical priors on the network weights. We present an expectation propagation (EP) approach for approximate integration over the posterior distribution of the weights, the hierarchical scale parameters of the priors, and the residual scale. Using a factorized posterior approximation we derive a computationally efficient algorithm, whose complexity scales similarly to an ensemble of independent sparse linear models. The approach enables flexible definition of weight priors with different sparseness properties such as independent Laplace priors with a common scale parameter or Gaussian automatic relevance determination (ARD) priors with different relevance parameters for all inputs. The approach can be extended beyond standard activation functions and NN model structures to form flexible nonlinear predictors from multiple sparse linear models. The effects of the hierarchical priors and the predictive performance of the algorithm are assessed using both simulated and real-world data. Comparisons are made to two alternative models with ARD priors: a Gaussian process with a NN covariance function and marginal maximum a posteriori estimates of the relevance parameters, and a NN with Markov chain Monte Carlo integration over all the unknown model parameters. [abs] [ pdf ][ bib ] &copy JMLR 2014. ( edit, beta )

PDF Details

JMLR Journal 2013 Journal Article

GPstuff: Bayesian Modeling with Gaussian Processes

Jarno Vanhatalo
Jaakko Riihimäki
Jouni Hartikainen
Pasi Jylänki
Ville Tolvanen
Aki Vehtari

The GPstuff toolbox is a versatile collection of Gaussian process models and computational tools required for Bayesian inference. The tools include, among others, various inference methods, sparse approximations and model assessment methods. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2013. ( edit, beta )

PDF Details

JMLR Journal 2013 Journal Article

Nested Expectation Propagation for Gaussian Process Classification with a Multinomial Probit Likelihood

Jaakko Riihimäki
Pasi Jylänki
Aki Vehtari

This paper considers probabilistic multinomial probit classification using Gaussian process (GP) priors. Challenges with multiclass GP classification are the integration over the non-Gaussian posterior distribution, and the increase of the number of unknown latent variables as the number of target classes grows. Expectation propagation (EP) has proven to be a very accurate method for approximate inference but the existing EP approaches for the multinomial probit GP classification rely on numerical quadratures, or independence assumptions between the latent values associated with different classes, to facilitate the computations. In this paper we propose a novel nested EP approach which does not require numerical quadratures, and approximates accurately all between-class posterior dependencies of the latent values, but still scales linearly in the number of classes. The predictive accuracy of the nested EP approach is compared to Laplace, variational Bayes, and Markov chain Monte Carlo (MCMC) approximations with various benchmark data sets. In the experiments nested EP was the most consistent method compared to MCMC sampling, but in terms of classification accuracy the differences between all the methods were small from a practical point of view. [abs] [ pdf ][ bib ] &copy JMLR 2013. ( edit, beta )

PDF Details

YNIMG Journal 2012 Journal Article

Dynamic retrospective filtering of physiological noise in BOLD fMRI: DRIFTER

Simo Särkkä
Arno Solin
Aapo Nummenmaa
Aki Vehtari
Toni Auranen
Simo Vanni
Fa-Hsuan Lin

In this article we introduce the DRIFTER algorithm, which is a new model based Bayesian method for retrospective elimination of physiological noise from functional magnetic resonance imaging (fMRI) data. In the method, we first estimate the frequency trajectories of the physiological signals with the interacting multiple models (IMM) filter algorithm. The frequency trajectories can be estimated from external reference signals, or if the temporal resolution is high enough, from the fMRI data. The estimated frequency trajectories are then used in a state space model in combination of a Kalman filter (KF) and Rauch–Tung–Striebel (RTS) smoother, which separates the signal into an activation related cleaned signal, physiological noise, and white measurement noise components. Using experimental data, we show that the method outperforms the RETROICOR algorithm if the shape and amplitude of the physiological signals change over time.

Details DOI

JMLR Journal 2011 Journal Article

Robust Gaussian Process Regression with a Student- t Likelihood

Pasi Jylänki
Jarno Vanhatalo
Aki Vehtari

This paper considers the robust and efficient implementation of Gaussian process regression with a Student- t observation model, which has a non-log-concave likelihood. The challenge with the Student- t model is the analytically intractable inference which is why several approximative methods have been proposed. Expectation propagation (EP) has been found to be a very accurate method in many empirical studies but the convergence of EP is known to be problematic with models containing non-log-concave site functions. In this paper we illustrate the situations where standard EP fails to converge and review different modifications and alternative algorithms for improving the convergence. We demonstrate that convergence problems may occur during the type-II maximum a posteriori (MAP) estimation of the hyperparameters and show that standard EP may not converge in the MAP values with some difficult data sets. We present a robust implementation which relies primarily on parallel EP updates and uses a moment-matching-based double-loop algorithm with adaptively selected step size in difficult cases. The predictive performance of EP is compared with Laplace, variational Bayes, and Markov chain Monte Carlo approximations. [abs] [ pdf ][ bib ] &copy JMLR 2011. ( edit, beta )

PDF Details

UAI Conference 2010 Conference Paper

Speeding up the binary Gaussian process classification

Jarno Vanhatalo
Aki Vehtari

Gaussian processes (GP) are attractive building blocks for many probabilistic models. Their drawbacks, however, are the rapidly increasing inference time and memory requirement alongside increasing data. The problem can be alleviated with compactly supported (CS) covariance functions, which produce sparse covariance matrices that are fast in computations and cheap to store. CS functions have previously been used in GP regression but here the focus is in a classification problem. This brings new challenges since the posterior inference has to be done approximately. We utilize the expectation propagation algorithm and show how its standard implementation has to be modified to obtain computational benefits from the sparse covariance matrices. We study four CS covariance functions and show that they may lead to substantial speed up in the inference time compared to globally supported functions.

Details

NeurIPS Conference 2009 Conference Paper

Gaussian process regression with Student-t likelihood

Jarno Vanhatalo
Pasi Jylänki
Aki Vehtari

In the Gaussian process regression the observation model is commonly assumed to be Gaussian, which is convenient in computational perspective. However, the drawback is that the predictive accuracy of the model can be significantly compromised if the observations are contaminated by outliers. A robust observation model, such as the Student-t distribution, reduces the influence of outlying observations and improves the predictions. The problem, however, is the analytically intractable inference. In this work, we discuss the properties of a Gaussian process regression model with the Student-t likelihood and utilize the Laplace approximation for approximate inference. We compare our approach to a variational approximation and a Markov chain Monte Carlo scheme, which utilize the commonly used scale mixture representation of the Student-t distribution.

PDF Details

UAI Conference 2008 Conference Paper

Modelling local and global phenomena with sparse Gaussian processes

Jarno Vanhatalo
Aki Vehtari

Much recent work has concerned sparse approximations to speed up the Gaussian process regression from the unfavorable O(n3 ) scaling in computational time to O(nm2 ). Thus far, work has concentrated on models with one covariance function. However, in many practical situations additive models with multiple covariance functions may perform better, since the data may contain both long and short length-scale phenomena. The long length-scales can be captured with global sparse approximations, such as fully independent conditional (FIC), and the short length-scales can be modeled naturally by covariance functions with compact support (CS). CS covariance functions lead to naturally sparse covariance matrices, which are computationally cheaper to handle than full covariance matrices. In this paper, we propose a new sparse Gaussian process model with two additive components: FIC for the long length-scales and CS covariance function for the short length-scales. We give theoretical and experimental results and show that under certain conditions the proposed model has the same computational complexity as FIC. We also compare the model performance of the proposed model to additive models approximated by fully and partially independent conditional (PIC). We use real data sets and show that our model outperforms FIC and PIC approximations for data sets with two additive phenomena.

Details

YNIMG Journal 2007 Journal Article

Automatic relevance determination based hierarchical Bayesian MEG inversion in practice

Aapo Nummenmaa
Toni Auranen
Matti S. Hämäläinen
Iiro P. Jääskeläinen
Mikko Sams
Aki Vehtari
Jouko Lampinen

In recent simulation studies, a hierarchical Variational Bayesian (VB) method, which can be seen as a generalisation of the traditional minimum-norm estimate (MNE), was introduced for reconstructing distributed MEG sources. Here, we studied how nonlinearities in the estimation process and hyperparameter selection affect the inverse solutions, the feasibility of a full Bayesian treatment of the hyperparameters, and multimodality of the true posterior, in an empirical dataset wherein a male subject was presented with pure tone and checkerboard reversal stimuli, alone and in combination. An MRI-based cortical surface model was employed. Our results show, with a comparison to the basic MNE, that the hierarchical VB approach yields robust and physiologically plausible estimates of distributed sources underlying MEG measurements, in a rather automated fashion.

Details DOI

YNIMG Journal 2007 Journal Article

Hierarchical Bayesian estimates of distributed MEG sources: Theoretical aspects and comparison of variational and MCMC methods

Aapo Nummenmaa
Toni Auranen
Matti S. Hämäläinen
Iiro P. Jääskeläinen
Jouko Lampinen
Mikko Sams
Aki Vehtari

Magnetoencephalography (MEG) provides millisecond-scale temporal resolution for noninvasive mapping of human brain functions, but the problem of reconstructing the underlying source currents from the extracranial data has no unique solution. Several distributed source estimation methods based on different prior assumptions have been suggested for the resolution of this inverse problem. Recently, a hierarchical Bayesian generalization of the traditional minimum norm estimate (MNE) was proposed, in which the variance of distributed current at each cortical location is considered as a random variable and estimated from the data using the variational Bayesian (VB) framework. Here, we introduce an alternative scheme for performing Bayesian inference in the context of this hierarchical model by using Markov chain Monte Carlo (MCMC) strategies. In principle, the MCMC method is capable of numerically representing the true posterior distribution of the currents whereas the VB approach is inherently approximative. We point out some potential problems related to hyperprior selection in the previous work and study some possible solutions. A hyperprior sensitivity analysis is then performed, and the structure of the posterior distribution as revealed by the MCMC method is investigated. We show that the structure of the true posterior is rather complex with multiple modes corresponding to different possible solutions to the source reconstruction problem. We compare the results from the VB algorithm to those obtained from the MCMC simulation under different hyperparameter settings. The difficulties in using a unimodal variational distribution as a proxy for a truly multimodal distribution are also discussed. Simulated MEG data with realistic sensor and source geometries are used in performing the analyses.

Details DOI

YNIMG Journal 2005 Journal Article

Bayesian analysis of the neuromagnetic inverse problem with ℓ-norm priors

Toni Auranen
Aapo Nummenmaa
Matti S. Hämäläinen
Iiro P. Jääskeläinen
Jouko Lampinen
Aki Vehtari
Mikko Sams

Magnetoencephalography (MEG) allows millisecond-scale non-invasive measurement of magnetic fields generated by neural currents in the brain. However, localization of the underlying current sources is ambiguous due to the so-called inverse problem. The most widely used source localization methods (i. e. , minimum-norm and minimum-current estimates (MNE and MCE) and equivalent current dipole (ECD) fitting) require ad hoc determination of the cortical current distribution (ℓ 2-, ℓ 1-norm priors and point-sized dipolar, respectively). In this article, we perform a Bayesian analysis of the MEG inverse problem with ℓ p -norm priors for the current sources. This way, we circumvent the arbitrary choice between ℓ 1- and ℓ 2-norm prior, which is instead rendered automatically based on the data. By obtaining numerical samples from the joint posterior probability distribution of the source current parameters and model hyperparameters (such as the ℓ p -norm order p) using Markov chain Monte Carlo (MCMC) methods, we calculated the spatial inverse estimates as expectation values of the source current parameters integrated over the hyperparameters. Real MEG data and simulated (known) source currents with realistic MRI-based cortical geometry and 306-channel MEG sensor array were used. While the proposed model is sensitive to source space discretization size and computationally rather heavy, it is mathematically straightforward, thus allowing incorporation of, for instance, a priori functional magnetic resonance imaging (fMRI) information.

Details DOI