Klaus-Robert Müller

NeurIPS Conference 2025 Conference Paper

Manipulating Feature Visualizations with Gradient Slingshots

• Dilyara Bareeva
• Marina Höhne
• Alexander Warnecke
• Lukas Pirch
• Klaus-Robert Müller
• Konrad Rieck
• Sebastian Lapuschkin
• Kirill Bykov

Feature Visualization (FV) is a widely used technique for interpreting concepts learned by Deep Neural Networks (DNNs), which synthesizes input patterns that maximally activate a given feature. Despite its popularity, the trustworthiness of FV explanations has received limited attention. We introduce Gradient Slingshots, a novel method that enables FV manipulation without modifying model architecture or significantly degrading performance. By shaping new trajectories in off-distribution regions of a feature's activation landscape, we coerce the optimization process to converge to a predefined visualization. We evaluate our approach on several DNN architectures, demonstrating its ability to replace faithful FVs with arbitrary targets. These results expose a critical vulnerability: auditors relying solely on FV may accept entirely fabricated explanations. To mitigate this risk, we propose a straightforward defense and quantitatively demonstrate its effectiveness.

NeurIPS Conference 2025 Conference Paper

Sampling 3D Molecular Conformers with Diffusion Transformers

• J. Thorben Frank
• Winfried Ripken
• Gregor Lied
• Klaus-Robert Müller
• Oliver Unke
• Stefan Chmiela

Diffusion Transformers (DiTs) have demonstrated strong performance in generative modeling, particularly in image synthesis, making them a compelling choice for molecular conformer generation. However, applying DiTs to molecules introduces novel challenges, such as integrating discrete molecular graph information with continuous 3D geometry, handling Euclidean symmetries, and designing conditioning mechanisms that generalize across molecules of varying sizes and structures. We propose DiTMC, a framework that adapts DiTs to address these challenges through a modular architecture that separates the processing of 3D coordinates from conditioning on atomic connectivity. To this end, we introduce two complementary graph-based conditioning strategies that integrate seamlessly with the DiT architecture. These are combined with different attention mechanisms, including both standard non-equivariant and SO(3)-equivariant formulations, enabling flexible control over the trade-off between between accuracy and computational efficiency. Experiments on standard conformer generation benchmarks (GEOM-QM9, -DRUGS, -XL) demonstrate that DiTMC achieves state-of-the-art precision and physical validity. Our results highlight how architectural choices and symmetry priors affect sample quality and efficiency, suggesting promising directions for large-scale generative modeling of molecular structures. Code is available at https: //github. com/ML4MolSim/dit_mc.

NeurIPS Conference 2025 Conference Paper

Smoothed Differentiation Efficiently Mitigates Shattered Gradients in Explanations

• Adrian Hill
• Neal McKee
• Johannes Maeß
• Stefan Bluecher
• Klaus-Robert Müller

Explaining complex machine learning models is a fundamental challenge when developing safe and trustworthy deep learning applications. To date, a broad selection of explainable AI (XAI) algorithms exist. One popular choice is SmoothGrad, which has been conceived to alleviate the well-known shattered gradient problem by smoothing gradients through convolution. SmoothGrad proposes to solve this high-dimensional convolution integral by sampling -- typically approximating the convolution with limited precision. Higher numbers of samples would amount to higher precision in approximating the convolution but also to higher computing demand, therefore in practice only few samples are used in SmoothGrad. In this work we propose a well founded novel method SmoothDiff to resolve this tradeoff yielding a speedup of over two orders of magnitude. Specifically, SmoothDiff leverages automatic differentiation to decompose the expected values of Jacobians across a network architecture, directly targeting only the non-linearities responsible for shattered gradients and making it easy to implement. We demonstrate SmoothDiff's excellent speed and performance in a number of experiments and benchmarks. Thus, SmoothDiff greatly enhances the usability (quality and speed) of SmoothGrad -- a popular workhorse of XAI.

NeurIPS Conference 2024 Conference Paper

MambaLRP: Explaining Selective State Space Sequence Models

• Farnoush Rezaei Jafari
• Grégoire Montavon
• Klaus-Robert Müller
• Oliver Eberle

Recent sequence modeling approaches using selective state space sequence models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models.

ICLR Conference 2024 Conference Paper

Set Learning for Accurate and Calibrated Models

• Lukas Muttenthaler
• Robert A. Vandermeulen
• Qiuyi (Richard) Zhang
• Thomas Unterthiner
• Klaus-Robert Müller

Model overconfidence and poor calibration are common in machine learning and difficult to account for when applying standard empirical risk minimization. In this work, we propose a novel method to alleviate these problems that we call odd-$k$-out learning (OKO), which minimizes the cross-entropy error for sets rather than for single examples. This naturally allows the model to capture correlations across data examples and achieves both better accuracy and calibration, especially in limited training data and class-imbalanced regimes. Perhaps surprisingly, OKO often yields better calibration even when training with hard labels and dropping any additional calibration parameter tuning, such as temperature scaling. We demonstrate this in extensive experimental analyses and provide a mathematical theory to interpret our findings. We emphasize that OKO is a general framework that can be easily adapted to many settings and a trained model can be applied to single examples at inference time, without significant run-time overhead or architecture changes.

NeurIPS Conference 2024 Conference Paper

xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology

• Julius Hense
• Mina Jamshidi Idaji
• Oliver Eberle
• Thomas Schnake
• Jonas Dippel
• Laure Ciernik
• Oliver Buchstab
• Andreas Mock

Multiple instance learning (MIL) is an effective and widely used approach for weakly supervised machine learning. In histopathology, MIL models have achieved remarkable success in tasks like tumor detection, biomarker prediction, and outcome prognostication. However, MIL explanation methods are still lagging behind, as they are limited to small bag sizes or disregard instance interactions. We revisit MIL through the lens of explainable AI (XAI) and introduce xMIL, a refined framework with more general assumptions. We demonstrate how to obtain improved MIL explanations using layer-wise relevance propagation (LRP) and conduct extensive evaluation experiments on three toy settings and four real-world histopathology datasets. Our approach consistently outperforms previous explanation attempts with particularly improved faithfulness scores on challenging biomarker prediction tasks. Finally, we showcase how xMIL explanations enable pathologists to extract insights from MIL models, representing a significant advance for knowledge discovery and model debugging in digital histopathology.

NeurIPS Conference 2023 Conference Paper

Physics-Informed Bayesian Optimization of Variational Quantum Circuits

• Kim Nicoli
• Christopher J. Anders
• Lena Funcke
• Tobias Hartung
• Karl Jansen
• Stefan Kühn
• Klaus-Robert Müller
• Paolo Stornati

In this paper, we propose a novel and powerful method to harness Bayesian optimization for variational quantum eigensolvers (VQEs) - a hybrid quantum-classical protocol used to approximate the ground state of a quantum Hamiltonian. Specifically, we derive a VQE-kernel which incorporates important prior information about quantum circuits: the kernel feature map of the VQE-kernel exactly matches the known functional form of the VQE's objective function and thereby significantly reduces the posterior uncertainty. Moreover, we propose a novel acquisition function for Bayesian optimization called \emph{Expected Maximum Improvement over Confident Regions} (EMICoRe) which can actively exploit the inductive bias of the VQE-kernel by treating regions with low predictive uncertainty as indirectly "observed". As a result, observations at as few as three points in the search domain are sufficient to determine the complete objective function along an entire one-dimensional subspace of the optimization landscape. Our numerical experiments demonstrate that our approach improves over state-of-the-art baselines.

ICML Conference 2023 Conference Paper

Relevant Walk Search for Explaining Graph Neural Networks

• Ping Xiong 0002
• Thomas Schnake
• Michael Gastegger
• Grégoire Montavon
• Klaus-Robert Müller
• Shinichi Nakajima

Graph Neural Networks (GNNs) have become important machine learning tools for graph analysis, and its explainability is crucial for safety, fairness, and robustness. Layer-wise relevance propagation for GNNs (GNN-LRP) evaluates the relevance of walks to reveal important information flows in the network, and provides higher-order explanations, which have been shown to be superior to the lower-order, i. e. , node-/edge-level, explanations. However, identifying relevant walks by GNN-LRP requires exponential computational complexity with respect to the network depth, which we will remedy in this paper. Specifically, we propose polynomial-time algorithms for finding top-$K$ relevant walks, which drastically reduces the computation and thus increases the applicability of GNN-LRP to large-scale problems. Our proposed algorithms are based on the max-product algorithm—a common tool for finding the maximum likelihood configurations in probabilistic graphical models—and can find the most relevant walks exactly at the neuron level and approximately at the node level. Our experiments demonstrate the performance of our algorithms at scale and their utility across application domains, i. e. , on epidemiology, molecular, and natural language benchmarks. We provide our codes under github. com/xiong-ping/rel_walk_gnnlrp.

ICML Conference 2022 Conference Paper

Efficient Computation of Higher-Order Subgraph Attribution via Message Passing

• Ping Xiong 0002
• Thomas Schnake
• Grégoire Montavon
• Klaus-Robert Müller
• Shinichi Nakajima

Explaining graph neural networks (GNNs) has become more and more important recently. Higher-order interpretation schemes, such as GNN-LRP (layer-wise relevance propagation for GNN), emerged as powerful tools for unraveling how different features interact thereby contributing to explaining GNNs. GNN-LRP gives a relevance attribution of walks between nodes at each layer, and the subgraph attribution is expressed as a sum over exponentially many such walks. In this work, we demonstrate that such exponential complexity can be avoided. In particular, we propose novel algorithms that enable to attribute subgraphs with GNN-LRP in linear-time (w. r. t. the network depth). Our algorithms are derived via message passing techniques that make use of the distributive property, thereby directly computing quantities for higher-order explanations. We further adapt our efficient algorithms to compute a generalization of subgraph attributions that also takes into account the neighboring graph features. Experimental results show the significant acceleration of the proposed algorithms and demonstrate the high usefulness and scalability of our novel generalized subgraph attribution method.

YNIMG Journal 2022 Journal Article

Harmoni: A method for eliminating spurious interactions due to the harmonic components in neuronal data

• Mina Jamshidi Idaji
• Juanli Zhang
• Tilman Stephani
• Guido Nolte
• Klaus-Robert Müller
• Arno Villringer
• Vadim V. Nikulin

Cross-frequency synchronization (CFS) has been proposed as a mechanism for integrating spatially and spectrally distributed information in the brain. However, investigating CFS in Magneto- and Electroencephalography (MEG/EEG) is hampered by the presence of spurious neuronal interactions due to the non-sinusoidal waveshape of brain oscillations. Such waveshape gives rise to the presence of oscillatory harmonics mimicking genuine neuronal oscillations. Until recently, however, there has been no methodology for removing these harmonics from neuronal data. In order to address this long-standing challenge, we introduce a novel method (called HARMOnic miNImization - Harmoni) that removes the signal components which can be harmonics of a non-sinusoidal signal. Harmoni's working principle is based on the presence of CFS between harmonic components and the fundamental component of a non-sinusoidal signal. We extensively tested Harmoni in realistic EEG simulations. The simulated couplings between the source signals represented genuine and spurious CFS and within-frequency phase synchronization. Using diverse evaluation criteria, including ROC analyses, we showed that the within- and cross-frequency spurious interactions are suppressed significantly, while the genuine activities are not affected. Additionally, we applied Harmoni to real resting-state EEG data revealing intricate remote connectivity patterns which are usually masked by the spurious connections. Given the ubiquity of non-sinusoidal neuronal oscillations in electrophysiological recordings, Harmoni is expected to facilitate novel insights into genuine neuronal interactions in various research fields, and can also serve as a steppingstone towards the development of further signal processing methods aiming at refining within- and cross-frequency synchronization in electrophysiological recordings.

NeurIPS Conference 2022 Conference Paper

So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systems

• Thorben Frank
• Oliver Unke
• Klaus-Robert Müller

The application of machine learning methods in quantum chemistry has enabled the study of numerous chemical phenomena, which are computationally intractable with traditional ab-initio methods. However, some quantum mechanical properties of molecules and materials depend on non-local electronic effects, which are often neglected due to the difficulty of modeling them efficiently. This work proposes a modified attention mechanism adapted to the underlying physics, which allows to recover the relevant non-local effects. Namely, we introduce spherical harmonic coordinates (SPHCs) to reflect higher-order geometric information for each atom in a molecule, enabling a non-local formulation of attention in the SPHC space. Our proposed model So3krates - a self-attention based message passing neural network - uncouples geometric information from atomic features, making them independently amenable to attention mechanisms. Thereby we construct spherical filters, which extend the concept of continuous filters in Euclidean space to SPHC space and serve as foundation for a spherical self-attention mechanism. We show that in contrast to other published methods, So3krates is able to describe non-local quantum mechanical effects over arbitrary length scales. Further, we find evidence that the inclusion of higher-order geometric correlations increases data efficiency and improves generalization. So3krates matches or exceeds state-of-the-art performance on popular benchmarks, notably, requiring a significantly lower number of parameters (0. 25 - 0. 4x) while at the same time giving a substantial speedup (6 - 14x for training and 2 - 11x for inference) compared to other models.

YNIMG Journal 2022 Journal Article

Towards the interpretability of deep learning models for multi-modal neuroimaging: Finding structural changes of the ageing brain

• Simon M. Hofmann
• Frauke Beyer
• Sebastian Lapuschkin
• Ole Goltermann
• Markus Loeffler
• Klaus-Robert Müller
• Arno Villringer
• Wojciech Samek

Brain-age (BA) estimates based on deep learning are increasingly used as neuroimaging biomarker for brain health; however, the underlying neural features have remained unclear. We combined ensembles of convolutional neural networks with Layer-wise Relevance Propagation (LRP) to detect which brain features contribute to BA. Trained on magnetic resonance imaging (MRI) data of a population-based study (n = 2637, 18-82 years), our models estimated age accurately based on single and multiple modalities, regionally restricted and whole-brain images (mean absolute errors 3.37-3.86 years). We find that BA estimates capture ageing at both small and large-scale changes, revealing gross enlargements of ventricles and subarachnoid spaces, as well as white matter lesions, and atrophies that appear throughout the brain. Divergence from expected ageing reflected cardiovascular risk factors and accelerated ageing was more pronounced in the frontal lobe. Applying LRP, our study demonstrates how superior deep learning models detect brain-ageing in healthy and at-risk individuals throughout adulthood.

ICML Conference 2022 Conference Paper

XAI for Transformers: Better Explanations through Conservative Propagation

• Ameen Ali
• Thomas Schnake
• Oliver Eberle
• Grégoire Montavon
• Klaus-Robert Müller
• Lior Wolf

Transformers have become an important workhorse of machine learning, with numerous applications. This necessitates the development of reliable methods for increasing their transparency. Multiple interpretability methods, often based on gradient information, have been proposed. We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. We identify Attention Heads and LayerNorm as main reasons for such unreliable explanations and propose a more stable way for propagation through these layers. Our proposal, which can be seen as a proper extension of the well-established LRP method to Transformers, is shown both theoretically and empirically to overcome the deficiency of a simple gradient-based approach, and achieves state-of-the-art explanation performance on a broad range of Transformer models and datasets.

NeurIPS Conference 2021 Conference Paper

Efficient hierarchical Bayesian inference for spatio-temporal regression models in neuroimaging

• Ali Hashemi
• Yijing Gao
• Chang Cai
• Sanjay Ghosh
• Klaus-Robert Müller
• Srikantan Nagarajan
• Stefan Haufe

Several problems in neuroimaging and beyond require inference on the parameters of multi-task sparse hierarchical regression models. Examples include M/EEG inverse problems, neural encoding models for task-based fMRI analyses, and climate science. In these domains, both the model parameters to be inferred and the measurement noise may exhibit a complex spatio-temporal structure. Existing work either neglects the temporal structure or leads to computationally demanding inference schemes. Overcoming these limitations, we devise a novel flexible hierarchical Bayesian framework within which the spatio-temporal dynamics of model parameters and noise are modeled to have Kronecker product covariance structure. Inference in our framework is based on majorization-minimization optimization and has guaranteed convergence properties. Our highly efficient algorithms exploit the intrinsic Riemannian geometry of temporal autocovariance matrices. For stationary dynamics described by Toeplitz matrices, the theory of circulant embeddings is employed. We prove convex bounding properties and derive update rules of the resulting algorithms. On both synthetic and real neural data from M/EEG, we demonstrate that our methods lead to improved performance.

ICLR Conference 2021 Conference Paper

Explainable Deep One-Class Classification

• Philipp Liznerski
• Lukas Ruff
• Robert A. Vandermeulen
• Billy Joe Franks
• Marius Kloft
• Klaus-Robert Müller

Deep one-class classification variants for anomaly detection learn a mapping that concentrates nominal samples in feature space causing anomalies to be mapped away. Because this transformation is highly non-linear, finding interpretations poses a significant challenge. In this paper we present an explainable deep one-class classification method, Fully Convolutional Data Description (FCDD), where the mapped samples are themselves also an explanation heatmap. FCDD yields competitive detection performance and provides reasonable explanations on common anomaly detection benchmarks with CIFAR-10 and ImageNet. On MVTec-AD, a recent manufacturing dataset offering ground-truth anomaly maps, FCDD sets a new state of the art in the unsupervised setting. Our method can incorporate ground-truth anomaly maps during training and using even a few of these (~5) improves performance significantly. Finally, using FCDD's explanations we demonstrate the vulnerability of deep one-class classification models to spurious image features such as image watermarks.

NeurIPS Conference 2021 Conference Paper

SE(3)-equivariant prediction of molecular wavefunctions and electronic densities

• Oliver Unke
• Mihail Bogojeski
• Michael Gastegger
• Mario Geiger
• Tess Smidt
• Klaus-Robert Müller

Machine learning has enabled the prediction of quantum chemical properties with high accuracy and efficiency, allowing to bypass computationally costly ab initio calculations. Instead of training on a fixed set of properties, more recent approaches attempt to learn the electronic wavefunction (or density) as a central quantity of atomistic systems, from which all other observables can be derived. This is complicated by the fact that wavefunctions transform non-trivially under molecular rotations, which makes them a challenging prediction target. To solve this issue, we introduce general SE(3)-equivariant operations and building blocks for constructing deep learning architectures for geometric point cloud data and apply them to reconstruct wavefunctions of atomistic systems with unprecedented accuracy. Our model achieves speedups of over three orders of magnitude compared to ab initio methods and reduces prediction errors by up to two orders of magnitude compared to the previous state-of-the-art. This accuracy makes it possible to derive properties such as energies and forces directly from the wavefunction in an end-to-end manner. We demonstrate the potential of our approach in a transfer learning application, where a model trained on low accuracy reference wavefunctions implicitly learns to correct for electronic many-body interactions from observables computed at a higher level of theory. Such machine-learned wavefunction surrogates pave the way towards novel semi-empirical methods, offering resolution at an electronic level while drastically decreasing computational cost. Additionally, the predicted wavefunctions can serve as initial guess in conventional ab initio methods, decreasing the number of iterations required to arrive at a converged solution, thus leading to significant speedups without any loss of accuracy or robustness. While we focus on physics applications in this contribution, the proposed equivariant framework for deep learning on point clouds is promising also beyond, say, in computer vision or graphics.

YNIMG Journal 2021 Journal Article

Unification of sparse Bayesian learning algorithms for electromagnetic brain imaging with the majorization minimization framework

• Ali Hashemi
• Chang Cai
• Gitta Kutyniok
• Klaus-Robert Müller
• Srikantan S. Nagarajan
• Stefan Haufe

Methods for electro- or magnetoencephalography (EEG/MEG) based brain source imaging (BSI) using sparse Bayesian learning (SBL) have been demonstrated to achieve excellent performance in situations with low numbers of distinct active sources, such as event-related designs. This paper extends the theory and practice of SBL in three important ways. First, we reformulate three existing SBL algorithms under the majorization-minimization (MM) framework. This unification perspective not only provides a useful theoretical framework for comparing different algorithms in terms of their convergence behavior, but also provides a principled recipe for constructing novel algorithms with specific properties by designing appropriate bounds of the Bayesian marginal likelihood function. Second, building on the MM principle, we propose a novel method called LowSNR-BSI that achieves favorable source reconstruction performance in low signal-to-noise-ratio (SNR) settings. Third, precise knowledge of the noise level is a crucial requirement for accurate source reconstruction. Here we present a novel principled technique to accurately learn the noise variance from the data either jointly within the source reconstruction procedure or using one of two proposed cross-validation strategies. Empirically, we could show that the monotonous convergence behavior predicted from MM theory is confirmed in numerical experiments. Using simulations, we further demonstrate the advantage of LowSNR-BSI over conventional SBL in low-SNR regimes, and the advantage of learned noise levels over estimates derived from baseline data. To demonstrate the usefulness of our novel approach, we show neurophysiologically plausible source reconstructions on averaged auditory evoked potential data.

AAAI Conference 2020 Conference Paper

Benign Examples: Imperceptible Changes Can Enhance Image Translation Performance

• Vignesh Srinivasan
• Klaus-Robert Müller
• Wojciech Samek
• Shinichi Nakajima

Unpaired image-to-image domain translation involves the task of transferring an image in one domain to another domain without having pairs of data for supervision. Several methods have been proposed to address this task using Generative Adversarial Networks (GANs) and cycle consistency constraint enforcing the translated image to be mapped back to the original domain. This way, a Deep Neural Network (DNN) learns mapping such that the input training distribution transferred to the target domain matches the target training distribution. However, not all test images are expected to fall inside the data manifold in the input space where the DNN has learned to perform the mapping very well. Such images can have a poor mapping to the target domain. In this paper, we propose to perform Langevin dynamics, which makes a subtle change in the input space bringing them close to the data manifold, producing benign examples. The effect is significant improvement of the mapped image on the target domain. We also show that the score function estimation by denoising autoencoder (DAE), can practically be replaced with any autoencoding structure, which most image-to-image translation methods contain intrinsically due to the cycle consistency constraint. Thus, no additional training is required. We show advantages of our approach for several state-of-theart image-to-image domain translation models. Quantitative evaluation shows that our proposed method leads to a substantial increase in the accuracy to the target label on multiple state-of-the-art image classifiers, while qualitative user study proves that our method better represents the target domain, achieving better human preference scores.

ICLR Conference 2020 Conference Paper

Deep Semi-Supervised Anomaly Detection

• Lukas Ruff
• Robert A. Vandermeulen
• Nico Görnitz
• Alexander Binder
• Emmanuel Müller
• Klaus-Robert Müller
• Marius Kloft

Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated as an unsupervised learning problem. In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous. Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples. Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific. In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection. We further introduce an information-theoretic framework for deep anomaly detection based on the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution, which can serve as a theoretical interpretation for our method. In extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, we demonstrate that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.

ICML Conference 2020 Conference Paper

Fairwashing explanations with off-manifold detergent

• Christopher J. Anders
• Plamen Pasliev
• Ann-Kathrin Dombrowski
• Klaus-Robert Müller
• Pan Kessel

Explanation methods promise to make black-box classifiers more transparent. As a result, it is hoped that they can act as proof for a sensible, fair and trustworthy decision-making process of the algorithm and thereby increase its acceptance by the end-users. In this paper, we show both theoretically and experimentally that these hopes are presently unfounded. Specifically, we show that, for any classifier $g$, one can always construct another classifier $\tilde{g}$ which has the same behavior on the data (same train, validation, and test error) but has arbitrarily manipulated explanation maps. We derive this statement theoretically using differential geometry and demonstrate it experimentally for various explanation methods, architectures, and datasets. Motivated by our theoretical insights, we then propose a modification of existing explanation methods which makes them significantly more robust.

YNIMG Journal 2020 Journal Article

Improved physiological noise regression in fNIRS: A multimodal extension of the General Linear Model using temporally embedded Canonical Correlation Analysis

• Alexander von Lühmann
• Xinge Li
• Klaus-Robert Müller
• David A. Boas
• Meryem A. Yücel

For the robust estimation of evoked brain activity from functional Near-Infrared Spectroscopy (fNIRS) signals, it is crucial to reduce nuisance signals from systemic physiology and motion. The current best practice incorporates short-separation (SS) fNIRS measurements as regressors in a General Linear Model (GLM). However, several challenging signal characteristics such as non-instantaneous and non-constant coupling are not yet addressed by this approach and additional auxiliary signals are not optimally exploited. We have recently introduced a new methodological framework for the unsupervised multivariate analysis of fNIRS signals using Blind Source Separation (BSS) methods. Building onto the framework, in this manuscript we show how to incorporate the advantages of regularized temporally embedded Canonical Correlation Analysis (tCCA) into the supervised GLM. This approach allows flexible integration of any number of auxiliary modalities and signals. We provide guidance for the selection of optimal parameters and auxiliary signals for the proposed GLM extension. Its performance in the recovery of evoked HRFs is then evaluated using both simulated ground truth data and real experimental data and compared with the GLM with short-separation regression. Our results show that the GLM with tCCA significantly improves upon the current best practice, yielding significantly better results across all applied metrics: Correlation (HbO max. +45%), Root Mean Squared Error (HbO max. −55%), F-Score (HbO up to 3. 25-fold) and p-value as well as power spectral density of the noise floor. The proposed method can be incorporated into the GLM in an easily applicable way that flexibly combines any available auxiliary signals into optimal nuisance regressors. This work has potential significance both for conventional neuroscientific fNIRS experiments as well as for emerging applications of fNIRS in everyday environments, medicine and BCI, where high Contrast to Noise Ratio is of importance for single trial analysis.

YNIMG Journal 2020 Journal Article

Nonlinear interaction decomposition (NID): A method for separation of cross-frequency coupled sources in human brain

• Mina Jamshidi Idaji
• Klaus-Robert Müller
• Guido Nolte
• Burkhard Maess
• Arno Villringer
• Vadim V. Nikulin

Cross-frequency coupling (CFC) between neuronal oscillations reflects an integration of spatially and spectrally distributed information in the brain. Here, we propose a novel framework for detecting such interactions in Magneto- and Electroencephalography (MEG/EEG), which we refer to as Nonlinear Interaction Decomposition (NID). In contrast to all previous methods for separation of cross-frequency (CF) sources in the brain, we propose that the extraction of nonlinearly interacting oscillations can be based on the statistical properties of their linear mixtures. The main idea of NID is that nonlinearly coupled brain oscillations can be mixed in such a way that the resulting linear mixture has a non-Gaussian distribution. We evaluate this argument analytically for amplitude-modulated narrow-band oscillations which are either phase-phase or amplitude-amplitude CF coupled. We validated NID extensively with simulated EEG obtained with realistic head modelling. The method extracted nonlinearly interacting components reliably even at SNRs as small as − 15 dB. Additionally, we applied NID to the resting-state EEG of 81 subjects to characterize CF phase-phase coupling between alpha and beta oscillations. The extracted sources were located in temporal, parietal and frontal areas, demonstrating the existence of diverse local and distant nonlinear interactions in resting-state EEG data. All codes are available publicly via GitHub.

YNIMG Journal 2019 Journal Article

A new blind source separation framework for signal analysis and artifact rejection in functional Near-Infrared Spectroscopy

• Alexander von Lühmann
• Zois Boukouvalas
• Klaus-Robert Müller
• Tülay Adalı

In the analysis of functional Near-Infrared Spectroscopy (fNIRS) signals from real-world scenarios, artifact rejection is essential. However, currently there exists no gold-standard. Although a plenitude of methodological approaches implicitly assume the presence of latent processes in the signals, elaborate Blind-Source-Separation methods have rarely been applied. A reason are challenging characteristics such as Non-instantaneous and non-constant coupling, correlated noise and statistical dependencies between signal components. We present a novel suitable BSS framework that tackles these issues by incorporating A) Independent Component Analysis methods that exploit both higher order statistics and sample dependency, B) multimodality, i. e. , fNIRS with accelerometer signals, and C) Canonical-Correlation Analysis with temporal embedding. This enables analysis of signal components and rejection of motion-induced physiological hemodynamic artifacts that would otherwise be hard to identify. We implement a method for Blind Source Separation and Accelerometer based Artifact Rejection and Detection (BLISS A 2 RD). It allows the analysis of a novel n-back based cognitive workload paradigm in freely moving subjects, that is also presented in this manuscript. We evaluate on the corresponding data set and simulated ground truth data, making use of metrics based on 1st and 2nd order statistics and SNR and compare with three established methods: PCA, Spline and Wavelet-based artifact removal. Across 17 subjects, the method is shown to reduce movement induced artifacts by up to two orders of magnitude, improves the SNR of continuous hemodynamic signals in single channels by up to 10 d B, and significantly outperforms conventional methods in the extraction of simulated Hemodynamic Response Functions from strongly contaminated data. The framework and methods presented can serve as an introduction to a new type of multivariate methods for the analysis of fNIRS signals and as a blueprint for artifact rejection in complex environments beyond the applied paradigm.

NeurIPS Conference 2019 Conference Paper

Explanations can be manipulated and geometry is to blame

• Ann-Kathrin Dombrowski
• Maximillian Alber
• Christopher Anders
• Marcel Ackermann
• Klaus-Robert Müller
• Pan Kessel

Explanation methods aim to make neural networks more trustworthy and interpretable. In this paper, we demonstrate a property of explanation methods which is disconcerting for both of these purposes. Namely, we show that explanations can be manipulated arbitrarily by applying visually hardly perceptible perturbations to the input that keep the network's output approximately constant. We establish theoretically that this phenomenon can be related to certain geometrical properties of neural networks. This allows us to derive an upper bound on the susceptibility of explanations to manipulations. Based on this result, we propose effective mechanisms to enhance the robustness of explanations.

JMLR Journal 2019 Journal Article

iNNvestigate Neural Networks!

• Maximilian Alber
• Sebastian Lapuschkin
• Philipp Seegerer
• Miriam Hägele
• Kristof T. Schütt
• Grégoire Montavon
• Wojciech Samek
• Klaus-Robert Müller

In recent years, deep neural networks have revolutionized many application domains of machine learning and are key components of many critical decision or predictive processes. Therefore, it is crucial that domain specialists can understand and analyze actions and predictions, even of the most complex neural network architectures. Despite these arguments neural networks are often treated as black boxes. In the attempt to alleviate this shortcoming many analysis methods were proposed, yet the lack of reference implementations often makes a systematic comparison between the methods a major effort. The presented library innvestigate addresses this by providing a common interface and out-of-the-box implementation for many analysis methods, including the reference implementation for PatternNet and PatternAttribution as well as for LRP-methods. To demonstrate the versatility of innvestigate, we provide an analysis of image classifications for variety of state-of-the-art neural network architectures. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2019. ( edit, beta )

IJCAI Conference 2018 Conference Paper

Curly: An AI-based Curling Robot Successfully Competing in the Olympic Discipline of Curling

• Dong-Ok Won
• Byung-Do Kim
• Ho-Jung Kim
• Tae-San Eom
• Klaus-Robert Müller
• Seong-Whan Lee

Most artificial intelligence (AI) based learning systems act in virtual or laboratory environments. Here we demonstrate an AI-based curling robot system named `Curly' that competes on a real-world curling ice sheet. Curly encompasses (1) an AI-based curling strategy and simulation engine under consideration of the high `icy' uncertainty, (2) the thrower robot enabled by autonomous driving with traction control, and (3) the skip robot that allows to recognize the curling field and stone configuration based on vision technology. The Curly performed well both: in classical game situations and when interacting with human opponents, namely, the top-ranked Korean amateur high school curling team.

JMLR Journal 2017 Journal Article

An Easy-to-hard Learning Paradigm for Multiple Classes and Multiple Labels

• Weiwei Liu
• Ivor W. Tsang
• Klaus-Robert Müller

Many applications, such as human action recognition and object detection, can be formulated as a multiclass classification problem. One-vs-rest (OVR) is one of the most widely used approaches for multiclass classification due to its simplicity and excellent performance. However, many confusing classes in such applications will degrade its results. For example, hand clap and boxing are two confusing actions. Hand clap is easily misclassified as boxing, and vice versa. Therefore, precisely classifying confusing classes remains a challenging task. To obtain better performance for multiclass classifications that have confusing classes, we first develop a classifier chain model for multiclass classification (CCMC) to transfer class information between classifiers. Then, based on an analysis of our proposed model, we propose an easy- to-hard learning paradigm for multiclass classification to automatically identify easy and hard classes and then use the predictions from simpler classes to help solve harder classes. Similar to CCMC, the classifier chain (CC) model is also proposed by Read et al. (2009) to capture the label dependency for multi-label classification. However, CC does not consider the order of difficulty of the labels and achieves degenerated performance when there are many confusing labels. Therefore, it is non- trivial to learn the appropriate label order for CC. Motivated by our analysis for CCMC, we also propose the easy-to-hard learning paradigm for multi-label classification to automatically identify easy and hard labels, and then use the predictions from simpler labels to help solve harder labels. We also demonstrate that our proposed strategy can be successfully applied to a wide range of applications, such as ordinal classification and relationship prediction. Extensive empirical studies validate our analysis and the effectiveness of our proposed easy-to-hard learning strategies. [abs] [ pdf ][ bib ] &copy JMLR 2017. ( edit, beta )

NeurIPS Conference 2017 Conference Paper

An Empirical Study on The Properties of Random Bases for Kernel Methods

• Maximilian Alber
• Pieter-Jan Kindermans
• Kristof Schütt
• Klaus-Robert Müller
• Fei Sha

Kernel machines as well as neural networks possess universal function approximation properties. Nevertheless in practice their ways of choosing the appropriate function class differ. Specifically neural networks learn a representation by adapting their basis functions to the data and the task at hand, while kernel methods typically use a basis that is not adapted during training. In this work, we contrast random features of approximated kernel machines with learned features of neural networks. Our analysis reveals how these random and adaptive basis functions affect the quality of learning. Furthermore, we present basis adaptation schemes that allow for a more compact representation, while retaining the generalization properties of kernel machines.

ICML Conference 2017 Conference Paper

Minimizing Trust Leaks for Robust Sybil Detection

• János Höner
• Shinichi Nakajima
• Alexander Bauer 0001
• Klaus-Robert Müller
• Nico Görnitz

Sybil detection is a crucial task to protect online social networks (OSNs) against intruders who try to manipulate automatic services provided by OSNs to their customers. In this paper, we first discuss the robustness of graph-based Sybil detectors SybilRank and Integro and refine theoretically their security guarantees towards more realistic assumptions. After that, we formally introduce adversarial settings for the graph-based Sybil detection problem and derive a corresponding optimal attacking strategy by exploitation of trust leaks. Based on our analysis, we propose transductive Sybil ranking (TSR), a robust extension to SybilRank and Integro that directly minimizes trust leaks. Our empirical evaluation shows significant advantages of TSR over state-of-the-art competitors on a variety of attacking scenarios on artificially generated data and real-world datasets.

NeurIPS Conference 2017 Conference Paper

SchNet: A continuous-filter convolutional neural network for modeling quantum interactions

• Kristof Schütt
• Pieter-Jan Kindermans
• Huziel Enoc Sauceda Felix
• Stefan Chmiela
• Alexandre Tkatchenko
• Klaus-Robert Müller

Deep learning has the potential to revolutionize quantum chemistry as it is ideally suited to learn representations for structured data and speed up the exploration of chemical space. While convolutional neural networks have proven to be the first choice for images, audio and video data, the atoms in molecules are not restricted to a grid. Instead, their precise locations contain essential physical information, that would get lost if discretized. Thus, we propose to use continuous-filter convolutional layers to be able to model local correlations without requiring the data to lie on a grid. We apply those layers in SchNet: a novel deep learning architecture modeling quantum interactions in molecules. We obtain a joint model for the total energy and interatomic forces that follows fundamental quantum-chemical principles. Our architecture achieves state-of-the-art performance for benchmarks of equilibrium molecules and molecular dynamics trajectories. Finally, we introduce a more challenging benchmark with chemical and structural variations that suggests the path for further work.

YNIMG Journal 2016 Journal Article

Analyzing neuroimaging data with subclasses: A shrinkage approach

• Johannes Höhne
• Daniel Bartz
• Martin N. Hebart
• Klaus-Robert Müller
• Benjamin Blankertz

Among the numerous methods used to analyze neuroimaging data, Linear Discriminant Analysis (LDA) is commonly applied for binary classification problems. LDAs popularity derives from its simplicity and its competitive classification performance, which has been reported for various types of neuroimaging data. Yet the standard LDA approach proves less than optimal for binary classification problems when additional label information (i. e. subclass labels) is present. Subclass labels allow to model structure in the data, which can be used to facilitate the classification task. In this paper, we illustrate how neuroimaging data exhibit subclass labels that may contain valuable information. We also show that the standard LDA classifier is unable to exploit subclass labels. We introduce a novel method that allows subclass labels to be incorporated efficiently into the classifier. The novel method, which we call Relevance Subclass LDA (RSLDA), computes an individual classification hyperplane for each subclass. It is based on regularized estimators of the subclass mean and uses other subclasses as regularization targets. We demonstrate the applicability and performance of our method on data drawn from two different neuroimaging modalities: (I) EEG data from brain–computer interfacing with event-related potentials, and (II) fMRI data in response to different levels of visual motion. We show that RSLDA outperforms the standard LDA approach for both types of datasets. These findings illustrate the benefits of exploiting subclass structure in neuroimaging data. Finally, we show that our classifier also outputs regularization profiles, enabling researchers to interpret the subclass structure in a meaningful way. RSLDA therefore yields increased classification accuracy as well as a better interpretation of neuroimaging data. Since both results are highly favorable, we suggest to apply RSLDA for various classification problems within neuroimaging and beyond.

YNIMG Journal 2016 Journal Article

Multiscale temporal neural dynamics predict performance in a complex sensorimotor task

• Wojciech Samek
• Duncan A.J. Blythe
• Gabriel Curio
• Klaus-Robert Müller
• Benjamin Blankertz
• Vadim V. Nikulin

Ongoing neuronal oscillations are pivotal in brain functioning and are known to influence subjects' performance. This modulation is usually studied on short time scales whilst multiple time scales are rarely considered. In our study we show that Long-Range Temporal Correlations (LRTCs) estimated from the amplitude of EEG oscillations over a range of time-scales predict performance in a complex sensorimotor task, based on Brain-Computer Interfacing (BCI). Our paradigm involved eighty subjects generating covert motor responses to dynamically changing visual cues and thus controlling a computer program through the modulation of neuronal oscillations. The neuronal dynamics were estimated with multichannel EEG. Our results show that: (a) BCI task accuracy may be predicted on the basis of LRTCs measured during the preceding training session, and (b) this result was not due to signal-to-noise ratio of the ongoing neuronal oscillations. Our results provide direct empirical evidence in addition to previous theoretical work suggesting that scale-free neuronal dynamics are important for optimal brain functioning.

YNIMG Journal 2016 Journal Article

The LDA beamformer: Optimal estimation of ERP source time series using linear discriminant analysis

• Matthias S. Treder
• Anne K. Porbadnigk
• Forooz Shahbazi Avarvand
• Klaus-Robert Müller
• Benjamin Blankertz

We introduce a novel beamforming approach for estimating event-related potential (ERP) source time series based on regularized linear discriminant analysis (LDA). The optimization problems in LDA and linearly-constrained minimum-variance (LCMV) beamformers are formally equivalent. The approaches differ in that, in LCMV beamformers, the spatial patterns are derived from a source model, whereas in an LDA beamformer the spatial patterns are derived directly from the data (i. e. , the ERP peak). Using a formal proof and MEG simulations, we show that the LDA beamformer is robust to correlated sources and offers a higher signal-to-noise ratio than the LCMV beamformer and PCA. As an application, we use EEG data from an oddball experiment to show how the LDA beamformer can be harnessed to detect single-trial ERP latencies and estimate connectivity between ERP sources. Concluding, the LDA beamformer optimally reconstructs ERP sources by maximizing the ERP signal-to-noise ratio. Hence, it is a highly suited tool for analyzing ERP source time series, particularly in EEG/MEG studies wherein a source model is not available.

JMLR Journal 2016 Journal Article

The LRP Toolbox for Artificial Neural Networks

• Sebastian Lapuschkin
• Alexander Binder
• Grégoire Montavon
• Klaus-Robert Müller
• Wojciech Samek

The Layer-wise Relevance Propagation (LRP) algorithm explains a classifier's prediction specific to a given data point by attributing relevance scores to important components of the input by using the topology of the learned model itself. With the LRP Toolbox we provide platform-agnostic implementations for explaining the predictions of pre-trained state of the art Caffe networks and stand-alone implementations for fully connected Neural Network models. The implementations for Matlab and python shall serve as a playing field to familiarize oneself with the LRP algorithm and are implemented with readability and transparency in mind. Models and data can be imported and exported using raw text formats, Matlab's.mat files and the.npy format for numpy or plain text. [abs] [ pdf ][ bib ] &copy JMLR 2016. ( edit, beta )

NeurIPS Conference 2016 Conference Paper

Wasserstein Training of Restricted Boltzmann Machines

• Grégoire Montavon
• Klaus-Robert Müller
• Marco Cuturi

Boltzmann machines are able to learn highly complex, multimodal, structured and multiscale real-world data distributions. Parameters of the model are usually learned by minimizing the Kullback-Leibler (KL) divergence from training samples to the learned model. We propose in this work a novel approach for Boltzmann machine training which assumes that a meaningful metric between observations is known. This metric between observations can then be used to define the Wasserstein distance between the distribution induced by the Boltzmann machine on the one hand, and that given by the training sample on the other hand. We derive a gradient of that distance with respect to the model parameters. Minimization of this new objective leads to generative models with different statistical properties. We demonstrate their practical potential on data completion and denoising, for which the metric between observations plays a crucial role.

YNIMG Journal 2015 Journal Article

Extracting latent brain states — Towards true labels in cognitive neuroscience experiments

• Anne K. Porbadnigk
• Nico Görnitz
• Claudia Sannelli
• Alexander Binder
• Mikio Braun
• Marius Kloft
• Klaus-Robert Müller

Neuroscientific data is typically analyzed based on the behavioral response of the participant. However, the errors made may or may not be in line with the neural processing. In particular in experiments with time pressure or studies where the threshold of perception is measured, the error distribution deviates from uniformity due to the structure in the underlying experimental set-up. When we base our analysis on the behavioral labels as usually done, then we ignore this problem of systematic and structured (non-uniform) label noise and are likely to arrive at wrong conclusions in our data analysis. This paper contributes a remedy to this important scenario: we present a novel approach for a) measuring label noise and b) removing structured label noise. We demonstrate its usefulness for EEG data analysis using a standard d2 test for visual attention (N=20 participants).

YNIMG Journal 2015 Journal Article

Identifying Granger causal relationships between neural power dynamics and variables of interest

• Irene Winkler
• Stefan Haufe
• Anne K. Porbadnigk
• Klaus-Robert Müller
• Sven Dähne

Power modulations of oscillations in electro- and magnetoencephalographic (EEG/MEG) signals have been linked to a wide range of brain functions. To date, most of the evidence is obtained by correlating bandpower fluctuations to specific target variables such as reaction times or task ratings, while the causal links between oscillatory activity and behavior remain less clear. Here, we propose to identify causal relationships by the statistical concept of Granger causality, and we investigate which methods are bests suited to reveal Granger causal links between the power of brain oscillations and experimental variables. As an alternative to testing such causal links on the sensor level, we propose to linearly combine the information contained in each sensor in order to create virtual channels, corresponding to estimates of underlying brain oscillations, the Granger-causal relations of which may be assessed. Such linear combinations of sensor can be given by source separation methods such as, for example, Independent Component Analysis (ICA) or by the recently developed Source Power Correlation (SPoC) method. Here we compare Granger causal analysis on power dynamics obtained from i) sensor directly, ii) spatial filtering methods that do not optimize for Granger causality (ICA and SPoC), and iii) a method that directly optimizes spatial filters to extract sources the power dynamics of which maximally Granger causes a given target variable. We refer to this method as Granger Causal Power Analysis (GrangerCPA). Using both simulated and real EEG recordings, we find that computing Granger causality on channel-wise spectral power suffers from a poor signal-to-noise ratio due to volume conduction, while all three multivariate approaches alleviate this issue. In real EEG recordings from subjects performing self-paced foot movements, all three multivariate methods identify neural oscillations with motor-related patterns at a similar performance level. In an auditory perception task, the application of GrangerCPA reveals significant Granger-causal links between alpha oscillations and reaction times in more subjects compared to conventional methods.

NeurIPS Conference 2014 Conference Paper

Covariance shrinkage for autocorrelated data

• Daniel Bartz
• Klaus-Robert Müller

The accurate estimation of covariance matrices is essential for many signal processing and machine learning algorithms. In high dimensional settings the sample covariance is known to perform poorly, hence regularization strategies such as analytic shrinkage of Ledoit/Wolf are applied. In the standard setting, i. i. d. data is assumed, however, in practice, time series typically exhibit strong autocorrelation structure, which introduces a pronounced estimation bias. Recent work by Sancetta has extended the shrinkage framework beyond i. i. d. data. We contribute in this work by showing that the Sancetta estimator, while being consistent in the high-dimensional limit, suffers from a high bias in finite sample sizes. We propose an alternative estimator, which is (1) unbiased, (2) less sensitive to hyperparameter choice and (3) yields superior performance in simulations on toy data and on a real world data set from an EEG-based Brain-Computer-Interfacing experiment.

YNIMG Journal 2014 Journal Article

Finding brain oscillations with power dependencies in neuroimaging data

• Sven Dähne
• Vadim V. Nikulin
• David Ramírez
• Peter J. Schreier
• Klaus-Robert Müller
• Stefan Haufe

Phase synchronization among neuronal oscillations within the same frequency band has been hypothesized to be a major mechanism for communication between different brain areas. On the other hand, cross-frequency communications are more flexible allowing interactions between oscillations with different frequencies. Among such cross-frequency interactions amplitude-to-amplitude interactions are of a special interest as they show how the strength of spatial synchronization in different neuronal populations relates to each other during a given task. While, previously, amplitude-to-amplitude correlations were studied primarily on the sensor level, we present a source separation approach using spatial filters which maximize the correlation between the envelopes of brain oscillations recorded with electro-/magnetoencephalography (EEG/MEG) or intracranial multichannel recordings. Our approach, which is called canonical source power correlation analysis (cSPoC), is thereby capable of extracting genuine brain oscillations solely based on their assumed coupling behavior even when the signal-to-noise ratio of the signals is low. In addition to using cSPoC for the analysis of cross-frequency interactions in the same subject, we show that it can also be utilized for studying amplitude dynamics of neuronal oscillations across subjects. We assess the performance of cSPoC in simulations as well as in three distinctively different analysis scenarios of real EEG data, each involving several subjects. In the simulations, cSPoC outperforms unsupervised state-of-the-art approaches. In the analysis of real EEG recordings, we demonstrate excellent unsupervised discovery of meaningful power-to-power couplings, within as well as across subjects and frequency bands.

YNIMG Journal 2014 Journal Article

SPoC: A novel framework for relating the amplitude of neuronal oscillations to behaviorally relevant parameters

• Sven Dähne
• Frank C. Meinecke
• Stefan Haufe
• Johannes Höhne
• Michael Tangermann
• Klaus-Robert Müller
• Vadim V. Nikulin

Previously, modulations in power of neuronal oscillations have been functionally linked to sensory, motor and cognitive operations. Such links are commonly established by relating the power modulations to specific target variables such as reaction times or task ratings. Consequently, the resulting spatio-spectral representation is subjected to neurophysiological interpretation. As an alternative, independent component analysis (ICA) or alternative decomposition methods can be applied and the power of the components may be related to the target variable. In this paper we show that these standard approaches are suboptimal as the first does not take into account the superposition of many sources due to volume conduction, while the second is unable to exploit available information about the target variable. To improve upon these approaches we introduce a novel (supervised) source separation framework called Source Power Comodulation (SPoC). SPoC makes use of the target variable in the decomposition process in order to give preference to components whose power comodulates with the target variable. We present two algorithms that implement the SPoC approach. Using simulations with a realistic head model, we show that the SPoC algorithms are able extract neuronal components exhibiting high correlation of power with the target variable. In this task, the SPoC algorithms outperform other commonly used techniques that are based on the sensor data or ICA approaches. Furthermore, using real electroencephalography (EEG) recordings during an auditory steady state paradigm, we demonstrate the utility of the SPoC algorithms by extracting neuronal components exhibiting high correlation of power with the intensity of the auditory input. Taking into account the results of the simulations and real EEG recordings, we conclude that SPoC represents an adequate approach for the optimal extraction of neuronal components showing coupling of power with continuously changing behaviorally relevant parameters.

YNIMG Journal 2014 Journal Article

Stereoscopic depth increases intersubject correlations of brain networks

• Michael Gaebler
• Felix Biessmann
• Jan-Peter Lamke
• Klaus-Robert Müller
• Henrik Walter
• Stefan Hetzer

Three-dimensional movies presented via stereoscopic displays have become more popular in recent years aiming at a more engaging viewing experience. However, neurocognitive processes associated with the perception of stereoscopic depth in complex and dynamic visual stimuli remain understudied. Here, we investigate the influence of stereoscopic depth on both neurophysiology and subjective experience. Using multivariate statistical learning methods, we compare the brain activity of subjects when freely watching the same movies in 2D and in 3D. Subjective reports indicate that 3D movies are more strongly experienced than 2D movies. On the neural level, we observe significantly higher intersubject correlations of cortical networks when subjects are watching 3D movies relative to the same movies in 2D. We demonstrate that increases in intersubject correlations of brain networks can serve as neurophysiological marker for stereoscopic depth and for the strength of the viewing experience.

YNIMG Journal 2014 Journal Article

The effect of linear mixing in the EEG on Hurst exponent estimation

• Duncan A.J. Blythe
• Stefan Haufe
• Klaus-Robert Müller
• Vadim V. Nikulin

Although the long-range temporal correlation (LRTC) of the amplitude fluctuations of neuronal EEG/MEG oscillations is widely acknowledged, the majority of studies to date have been performed in sensor space, disregarding the mixing effects implied by volume conduction and confounding noise. While the effect of mixing on the evaluation of evoked responses and connectivity measures has been extensively studied, there are, to date, no studies reporting on the differences in the values of the estimated Hurst exponents when moving between sensor and source space representations of the multivariate data or on the effect of noise. Such differences, if not duly acknowledged, may lead to erroneous data interpretations. We show in simulations and in theory that measuring Hurst exponents in sensor space may lead to an incomplete picture of the LRTC properties of the underlying data and that noise may significantly bias the estimate of the Hurst exponent of the underlying signal. Moreover, these predictions are confirmed in real data, where we analyze the amplitude dynamics of neuronal oscillations in the resting state from EEG data. By moving either to an independent components representation or to a source representation which maximizes the signal to noise ratio in the alpha frequency range, we observe greater variance, skewness and kurtosis over measured Hurst exponents than in sensor space. We confirm the suitability of conventional source separation methodology by introducing a novel algorithm HeMax which obtains a source maximizing the Hurst exponent in the amplitude dynamics of narrow band oscillations. Our findings imply that the long-range correlative properties of the EEG should be studied in source space, in such a way that the SNR is maximized, or at least with spatial decomposition techniques approximating source activities, rather than in sensor space.

YNIMG Journal 2013 Journal Article

A critical assessment of connectivity measures for EEG data: A simulation study

• Stefan Haufe
• Vadim V. Nikulin
• Klaus-Robert Müller
• Guido Nolte

Information flow between brain areas is difficult to estimate from EEG measurements due to the presence of noise as well as due to volume conduction. We here test the ability of popular measures of effective connectivity to detect an underlying neuronal interaction from simulated EEG data, as well as the ability of commonly used inverse source reconstruction techniques to improve the connectivity estimation. We find that volume conduction severely limits the neurophysiological interpretability of sensor-space connectivity analyses. Moreover, it may generally lead to conflicting results depending on the connectivity measure and statistical testing approach used. In particular, we note that the application of Granger-causal (GC) measures combined with standard significance testing leads to the detection of spurious connectivity regardless of whether the analysis is performed on sensor-space data or on sources estimated using three different established inverse methods. This empirical result follows from the definition of GC. The phase-slope index (PSI) does not suffer from this theoretical limitation and therefore performs well on our simulated data. We develop a theoretical framework to characterize artifacts of volume conduction, which may still be present even in reconstructed source time series as zero-lag correlations, and to distinguish their time-delayed brain interaction. Based on this theory we derive a procedure which suppresses the influence of volume conduction, but preserves effects related to time-lagged brain interaction in connectivity estimates. This is achieved by using time-reversed data as surrogates for statistical testing. We demonstrate that this robustification makes Granger-causal connectivity measures applicable to EEG data, achieving similar results as PSI. Integrating the insights of our study, we provide a guidance for measuring brain interaction from EEG data. Software for generating benchmark data is made available.

NeurIPS Conference 2013 Conference Paper

Generalizing Analytic Shrinkage for Arbitrary Covariance Structures

• Daniel Bartz
• Klaus-Robert Müller

Analytic shrinkage is a statistical technique that offers a fast alternative to cross-validation for the regularization of covariance matrices and has appealing consistency properties. We show that the proof of consistency implies bounds on the growth rates of eigenvalues and their dispersion, which are often violated in data. We prove consistency under assumptions which do not restrict the covariance structure and therefore better match real world data. In addition, we propose an extension of analytic shrinkage --orthogonal complement shrinkage-- which adapts to the covariance structure. Finally we demonstrate the superior performance of our novel approach on data from the domains of finance, spoken letter and optical character recognition, and neuroscience.

NeurIPS Conference 2013 Conference Paper

Robust Spatial Filtering with Beta Divergence

• Wojciech Samek
• Duncan Blythe
• Klaus-Robert Müller
• Motoaki Kawanabe

The efficiency of Brain-Computer Interfaces (BCI) largely depends upon a reliable extraction of informative features from the high-dimensional EEG signal. A crucial step in this protocol is the computation of spatial filters. The Common Spatial Patterns (CSP) algorithm computes filters that maximize the difference in band power between two conditions, thus it is tailored to extract the relevant information in motor imagery experiments. However, CSP is highly sensitive to artifacts in the EEG data, i. e. few outliers may alter the estimate drastically and decrease classification performance. Inspired by concepts from the field of information geometry we propose a novel approach for robustifying CSP. More precisely, we formulate CSP as a divergence maximization problem and utilize the property of a particular type of divergence, namely beta divergence, for robustifying the estimation of spatial filters in the presence of artifacts in the data. We demonstrate the usefulness of our method on toy data and on EEG recordings from 80 subjects.

JMLR Journal 2012 Journal Article

Algebraic Geometric Comparison of Probability Distributions

• Franz J. Király
• Paul von Bünau
• Frank C. Meinecke
• Duncan A.J. Blythe
• Klaus-Robert Müller

We propose a novel algebraic algorithmic framework for dealing with probability distributions represented by their cumulants such as the mean and covariance matrix. As an example, we consider the unsupervised learning problem of finding the subspace on which several probability distributions agree. Instead of minimizing an objective function involving the estimated cumulants, we show that by treating the cumulants as elements of the polynomial ring we can directly solve the problem, at a lower computational cost and with higher accuracy. Moreover, the algebraic viewpoint on probability distributions allows us to invoke the theory of algebraic geometry, which we demonstrate in a compact proof for an identifiability criterion. [abs] [ pdf ][ bib ] &copy JMLR 2012. ( edit, beta )

YNIMG Journal 2012 Journal Article

Enhanced performance by a hybrid NIRS–EEG brain computer interface

• Siamac Fazli
• Jan Mehnert
• Jens Steinbrink
• Gabriel Curio
• Arno Villringer
• Klaus-Robert Müller
• Benjamin Blankertz

Noninvasive Brain Computer Interfaces (BCI) have been promoted to be used for neuroprosthetics. However, reports on applications with electroencephalography (EEG) show a demand for a better accuracy and stability. Here we investigate whether near-infrared spectroscopy (NIRS) can be used to enhance the EEG approach. In our study both methods were applied simultaneously in a real-time Sensory Motor Rhythm (SMR)-based BCI paradigm, involving executed movements as well as motor imagery. We tested how the classification of NIRS data can complement ongoing real-time EEG classification. Our results show that simultaneous measurements of NIRS and EEG can significantly improve the classification accuracy of motor imagery in over 90% of considered subjects and increases performance by 5% on average (p<0: 01). However, the long time delay of the hemodynamic response may hinder an overall increase of bit-rates. Furthermore we find that EEG and NIRS complement each other in terms of information content and are thus a viable multimodal imaging technique, suitable for BCI.

YNIMG Journal 2012 Journal Article

Improved decoding of neural activity from fMRI signals using non-separable spatiotemporal deconvolutions

• Felix Bießmann
• Yusuke Murayama
• Nikos K. Logothetis
• Klaus-Robert Müller
• Frank C. Meinecke

The goal of most functional Magnetic Resonance Imaging (fMRI) analyses is to investigate neural activity. Many fMRI analysis methods assume that the temporal dynamics of the hemodynamic response function (HRF) to neural activation is separable from its spatial dynamics. Although there is empirical evidence that the HRF is more complex than suggested by space–time separable canonical HRF models, it is difficult to assess how much information about neural activity is lost when assuming space–time separability. In this study we directly test whether spatiotemporal variability in the HRF that is not captured by separable models contains information about neural signals. We predict intracranially measured neural activity from simultaneously recorded fMRI data using separable and non-separable spatiotemporal deconvolutions of voxel time series around the recording electrode. Our results show that abandoning the spatiotemporal separability assumption consistently improves the decoding accuracy of neural signals from fMRI data. We compare our findings with results from optical imaging and fMRI studies and discuss potential implications for classical fMRI analyses without invasive electrophysiological recordings.

NeurIPS Conference 2012 Conference Paper

Learning Invariant Representations of Molecules for Atomization Energy Prediction

• Grégoire Montavon
• Katja Hansen
• Siamac Fazli
• Matthias Rupp
• Franziska Biegler
• Andreas Ziehe
• Alexandre Tkatchenko
• Anatole Lilienfeld

The accurate prediction of molecular energetics in chemical compound space is a crucial ingredient for rational compound design. The inherently graph-like, non-vectorial nature of molecular data gives rise to a unique and difficult machine learning problem. In this paper, we adopt a learning-from-scratch approach where quantum-mechanical molecular energies are predicted directly from the raw molecular geometry. The study suggests a benefit from setting flexible priors and enforcing invariance stochastically rather than structurally. Our results improve the state-of-the-art by a factor of almost three, bringing statistical methods one step closer to the holy grail of ''chemical accuracy''.

YNIMG Journal 2011 Journal Article

Introduction to machine learning for brain imaging

• Steven Lemm
• Benjamin Blankertz
• Thorsten Dickhaus
• Klaus-Robert Müller

Machine learning and pattern recognition algorithms have in the past years developed to become a working horse in brain imaging and the computational neurosciences, as they are instrumental for mining vast amounts of neural data of ever increasing measurement precision and detecting minuscule signals from an overwhelming noise floor. They provide the means to decode and characterize task relevant brain states and to distinguish them from non-informative brain signals. While undoubtedly this machinery has helped to gain novel biological insights, it also holds the danger of potential unintentional abuse. Ideally machine learning techniques should be usable for any non-expert, however, unfortunately they are typically not. Overfitting and other pitfalls may occur and lead to spurious and nonsensical interpretation. The goal of this review is therefore to provide an accessible and clear introduction to the strengths and also the inherent dangers of machine learning usage in the neurosciences.

JMLR Journal 2011 Journal Article

Kernel Analysis of Deep Networks

• Grégoire Montavon
• Mikio L. Braun
• Klaus-Robert Müller

When training deep networks it is common knowledge that an efficient and well generalizing representation of the problem is formed. In this paper we aim to elucidate what makes the emerging representation successful. We analyze the layer-wise evolution of the representation in a deep network by building a sequence of deeper and deeper kernels that subsume the mapping performed by more and more layers of the deep network and measuring how these increasingly complex kernels fit the learning problem. We observe that deep networks create increasingly better representations of the learning problem and that the structure of the deep network controls how fast the representation of the task is formed layer after layer. [abs] [ pdf ][ bib ] &copy JMLR 2011. ( edit, beta )

YNIMG Journal 2011 Journal Article

Large-scale EEG/MEG source localization with spatial flexibility

• Stefan Haufe
• Ryota Tomioka
• Thorsten Dickhaus
• Claudia Sannelli
• Benjamin Blankertz
• Guido Nolte
• Klaus-Robert Müller

We propose a novel approach to solving the electro-/magnetoencephalographic (EEG/MEG) inverse problem which is based upon a decomposition of the current density into a small number of spatial basis fields. It is designed to recover multiple sources of possibly different extent and depth, while being invariant with respect to phase angles and rotations of the coordinate system. We demonstrate the method's ability to reconstruct simulated sources of random shape and show that the accuracy of the recovered sources can be increased, when interrelated field patterns are co-localized. Technically, this leads to large-scale mathematical problems, which are solved using recent advances in convex optimization. We apply our method for localizing brain areas involved in different types of motor imagery using real data from Brain–Computer Interface (BCI) sessions. Our approach based on single-trial localization of complex Fourier coefficients yields class-specific focal sources in the sensorimotor cortices.

YNIMG Journal 2011 Journal Article

Single-trial analysis and classification of ERP components — A tutorial

• Benjamin Blankertz
• Steven Lemm
• Matthias Treder
• Stefan Haufe
• Klaus-Robert Müller

Analyzing brain states that correspond to event related potentials (ERPs) on a single trial basis is a hard problem due to the high trial-to-trial variability and the unfavorable ratio between signal (ERP) and noise (artifacts and neural background activity). In this tutorial, we provide a comprehensive framework for decoding ERPs, elaborating on linear concepts, namely spatio-temporal patterns and filters as well as linear ERP classification. However, the bottleneck of these techniques is that they require an accurate covariance matrix estimation in high dimensional sensor spaces which is a highly intricate problem. As a remedy, we propose to use shrinkage estimators and show that appropriate regularization of linear discriminant analysis (LDA) by shrinkage yields excellent results for single-trial ERP classification that are far superior to classical LDA classification. Furthermore, we give practical hints on the interpretation of what classifiers learned from the data and demonstrate in particular that the trade-off between goodness-of-fit and model complexity in regularized LDA relates to a morphing between a difference pattern of ERPs and a spatial filter which cancels non task-related brain activity.

JMLR Journal 2011 Journal Article

The Stationary Subspace Analysis Toolbox

• Jan Saputra Müller
• Paul von Bünau
• Frank C. Meinecke
• Franz J. Király
• Klaus-Robert Müller

The Stationary Subspace Analysis (SSA) algorithm linearly factorizes a high-dimensional time series into stationary and non-stationary components. The SSA Toolbox is a platform-independent efficient stand-alone implementation of the SSA algorithm with a graphical user interface written in Java, that can also be invoked from the command line and from Matlab. The graphical interface guides the user through the whole process; data can be imported and exported from comma separated values (CSV) and Matlab's.mat files. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2011. ( edit, beta )

YNIMG Journal 2011 Journal Article

ℓ1-penalized linear mixed-effects models for high dimensional data with application to BCI

• Siamac Fazli
• Márton Danóczy
• Jürg Schelldorfer
• Klaus-Robert Müller

Recently, a novel statistical model has been proposed to estimate population effects and individual variability between subgroups simultaneously, by extending Lasso methods. We will for the first time apply this so-called ℓ1-penalized linear regression mixed-effects model for a large scale real world problem: we study a large set of brain computer interface data and through the novel estimator are able to obtain a subject-independent classifier that compares favorably with prior zero-training algorithms. This unifying model inherently compensates shifts in the input space attributed to the individuality of a subject. In particular we are now for the first time able to differentiate within-subject and between-subject variability. Thus a deeper understanding both of the underlying statistical and physiological structures of the data is gained.

YNIMG Journal 2010 Journal Article

A regularized discriminative framework for EEG analysis with application to brain–computer interface

• Ryota Tomioka
• Klaus-Robert Müller

We propose a framework for signal analysis of electroencephalography (EEG) that unifies tasks such as feature extraction, feature selection, feature combination, and classification, which are often independently tackled conventionally, under a regularized empirical risk minimization problem. The features are automatically learned, selected and combined through a convex optimization problem. Moreover we propose regularizers that induce novel types of sparsity providing a new technique for visualizing EEG of subjects during tasks from a discriminative point of view. The proposed framework is applied to two typical BCI problems, namely the P300 speller system and the prediction of self-paced finger tapping. In both datasets the proposed approach shows competitive performance against conventional methods, while at the same time the results are easier accessible to neurophysiological interpretation. Note that our novel approach is not only applicable to Brain imaging beyond EEG but also to general discriminative modeling of experimental paradigms beyond BCI.

JMLR Journal 2010 Journal Article

Approximate Tree Kernels

• Konrad Rieck
• Tammo Krueger
• Ulf Brefeld
• Klaus-Robert Müller

Convolution kernels for trees provide simple means for learning with tree-structured data. The computation time of tree kernels is quadratic in the size of the trees, since all pairs of nodes need to be compared. Thus, large parse trees, obtained from HTML documents or structured network data, render convolution kernels inapplicable. In this article, we propose an effective approximation technique for parse tree kernels. The approximate tree kernels (ATKs) limit kernel computation to a sparse subset of relevant subtrees and discard redundant structures, such that training and testing of kernel-based learning methods are significantly accelerated. We devise linear programming approaches for identifying such subsets for supervised and unsupervised learning tasks, respectively. Empirically, the approximate tree kernels attain run-time improvements up to three orders of magnitude while preserving the predictive accuracy of regular tree kernels. For unsupervised tasks, the approximate tree kernels even lead to more accurate predictions by identifying relevant dimensions in feature space. [abs] [ pdf ][ bib ] &copy JMLR 2010. ( edit, beta )

JMLR Journal 2010 Journal Article

How to Explain Individual Classification Decisions

• David Baehrens
• Timon Schroeter
• Stefan Harmeling
• Motoaki Kawanabe
• Katja Hansen
• Klaus-Robert Müller

After building a classifier with modern tools of machine learning we typically have a black box at hand that is able to predict well for unseen data. Thus, we get an answer to the question what is the most likely label of a given unseen data point. However, most methods will provide no answer why the model predicted a particular label for a single instance and what features were most influential for that particular instance. The only method that is currently able to provide such explanations are decision trees. This paper proposes a procedure which (based on a set of assumptions) allows to explain the decisions of any classification method. [abs] [ pdf ][ bib ] &copy JMLR 2010. ( edit, beta )

NeurIPS Conference 2010 Conference Paper

Layer-wise analysis of deep networks with Gaussian kernels

• Grégoire Montavon
• Klaus-Robert Müller
• Mikio Braun

Deep networks can potentially express a learning problem more efficiently than local learning machines. While deep networks outperform local learning machines on some problems, it is still unclear how their nice representation emerges from their complex structure. We present an analysis based on Gaussian kernels that measures how the representation of the learning problem evolves layer after layer as the deep network builds higher-level abstract representations of the input. We use this analysis to show empirically that deep networks build progressively better representations of the learning problem and that the best representations are obtained when the deep network discriminates only in the last layers.

YNIMG Journal 2010 Journal Article

Neurophysiological predictor of SMR-based BCI performance

• Benjamin Blankertz
• Claudia Sannelli
• Sebastian Halder
• Eva M. Hammer
• Andrea Kübler
• Klaus-Robert Müller
• Gabriel Curio
• Thorsten Dickhaus

Brain–computer interfaces (BCIs) allow a user to control a computer application by brain activity as measured, e. g. , by electroencephalography (EEG). After about 30years of BCI research, the success of control that is achieved by means of a BCI system still greatly varies between subjects. For about 20% of potential users the obtained accuracy does not reach the level criterion, meaning that BCI control is not accurate enough to control an application. The determination of factors that may serve to predict BCI performance, and the development of methods to quantify a predictor value from psychological and/or physiological data serve two purposes: a better understanding of the ‘BCI-illiteracy phenomenon’, and avoidance of a costly and eventually frustrating training procedure for participants who might not obtain BCI control. Furthermore, such predictors may lead to approaches to antagonize BCI illiteracy. Here, we propose a neurophysiological predictor of BCI performance which can be determined from a two minute recording of a ‘relax with eyes open’ condition using two Laplacian EEG channels. A correlation of r =0. 53 between the proposed predictor and BCI feedback performance was obtained on a large data base with N =80 BCI-naive participants in their first session with the Berlin brain–computer interface (BBCI) system which operates on modulations of sensory motor rhythms (SMRs).

NeurIPS Conference 2009 Conference Paper

Efficient and Accurate Lp-Norm Multiple Kernel Learning

• Marius Kloft
• Ulf Brefeld
• Pavel Laskov
• Klaus-Robert Müller
• Alexander Zien
• Sören Sonnenburg

Learning linear combinations of multiple kernels is an appealing strategy when the right choice of features is unknown. Previous approaches to multiple kernel learning (MKL) promote sparse kernel combinations and hence support interpretability. Unfortunately, L1-norm MKL is hardly observed to outperform trivial baselines in practical applications. To allow for robust kernel mixtures, we generalize MKL to arbitrary Lp-norms. We devise new insights on the connection between several existing MKL formulations and develop two efficient interleaved optimization strategies for arbitrary p>1. Empirically, we demonstrate that the interleaved optimization strategies are much faster compared to the traditionally used wrapper approaches. Finally, we apply Lp-norm MKL to real-world problems from computational biology, showing that non-sparse MKL achieves accuracies that go beyond the state-of-the-art.

NeurIPS Conference 2009 Conference Paper

Subject independent EEG-based BCI decoding

• Siamac Fazli
• Cristian Grozea
• Marton Danoczy
• Benjamin Blankertz
• Florin Popescu
• Klaus-Robert Müller

In the quest to make Brain Computer Interfacing (BCI) more usable, dry electrodes have emerged that get rid of the initial 30 minutes required for placing an electrode cap. Another time consuming step is the required individualized adaptation to the BCI user, which involves another 30 minutes calibration for assessing a subjects brain signature. In this paper we aim to also remove this calibration proceedure from BCI setup time by means of machine learning. In particular, we harvest a large database of EEG BCI motor imagination recordings (83 subjects) for constructing a library of subject-specific spatio-temporal filters and derive a subject independent BCI classifier. Our offline results indicate that BCI-na{i}ve users could start real-time BCI use with no prior calibration at only a very moderate performance loss. "

YNIMG Journal 2008 Journal Article

Combining sparsity and rotational invariance in EEG/MEG source reconstruction

• Stefan Haufe
• Vadim V. Nikulin
• Andreas Ziehe
• Klaus-Robert Müller
• Guido Nolte

We introduce Focal Vector Field Reconstruction (FVR), a novel technique for the inverse imaging of vector fields. The method was designed to simultaneously achieve two goals: a) invariance with respect to the orientation of the coordinate system, and b) a preference for sparsity of the solutions and their spatial derivatives. This was achieved by defining the regulating penalty function, which renders the solutions unique, as a global ℓ 1-norm of local ℓ 2-norms. We show that the method can be successfully used for solving the EEG inverse problem. In the joint localization of 2–3 simulated dipoles, FVR always reliably recovers the true sources. The competing methods have limitations in distinguishing close sources because their estimates are either too smooth (LORETA, Minimum ℓ 1 -norm) or too scattered (Minimum ℓ 2 -norm). In both noiseless and noisy simulations, FVR has the smallest localization error according to the Earth Mover's Distance (EMD), which is introduced here as a meaningful measure to compare arbitrary source distributions. We also apply the method to the simultaneous localization of left and right somatosensory N20 generators from real EEG recordings. Compared to its peers FVR was the only method that delivered correct location of the source in the somatosensory area of each hemisphere in accordance with neurophysiological prior knowledge.

NeurIPS Conference 2008 Conference Paper

Estimating vector fields using sparse basis field expansions

• Stefan Haufe
• Vadim Nikulin
• Andreas Ziehe
• Klaus-Robert Müller
• Guido Nolte

We introduce a novel framework for estimating vector fields using sparse basis field expansions (S-FLEX). The notion of basis fields, which are an extension of scalar basis functions, arises naturally in our framework from a rotational invariance requirement. We consider a regression setting as well as inverse problems. All variants discussed lead to second-order cone programming formulations. While our framework is generally applicable to any type of vector field, we focus in this paper on applying it to solving the EEG/MEG inverse problem. It is shown that significantly more precise and neurophysiologically more plausible location and shape estimates of cerebral current sources from EEG/MEG measurements become possible with our method when comparing to the state-of-the-art.

JMLR Journal 2008 Journal Article

On Relevant Dimensions in Kernel Feature Spaces

• Mikio L. Braun
• Joachim M. Buhmann
• Klaus-Robert Müller

We show that the relevant information of a supervised learning problem is contained up to negligible error in a finite number of leading kernel PCA components if the kernel matches the underlying learning problem in the sense that it can asymptotically represent the function to be learned and is sufficiently smooth. Thus, kernels do not only transform data sets such that good generalization can be achieved using only linear discriminant functions, but this transformation is also performed in a manner which makes economical use of feature space dimensions. In the best case, kernels provide efficient implicit representations of the data for supervised learning problems. Practically, we propose an algorithm which enables us to recover the number of leading kernel PCA components relevant for good classification. Our algorithm can therefore be applied (1) to analyze the interplay of data set and kernel in a geometric fashion, (2) to aid in model selection, and (3) to denoise in feature space in order to yield better classification results. [abs] [ pdf ][ bib ] &copy JMLR 2008. ( edit, beta )

NeurIPS Conference 2008 Conference Paper

Playing Pinball with non-invasive BCI

• Matthias Krauledat
• Konrad Grzeska
• Max Sagebaum
• Benjamin Blankertz
• Carmen Vidaurre
• Klaus-Robert Müller
• Michael Schröder

Compared to invasive Brain-Computer Interfaces (BCI), non-invasive BCI systems based on Electroencephalogram (EEG) signals have not been applied successfully for complex control tasks. In the present study, however, we demonstrate this is possible and report on the interaction of a human subject with a complex real device: a pinball machine. First results in this single subject study clearly show that fast and well-timed control well beyond chance level is possible, even though the environment is extremely rich and requires complex predictive behavior. Using machine learning methods for mental state decoding, BCI-based pinball control is possible within the first session without the necessity to employ lengthy subject training. While the current study is still of anecdotal nature, it clearly shows that very compelling control with excellent timing and dynamics is possible for a non-invasive BCI.

ICML Conference 2008 Conference Paper

Stopping conditions for exact computation of leave-one-out error in support vector machines

• Vojtech Franc
• Pavel Laskov
• Klaus-Robert Müller

ICML Conference 2007 Conference Paper

Asymptotic Bayesian generalization error when training and test distributions are different

• Keisuke Yamazaki
• Motoaki Kawanabe
• Sumio Watanabe
• Masashi Sugiyama
• Klaus-Robert Müller

In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian generalization error and establish its upper bound under a very general setting. Our important finding is that lower order terms---which can be ignored in the absence of the distribution change---play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.

JMLR Journal 2007 Journal Article

Covariate Shift Adaptation by Importance Weighted Cross Validation

• Masashi Sugiyama
• Matthias Krauledat
• Klaus-Robert Müller

A common assumption in supervised learning is that the input points in the training set follow the same probability distribution as the input points that will be given in the future test phase. However, this assumption is not satisfied, for example, when the outside of the training region is extrapolated. The situation where the training input points and test input points follow different distributions while the conditional distribution of output values given input points is unchanged is called the covariate shift. Under the covariate shift, standard model selection techniques such as cross validation do not work as desired since its unbiasedness is no longer maintained. In this paper, we propose a new method called importance weighted cross validation (IWCV), for which we prove its unbiasedness even under the covariate shift. The IWCV procedure is the only one that can be applied for unbiased classification under covariate shift, whereas alternatives to IWCV exist for regression. The usefulness of our proposed method is illustrated by simulations, and furthermore demonstrated in the brain-computer interface, where strong non-stationarity effects can be seen between training and test sessions. [abs] [ pdf ][ bib ] &copy JMLR 2007. ( edit, beta )

NeurIPS Conference 2007 Conference Paper

Heterogeneous Component Analysis

• Shigeyuki Oba
• Motoaki Kawanabe
• Klaus-Robert Müller
• Shin Ishii

In bioinformatics it is often desirable to combine data from various measurement sources and thus structured feature vectors are to be analyzed that possess different intrinsic blocking characteristics (e. g. , different patterns of missing values, obser- vation noise levels, effective intrinsic dimensionalities). We propose a new ma- chine learning tool, heterogeneous component analysis (HCA), for feature extrac- tion in order to better understand the factors that underlie such complex structured heterogeneous data. HCA is a linear block-wise sparse Bayesian PCA based not only on a probabilistic model with block-wise residual variance terms but also on a Bayesian treatment of a block-wise sparse factor-loading matrix. We study vari- ous algorithms that implement our HCA concept extracting sparse heterogeneous structure by obtaining common components for the blocks and specific compo- nents within each block. Simulations on toy and bioinformatics data underline the usefulness of the proposed structured matrix factorization concept.

NeurIPS Conference 2007 Conference Paper

Invariant Common Spatial Patterns: Alleviating Nonstationarities in Brain-Computer Interfacing

• Benjamin Blankertz
• Motoaki Kawanabe
• Ryota Tomioka
• Friederike Hohlefeld
• Klaus-Robert Müller
• Vadim Nikulin

Brain-Computer Interfaces can suffer from a large variance of the subject condi- tions within and across sessions. For example vigilance fluctuations in the indi- vidual, variable task involvement, workload etc. alter the characteristics of EEG signals and thus challenge a stable BCI operation. In the present work we aim to define features based on a variant of the common spatial patterns (CSP) algorithm that are constructed invariant with respect to such nonstationarities. We enforce invariance properties by adding terms to the denominator of a Rayleigh coefficient representation of CSP such as disturbance covariance matrices from fluctuations in visual processing. In this manner physiological prior knowledge can be used to shape the classification engine for BCI. As a proof of concept we present a BCI classifier that is robust to changes in the level of parietal a -activity. In other words, the EEG decoding still works when there are lapses in vigilance.

JMLR Journal 2007 Journal Article

The Need for Open Source Software in Machine Learning

• Sören Sonnenburg
• Mikio L. Braun
• Cheng Soon Ong
• Samy Bengio
• Leon Bottou
• Geoffrey Holmes
• Yann LeCun
• Klaus-Robert Müller

Open source tools have recently reached a level of maturity which makes them suitable for building large-scale real-world systems. At the same time, the field of machine learning has developed a large body of powerful learning algorithms for diverse applications. However, the true potential of these methods is not used, since existing implementations are not openly shared, resulting in software with low usability, and weak interoperability. We argue that this situation can be significantly improved by increasing incentives for researchers to publish their software under an open source model. Additionally, we outline the problems authors are faced with when trying to publish algorithmic implementations of machine learning methods. We believe that a resource of peer reviewed software accompanied by short articles would be highly valuable to both the machine learning and the general scientific community. [abs] [ pdf ][ bib ] &copy JMLR 2007. ( edit, beta )

YNIMG Journal 2007 Journal Article

The non-invasive Berlin Brain–Computer Interface: Fast acquisition of effective performance in untrained subjects

• Benjamin Blankertz
• Guido Dornhege
• Matthias Krauledat
• Klaus-Robert Müller
• Gabriel Curio

Brain–Computer Interface (BCI) systems establish a direct communication channel from the brain to an output device. These systems use brain signals recorded from the scalp, the surface of the cortex, or from inside the brain to enable users to control a variety of applications. BCI systems that bypass conventional motor output pathways of nerves and muscles can provide novel control options for paralyzed patients. One classical approach to establish EEG-based control is to set up a system that is controlled by a specific EEG feature which is known to be susceptible to conditioning and to let the subjects learn the voluntary control of that feature. In contrast, the Berlin Brain–Computer Interface (BBCI) uses well established motor competencies of its users and a machine learning approach to extract subject-specific patterns from high-dimensional features optimized for detecting the user's intent. Thus the long subject training is replaced by a short calibration measurement (20 min) and machine learning (1 min). We report results from a study in which 10 subjects, who had no or little experience with BCI feedback, controlled computer applications by voluntary imagination of limb movements: these intentions led to modulations of spontaneous brain activity specifically, somatotopically matched sensorimotor 7–30 Hz rhythms were diminished over pericentral cortices. The peak information transfer rate was above 35 bits per minute (bpm) for 3 subjects, above 23 bpm for two, and above 12 bpm for 3 subjects, while one subject could achieve no BCI control. Compared to other BCI systems which need longer subject training to achieve comparable results, we propose that the key to quick efficiency in the BBCI system is its flexibility due to complex but physiologically meaningful features and its adaptivity which respects the enormous inter-subject variability.

NeurIPS Conference 2006 Conference Paper

Denoising and Dimension Reduction in Feature Space

• Mikio Braun
• Klaus-Robert Müller
• Joachim Buhmann

We show that the relevant information about a classification problem in feature space is contained up to negligible error in a finite number of leading kernel PCA components if the kernel matches the underlying learning problem. Thus, kernels not only transform data sets such that good generalization can be achieved even by linear discriminant functions, but this transformation is also performed in a manner which makes economic use of feature space dimensions. In the best case, kernels provide efficient implicit representations of the data to perform classification. Practically, we propose an algorithm which enables us to recover the subspace and dimensionality relevant for good classification. Our algorithm can therefore be applied (1) to analyze the interplay of data set and kernel in a geometric fashion, (2) to help in model selection, and to (3) de-noise in feature space in order to yield better classification results.

JMLR Journal 2006 Journal Article

In Search of Non-Gaussian Components of a High-Dimensional Distribution

• Gilles Blanchard
• Motoaki Kawanabe
• Masashi Sugiyama
• Vladimir Spokoiny
• Klaus-Robert Müller

Finding non-Gaussian components of high-dimensional data is an important preprocessing step for efficient information processing. This article proposes a new linear method to identify the "non-Gaussian subspace" within a very general semi-parametric framework. Our proposed method, called NGCA (non-Gaussian component analysis), is based on a linear operator which, to any arbitrary nonlinear (smooth) function, associates a vector belonging to the low dimensional non-Gaussian target subspace, up to an estimation error. By applying this operator to a family of different nonlinear functions, one obtains a family of different vectors lying in a vicinity of the target space. As a final step, the target space itself is estimated by applying PCA to this family of vectors. We show that this procedure is consistent in the sense that the estimaton error tends to zero at a parametric rate, uniformly over the family, Numerical examples demonstrate the usefulness of our method. [abs] [ pdf ][ bib ] &copy JMLR 2006. ( edit, beta )

JMLR Journal 2006 Journal Article

Incremental Support Vector Learning: Analysis, Implementation and Applications

• Pavel Laskov
• Christian Gehl
• Stefan Krüger
• Klaus-Robert Müller

Incremental Support Vector Machines (SVM) are instrumental in practical applications of online learning. This work focuses on the design and analysis of efficient incremental SVM learning, with the aim of providing a fast, numerically stable and robust implementation. A detailed analysis of convergence and of algorithmic complexity of incremental SVM learning is carried out. Based on this analysis, a new design of storage and numerical operations is proposed, which speeds up the training of an incremental SVM by a factor of 5 to 20. The performance of the new algorithm is demonstrated in two scenarios: learning with limited resources and active learning. Various applications of the algorithm, such as in drug discovery, online monitoring of industrial devices and and surveillance of network traffic, can be foreseen. [abs] [ pdf ][ bib ] &copy JMLR 2006. ( edit, beta )

NeurIPS Conference 2006 Conference Paper

Inducing Metric Violations in Human Similarity Judgements

• Julian Laub
• Klaus-Robert Müller
• Felix A. Wichmann
• Jakob Macke

Attempting to model human categorization and similarity judgements is both a very interesting but also an exceedingly difficult challenge. Some of the difficulty arises because of conflicting evidence whether human categorization and similarity judgements should or should not be modelled as to operate on a mental representation that is essentially metric. Intuitively, this has a strong appeal as it would allow (dis)similarity to be represented geometrically as distance in some internal space. Here we show how a single stimulus, carefully constructed in a psychophysical experiment, introduces l2 violations in what used to be an internal similarity space that could be adequately modelled as Euclidean. We term this one influential data point a conflictual judgement. We present an algorithm of how to analyse such data and how to identify the crucial point. Thus there may not be a strict dichotomy between either a metric or a non-metric internal space but rather degrees to which potentially large subsets of stimuli are represented metrically with a small subset causing a global violation of metricity.

NeurIPS Conference 2006 Conference Paper

Logistic Regression for Single Trial EEG Classification

• Ryota Tomioka
• Kazuyuki Aihara
• Klaus-Robert Müller

We propose a novel framework for the classification of single trial ElectroEncephaloGraphy (EEG), based on regularized logistic regression. Framed in this robust statistical framework no prior feature extraction or outlier removal is required. We present two variations of parameterizing the regression function: (a) with a full rank symmetric matrix coefficient and (b) as a difference of two rank=1 matrices. In the first case, the problem is convex and the logistic regression is optimal under a generative model. The latter case is shown to be related to the Common Spatial Pattern (CSP) algorithm, which is a popular technique in Brain Computer Interfacing. The regression coefficients can also be topographically mapped onto the scalp similarly to CSP pro jections, which allows neuro-physiological interpretation. Simulations on 162 BCI datasets demonstrate that classification accuracy and robustness compares favorably against conventional CSP based classifiers.

NeurIPS Conference 2006 Conference Paper

Reducing Calibration Time For Brain-Computer Interfaces: A Clustering Approach

• Matthias Krauledat
• Michael Schröder
• Benjamin Blankertz
• Klaus-Robert Müller

Up to now even subjects that are experts in the use of machine learning based BCI systems still have to undergo a calibration session of about 20-30 min. From this data their (movement) intentions are so far infered. We now propose a new paradigm that allows to completely omit such calibration and instead transfer knowledge from prior sessions. To achieve this goal we first define normalized CSP features and distances in-between. Second, we derive prototypical features across sessions: (a) by clustering or (b) by feature concatenation methods. Finally, we construct a classifier based on these individualized prototypes and show that, indeed, classifiers can be successfully transferred to a new session for a number of subjects.

NeurIPS Conference 2005 Conference Paper

Analyzing Coupled Brain Sources: Distinguishing True from Spurious Interaction

• Guido Nolte
• Andreas Ziehe
• Frank Meinecke
• Klaus-Robert Müller

When trying to understand the brain, it is of fundamental importance to analyse (e. g. from EEG/MEG measurements) what parts of the cortex interact with each other in order to infer more accurate models of brain activity. Common techniques like Blind Source Separation (BSS) can estimate brain sources and single out artifacts by using the underlying assumption of source signal independence. However, physiologically interesting brain sources typically interact, so BSS will--by construction-- fail to characterize them properly. Noting that there are truly interacting sources and signals that only seemingly interact due to effects of volume conduction, this work aims to contribute by distinguishing these effects. For this a new BSS technique is proposed that uses anti-symmetrized cross-correlation matrices and subsequent diagonalization. The resulting decomposition consists of the truly interacting brain sources and suppresses any spurious interaction stemming from volume conduction. Our new concept of interacting source analysis (ISA) is successfully demonstrated on MEG data.

JMLR Journal 2005 Journal Article

Estimating Functions for Blind Separation When Sources Have Variance Dependencies

• Motoaki Kawanabe
• Klaus-Robert Müller

A blind separation problem where the sources are not independent, but have variance dependencies is discussed. For this scenario Hyvärinen and Hurri (2004) proposed an algorithm which requires no assumption on distributions of sources and no parametric model of dependencies between components. In this paper, we extend the semiparametric approach of Amari and Cardoso (1997) to variance dependencies and study estimating functions for blind separation of such dependent sources. In particular, we show that many ICA algorithms are applicable to the variance-dependent model as well under mild conditions, although they should in principle not. Our results indicate that separation can be done based only on normalized sources which are adjusted to have stationary variances and is not affected by the dependent activity levels. We also study the asymptotic distribution of the quasi maximum likelihood method and the stability of the natural gradient learning in detail. Simulation results of artificial and realistic examples match well with our theoretical findings. [abs] [ pdf ][ bib ] &copy JMLR 2005. ( edit, beta )

NeurIPS Conference 2005 Conference Paper

Non-Gaussian Component Analysis: a Semi-parametric Framework for Linear Dimension Reduction

• Gilles Blanchard
• Masashi Sugiyama
• Motoaki Kawanabe
• Vladimir Spokoiny
• Klaus-Robert Müller

We propose a new linear method for dimension reduction to identify nonGaussian components in high dimensional data. Our method, NGCA (non-Gaussian component analysis), uses a very general semi-parametric framework. In contrast to existing projection methods we define what is uninteresting (Gaussian): by projecting out uninterestingness, we can estimate the relevant non-Gaussian subspace. We show that the estimation error of finding the non-Gaussian components tends to zero at a parametric rate. Once NGCA components are identified and extracted, various tasks can be applied in the data analysis process, like data visualization, clustering, denoising or classification. A numerical study demonstrates the usefulness of our method.

NeurIPS Conference 2005 Conference Paper

Optimizing spatio-temporal filters for improving Brain-Computer Interfacing

• Guido Dornhege
• Benjamin Blankertz
• Matthias Krauledat
• Florian Losch
• Gabriel Curio
• Klaus-Robert Müller

Brain-Computer Interface (BCI) systems create a novel communication channel from the brain to an output device by bypassing conventional motor output pathways of nerves and muscles. Therefore they could provide a new communication and control option for paralyzed patients. Modern BCI technology is essentially based on techniques for the clas- sification of single-trial brain signals. Here we present a novel technique that allows the simultaneous optimization of a spatial and a spectral filter enhancing discriminability of multi-channel EEG single-trials. The eval- uation of 60 experiments involving 22 different subjects demonstrates the superiority of the proposed algorithm. Apart from the enhanced clas- sification, the spatial and/or the spectral filter that are determined by the algorithm can also be used for further analysis of the data, e. g. , for source localization of the respective brain rhythms.

JMLR Journal 2004 Journal Article

A Fast Algorithm for Joint Diagonalization with Non-orthogonal Transformations and its Application to Blind Source Separation

• Andreas Ziehe
• Pavel Laskov
• Guido Nolte
• Klaus-Robert Müller

A new efficient algorithm is presented for joint diagonalization of several matrices. The algorithm is based on the Frobenius-norm formulation of the joint diagonalization problem, and addresses diagonalization with a general, non-orthogonal transformation. The iterative scheme of the algorithm is based on a multiplicative update which ensures the invertibility of the diagonalizer. The algorithm's efficiency stems from the special approximation of the cost function resulting in a sparse, block-diagonal Hessian to be used in the computation of the quasi-Newton update step. Extensive numerical simulations illustrate the performance of the algorithm and provide a comparison to other leading diagonalization methods. The results of such comparison demonstrate that the proposed algorithm is a viable alternative to existing state-of-the-art joint diagonalization algorithms. The practical use of our algorithm is shown for blind source separation problems. [abs] [ pdf ] [ ps.gz ] [ ps ]

JMLR Journal 2004 Journal Article

Feature Discovery in Non-Metric Pairwise Data

• Julian Laub
• Klaus-Robert Müller

Pairwise proximity data, given as similarity or dissimilarity matrix, can violate metricity. This occurs either due to noise, fallible estimates, or due to intrinsic non-metric features such as they arise from human judgments. So far the problem of non-metric pairwise data has been tackled by essentially omitting the negative eigenvalues or shifting the spectrum of the associated (pseudo-)covariance matrix for a subsequent embedding. However, little attention has been paid to the negative part of the spectrum itself. In particular no answer was given to whether the directions associated to the negative eigenvalues would at all code variance other than noise related. We show by a simple, exploratory analysis that the negative eigenvalues can code for relevant structure in the data, thus leading to the discovery of new features, which were lost by conventional data analysis techniques. The information hidden in the negative eigenvalue part of the spectrum is illustrated and discussed for three data sets, namely USPS handwritten digits, text-mining and data from cognitive psychology. [abs] [ pdf ] [ ps.gz ] [ ps ]

JMLR Journal 2003 Journal Article

Blind Separation of Post-nonlinear Mixtures using Linearizing Transformations and Temporal Decorrelation

• Andreas Ziehe
• Motoaki Kawanabe
• Stefan Harmeling
• Klaus-Robert Müller

We propose two methods that reduce the post-nonlinear blind source separation problem (PNL-BSS) to a linear BSS problem. The first method is based on the concept of maximal correlation: we apply the alternating conditional expectation (ACE) algorithm---a powerful technique from non-parametric statistics---to approximately invert the componentwise non-linear functions. The second method is a Gaussianizing transformation, which is motivated by the fact that linearly mixed signals before nonlinear transformation are approximately Gaussian distributed. This heuristic, but simple and efficient procedure works as good as the ACE method. Using the framework provided by ACE, convergence can be proven. The optimal transformations obtained by ACE coincide with the sought-after inverse functions of the nonlinearities. After equalizing the nonlinearities, temporal decorrelation separation (TDSEP) allows us to recover the source signals. Numerical simulations testing "ACE-TD" and "Gauss-TD" on realistic examples are performed with excellent results. [abs] [ pdf ][ ps.gz ][ ps ]

NeurIPS Conference 2003 Conference Paper

Increase Information Transfer Rates in BCI by CSP Extension to Multi-class

• Guido Dornhege
• Benjamin Blankertz
• Gabriel Curio
• Klaus-Robert Müller

Brain-Computer Interfaces (BCI) are an interesting emerging technology that is driven by the motivation to develop an effective communication in- terface translating human intentions into a control signal for devices like computers or neuroprostheses. If this can be done bypassing the usual hu- man output pathways like peripheral nerves and muscles it can ultimately become a valuable tool for paralyzed patients. Most activity in BCI re- search is devoted to finding suitable features and algorithms to increase information transfer rates (ITRs). The present paper studies the implica- tions of using more classes, e. g. , left vs. right hand vs. foot, for operating a BCI. We contribute by (1) a theoretical study showing under some mild assumptions that it is practically not useful to employ more than three or four classes, (2) two extensions of the common spatial pattern (CSP) algorithm, one interestingly based on simultaneous diagonalization, and (3) controlled EEG experiments that underline our theoretical findings and show excellent improved ITRs.

NeurIPS Conference 2002 Conference Paper

Clustering with the Fisher Score

• Koji Tsuda
• Motoaki Kawanabe
• Klaus-Robert Müller

Recently the Fisher score (or the Fisher kernel) is increasingly used as a feature extractor for classification problems. The Fisher score is a vector of parameter derivatives of loglikelihood of a probabilistic model. This paper gives a theoretical analysis about how class information is pre- served in the space of the Fisher score, which turns out that the Fisher score consists of a few important dimensions with class information and many nuisance dimensions. When we perform clustering with the Fisher score, K-Means type methods are obviously inappropriate because they make use of all dimensions. So we will develop a novel but simple clus- tering algorithm specialized for the Fisher score, which can exploit im- portant dimensions. This algorithm is successfully tested in experiments with artificial data and real data (amino acid sequences). as follows: Let us assume that a probabilistic model   parameter estimate 

NeurIPS Conference 2002 Conference Paper

Combining Features for BCI

• Guido Dornhege
• Benjamin Blankertz
• Gabriel Curio
• Klaus-Robert Müller

Recently, interest is growing to develop an effective communication in- terface connecting the human brain to a computer, the ’Brain-Computer Interface’ (BCI). One motivation of BCI research is to provide a new communication channel substituting normal motor output in patients with severe neuromuscular disabilities. In the last decade, various neuro- physiological cortical processes, such as slow potential shifts, movement related potentials (MRPs) or event-related desynchronization (ERD) of spontaneous EEG rhythms, were shown to be suitable for BCI, and, con- sequently, different independent approaches of extracting BCI-relevant EEG-features for single-trial analysis are under investigation. Here, we present and systematically compare several concepts for combining such EEG-features to improve the single-trial classification. Feature combi- nations are evaluated on movement imagination experiments with 3 sub- jects where EEG-features are based on either MRPs or ERD, or both. Those combination methods that incorporate the assumption that the sin- gle EEG-features are physiologically mutually independent outperform the plain method of ’adding’ evidence where the single-feature vectors are simply concatenated. These results strengthen the hypothesis that MRP and ERD reflect at least partially independent aspects of cortical processes and open a new perspective to boost BCI effectiveness.

NeurIPS Conference 2002 Conference Paper

Going Metric: Denoising Pairwise Data

• Volker Roth
• Julian Laub
• Klaus-Robert Müller
• Joachim Buhmann

Pairwise data in empirical sciences typically violate metricity, ei(cid: 173) ther due to noise or due to fallible estimates, and therefore are hard to analyze by conventional machine learning technology. In this paper we therefore study ways to work around this problem. First, we present an alternative embedding to multi-dimensional scaling (MDS) that allows us to apply a variety of classical ma(cid: 173) chine learning and signal processing algorithms. The class of pair(cid: 173) wise grouping algorithms which share the shift-invariance property is statistically invariant under this embedding procedure, leading to identical assignments of objects to clusters. Based on this new vectorial representation, denoising methods are applied in a sec(cid: 173) ond step. Both steps provide a theoretically well controlled setup to translate from pairwise data to the respective denoised met(cid: 173) ric representation. We demonstrate the practical usefulness of our theoretical reasoning by discovering structure in protein sequence data bases, visibly improving performance upon existing automatic methods.

JMLR Journal 2002 Journal Article

The Subspace Information Criterion for Infinite Dimensional Hypothesis Spaces (Kernel Machines Section)

• Masashi Sugiyama
• Klaus-Robert Müller

A central problem in learning is selection of an appropriate model. This is typically done by estimating the unknown generalization errors of a set of models to be selected from and then choosing the model with minimal generalization error estimate. In this article, we discuss the problem of model selection and generalization error estimation in the context of kernel regression models, e.g., kernel ridge regression, kernel subset regression or Gaussian process regression. Previously, a non-asymptotic generalization error estimator called the subspace information criterion (SIC) was proposed, that could be successfully applied to finite dimensional subspace models. SIC is an unbiased estimator of the generalization error for the finite sample case under the conditions that the learning target function belongs to a specified reproducing kernel Hilbert space (RKHS) H and the reproducing kernels centered on training sample points span the whole space H. These conditions hold only if dim H < l, where l < infinity is the number of training examples. Therefore, SIC could be applied only to finite dimensional RKHSs. In this paper, we extend the range of applicability of SIC, and show that even if the reproducing kernels centered on training sample points do not span the whole space H, SIC is an unbiased estimator of an essential part of the generalization error. Our extension allows the use of any RKHSs including infinite dimensional ones, i.e., richer function classes commonly used in Gaussian processes, support vector machines or boosting. We further show that when the kernel matrix is invertible, SIC can be expressed in a much simpler form, making its computation highly efficient. In computer simulations on ridge parameter selection with real and artificial data sets, SIC is compared favorably with other standard model selection techniques for instance leave-one-out cross-validation or an empirical Bayesian method.

NeurIPS Conference 2001 Conference Paper

A New Discriminative Kernel From Probabilistic Models

• Koji Tsuda
• Motoaki Kawanabe
• Gunnar Rätsch
• Sören Sonnenburg
• Klaus-Robert Müller

Recently, Jaakkola and Haussler proposed a method for construct(cid: 173) ing kernel functions from probabilistic models. Their so called "Fisher kernel" has been combined with discriminative classifiers such as SVM and applied successfully in e. g. DNA and protein analysis. Whereas the Fisher kernel (FK) is calculated from the marginal log-likelihood, we propose the TOP kernel derived from Tangent vectors Of Posterior log-odds. Furthermore we develop a theoretical framework on feature extractors from probabilistic models and use it for analyzing FK and TOP. In experiments our new discriminative TOP kernel compares favorably to the Fisher kernel.

NeurIPS Conference 2001 Conference Paper

Classifying Single Trial EEG: Towards Brain Computer Interfacing

• Benjamin Blankertz
• Gabriel Curio
• Klaus-Robert Müller

Driven by the progress in the field of single-trial analysis of EEG, there is a growing interest in brain computer interfaces (BCIs), i. e. , systems that enable human subjects to control a computer only by means of their brain signals. In a pseudo-online simulation our BCI detects upcoming finger movements in a natural keyboard typing condition and predicts their lat- erality. This can be done on average 100–230 ms before the respective key is actually pressed, i. e. , long before the onset of EMG. Our approach is appealing for its short response time and high classification accuracy (>96%) in a binary decision where no human training is involved. We compare discriminative classifiers like Support Vector Machines (SVMs) and different variants of Fisher Discriminant that possess favorable reg- ularization properties for dealing with high noise cases (inter-trial vari- ablity).

NeurIPS Conference 2001 Conference Paper

Estimating the Reliability of ICA Projections

• Frank Meinecke
• Andreas Ziehe
• Motoaki Kawanabe
• Klaus-Robert Müller

When applying unsupervised learning techniques like ICA or tem(cid: 173) poral decorrelation, a key question is whether the discovered pro(cid: 173) jections are reliable. In other words: can we give error bars or can we assess the quality of our separation? We use resampling meth(cid: 173) ods to tackle these questions and show experimentally that our proposed variance estimations are strongly correlated to the sepa(cid: 173) ration error. We demonstrate that this reliability estimation can be used to choose the appropriate ICA-model, to enhance signifi(cid: 173) cantly the separation performance, and, most important, to mark the components that have a actual physical meaning. Application to 49-channel-data from an magneto encephalography (MEG) ex(cid: 173) periment underlines the usefulness of our approach.

NeurIPS Conference 2001 Conference Paper

Kernel Feature Spaces and Nonlinear Blind Souce Separation

• Stefan Harmeling
• Andreas Ziehe
• Motoaki Kawanabe
• Klaus-Robert Müller

In kernel based learning the data is mapped to a kernel feature space of a dimension that corresponds to the number of training data points. In practice, however, the data forms a smaller submanifold in feature space, a fact that has been used e. g. by reduced set techniques for SVMs. We propose a new mathematical construction that permits to adapt to the in- trinsic dimension and to find an orthonormal basis of this submanifold. In doing so, computations get much simpler and more important our theoretical framework allows to derive elegant kernelized blind source separation (BSS) algorithms for arbitrary invertible nonlinear mixings. Experiments demonstrate the good performance and high computational efficiency of our kTDSEP algorithm for the problem of nonlinear BSS.

NeurIPS Conference 2000 Conference Paper

A Mathematical Programming Approach to the Kernel Fisher Algorithm

• Sebastian Mika
• Gunnar Rätsch
• Klaus-Robert Müller

We investigate a new kernel-based classifier: the Kernel Fisher Discrim(cid: 173) inant (KFD). A mathematical programming formulation based on the ob(cid: 173) servation that KFD maximizes the average margin permits an interesting modification of the original KFD algorithm yielding the sparse KFD. We find that both, KFD and the proposed sparse KFD, can be understood in an unifying probabilistic context. Furthermore, we show connections to Support Vector Machines and Relevance Vector Machines. From this understanding, we are able to outline an interesting kernel-regression technique based upon the KFD algorithm. Simulations support the use(cid: 173) fulness of our approach.

NeurIPS Conference 1999 Conference Paper

Invariant Feature Extraction and Classification in Kernel Spaces

• Sebastian Mika
• Gunnar Rätsch
• Jason Weston
• Bernhard Schölkopf
• Alex Smola
• Klaus-Robert Müller

In hyperspectral imagery one pixel typically consists of a mixture of the reflectance spectra of several materials, where the mixture coefficients correspond to the abundances of the constituting ma(cid: 173) terials. We assume linear combinations of reflectance spectra with some additive normal sensor noise and derive a probabilistic MAP framework for analyzing hyperspectral data. As the material re(cid: 173) flectance characteristics are not know a priori, we face the problem of unsupervised linear unmixing. The incorporation of different prior information (e. g. positivity and normalization of the abun(cid: 173) dances) naturally leads to a family of interesting algorithms, for example in the noise-free case yielding an algorithm that can be understood as constrained independent component analysis (ICA). Simulations underline the usefulness of our theory.

NeurIPS Conference 1999 Conference Paper

Unmixing Hyperspectral Data

• Lucas Parra
• Clay Spence
• Paul Sajda
• Andreas Ziehe
• Klaus-Robert Müller

In hyperspectral imagery one pixel typically consists of a mixture of the reflectance spectra of several materials, where the mixture coefficients correspond to the abundances of the constituting ma(cid: 173) terials. We assume linear combinations of reflectance spectra with some additive normal sensor noise and derive a probabilistic MAP framework for analyzing hyperspectral data. As the material re(cid: 173) flectance characteristics are not know a priori, we face the problem of unsupervised linear unmixing. The incorporation of different prior information (e. g. positivity and normalization of the abun(cid: 173) dances) naturally leads to a family of interesting algorithms, for example in the noise-free case yielding an algorithm that can be understood as constrained independent component analysis (ICA). Simulations underline the usefulness of our theory.

NeurIPS Conference 1999 Conference Paper

v-Arc: Ensemble Learning in the Presence of Outliers

• Gunnar Rätsch
• Bernhard Schölkopf
• Alex Smola
• Klaus-Robert Müller
• Takashi Onoda
• Sebastian Mika

AdaBoost and other ensemble methods have successfully been ap(cid: 173) plied to a number of classification tasks, seemingly defying prob(cid: 173) lems of overfitting. AdaBoost performs gradient descent in an error function with respect to the margin, asymptotically concentrating on the patterns which are hardest to learn. For very noisy prob(cid: 173) lems, however, this can be disadvantageous. Indeed, theoretical analysis has shown that the margin distribution, as opposed to just the minimal margin, plays a crucial role in understanding this phe(cid: 173) nomenon. Loosely speaking, some outliers should be tolerated if this has the benefit of substantially increasing the margin on the remaining points. We propose a new boosting algorithm which al(cid: 173) lows for the possibility of a pre-specified fraction of points to lie in the margin area Or even on the wrong side of the decision boundary.

NeurIPS Conference 1998 Conference Paper

Kernel PCA and De-Noising in Feature Spaces

• Sebastian Mika
• Bernhard Schölkopf
• Alex Smola
• Klaus-Robert Müller
• Matthias Scholz
• Gunnar Rätsch

Kernel PCA as a nonlinear feature extractor has proven powerful as a preprocessing step for classification algorithms. But it can also be con(cid: 173) sidered as a natural generalization of linear principal component anal(cid: 173) ysis. This gives rise to the question how to use nonlinear features for data compression, reconstruction, and de-noising, applications common in linear PCA. This is a nontrivial task, as the results provided by ker(cid: 173) nel PCA live in some high dimensional feature space and need not have pre-images in input space. This work presents ideas for finding approxi(cid: 173) mate pre-images, focusing on Gaussian kernels, and shows experimental results using these pre-images in data reconstruction and de-noising on toy examples as well as on real world data. 1 peA and Feature Spaces Principal Component Analysis (PC A) (e. g. [3]) is an orthogonal basis transformation. The new basis is found by diagonalizing the centered covariance matrix of a data set {Xk E RNlk = 1, .. ., f}, defined by C = ((Xi - nates in the Eigenvector basis are called principal components. The size of an Eigenvalue >. corresponding to an Eigenvector v of C equals the amount of variance in the direction of v. Furthermore, the directions of the first n Eigenvectors corresponding to the biggest n Eigenvalues cover as much variance as possible by n orthogonal directions. In many ap(cid: 173) plications they contain the most interesting information: for instance, in data compression, where we project onto the directions with biggest variance to retain as much information as possible, or in de-noising, where we deliberately drop directions with small variance. (Xk))T). The coordi(cid: 173)

NeurIPS Conference 1997 Conference Paper

Analysis of Drifting Dynamics with Neural Network Hidden Markov Models

• Jens Kohlmorgen
• Klaus-Robert Müller
• Klaus Pawelzik

We present a method for the analysis of nonstationary time se(cid: 173) ries with multiple operating modes. In particular, it is possible to detect and to model both a switching of the dynamics and a less abrupt, time consuming drift from one mode to another. This is achieved in two steps. First, an unsupervised training method pro(cid: 173) vides prediction experts for the inherent dynamical modes. Then, the trained experts are used in a hidden Markov model that allows to model drifts. An application to physiological wake/sleep data demonstrates that analysis and modeling of real-world time series can be improved when the drift paradigm is taken into account.

NeurIPS Conference 1996 Conference Paper

Adaptive On-line Learning in Changing Environments

• Noboru Murata
• Klaus-Robert Müller
• Andreas Ziehe
• Shun-ichi Amari

An adaptive on-line algorithm extending the learning of learning idea is proposed and theoretically motivated. Relying only on gra(cid: 173) dient flow information it can be applied to learning continuous functions or distributions, even when no explicit loss function is gi(cid: 173) ven and the Hessian is not available. Its efficiency is demonstrated for a non-stationary blind separation task of acoustic signals.

NeurIPS Conference 1995 Conference Paper

Statistical Theory of Overtraining - Is Cross-Validation Asymptotically Effective?

• Shun-ichi Amari
• Noboru Murata
• Klaus-Robert Müller
• Michael Finke
• Howard Yang

A statistical theory for overtraining is proposed. The analysis treats realizable stochastic neural networks, trained with Kullback(cid: 173) Leibler loss in the asymptotic case. It is shown that the asymptotic gain in the generalization error is small if we perform early stop(cid: 173) ping, even if we have access to the optimal stopping time. Consider(cid: 173) ing cross-validation stopping we answer the question: In what ratio the examples should be divided into training and testing sets in or(cid: 173) der to obtain the optimum performance. In the non-asymptotic region cross-validated early stopping always decreases the general(cid: 173) ization error. Our large scale simulations done on a CM5 are in nice agreement with our analytical findings.