Author name cluster

Sebastian Mika

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

ICML Conference 2004 Conference Paper

A kernel view of the dimensionality reduction of manifolds

Jihun Ham
Daniel D. Lee
Sebastian Mika
Bernhard Schölkopf

We interpret several well-known algorithms for dimensionality reduction of manifolds as kernel methods. Isomap, graph Laplacian eigenmap, and locally linear embedding (LLE) all utilize local neighborhood information to construct a global embedding of the manifold. We show how all three algorithms can be described as kernel PCA on specially constructed Gram matrices, and illustrate the similarities and differences between the algorithms with representative examples.

Details

NeurIPS Conference 2002 Conference Paper

Adapting Codes and Embeddings for Polychotomies

Gunnar Rätsch
Sebastian Mika
Alex Smola

In this paper we consider formulations of multi-class problems based on a generalized notion of a margin and using output coding. This includes, but is not restricted to, standard multi-class SVM formulations. Differ- ently from many previous approaches we learn the code as well as the embedding function. We illustrate how this can lead to a formulation that allows for solving a wider range of problems with for instance many classes or even “missing classes”. To keep our optimization problems tractable we propose an algorithm capable of solving them using two- class classiﬁers, similar in spirit to Boosting.

PDF Details

NeurIPS Conference 2001 Conference Paper

On the Convergence of Leveraging

Gunnar Rätsch
Sebastian Mika
Manfred K. Warmuth

We give an unified convergence analysis of ensemble learning meth- ods including e. g. AdaBoost, Logistic Regression and the Least-Square- Boost algorithm for regression. These methods have in common that they iteratively call a base learning algorithm which returns hypotheses that are then linearly combined. We show that these methods are related to the Gauss-Southwell method known from numerical optimization and state non-asymptotical convergence results for all these methods. Our analysis includes ` 1 -norm regularized cost functions leading to a clean and general way to regularize ensemble learning. 1 Introduction We show convergence rates of ensemble learning methods such as AdaBoost [10], Logistic Regression (LR) [11, 5] and the Least-Square (LS) regression algorithm called LS-Boost [12]. These algorithms have in common that they iteratively call a base learning algorithm L (also called weak learner) on a weighted training sample. The base learner is expected to return in each iteration t a hypothesis ^ h t from some hypothesis set of weak hypotheses H that has small weighted training error. This is the weighted number of false predictions in classification and weighted estimation error in regression. These hypotheses are then linearly combined to form the final hypothesis f ^ (x) =

PDF Details

JMLR Journal 2001 Journal Article

Regularized Principal Manifolds (Kernel Machines Section)

Alexander J. Smola
Sebastian Mika
Bernhard Schölkopf
Robert C. Williamson

Many settings of unsupervised learning can be viewed as quantization problems - the minimization of the expected quantization error subject to some restrictions. This allows the use of tools such as regularization from the theory of (supervised) risk minimization for unsupervised learning. This setting turns out to be closely related to principal curves, the generative topographic map, and robust coding. We explore this connection in two ways: (1) we propose an algorithm for finding principal manifolds that can be regularized in a variety of ways; and (2) we derive uniform convergence bounds and hence bounds on the learning rates of the algorithm. In particular, we give bounds on the covering numbers which allows us to obtain nearly optimal learning rates for certain types of regularization operators. Experimental results demonstrate the feasibility of the approach.

PDF Details

NeurIPS Conference 2000 Conference Paper

A Mathematical Programming Approach to the Kernel Fisher Algorithm

Sebastian Mika
Gunnar Rätsch
Klaus-Robert Müller

We investigate a new kernel-based classifier: the Kernel Fisher Discrim(cid: 173) inant (KFD). A mathematical programming formulation based on the ob(cid: 173) servation that KFD maximizes the average margin permits an interesting modification of the original KFD algorithm yielding the sparse KFD. We find that both, KFD and the proposed sparse KFD, can be understood in an unifying probabilistic context. Furthermore, we show connections to Support Vector Machines and Relevance Vector Machines. From this understanding, we are able to outline an interesting kernel-regression technique based upon the KFD algorithm. Simulations support the use(cid: 173) fulness of our approach.

PDF Details

NeurIPS Conference 1999 Conference Paper

Invariant Feature Extraction and Classification in Kernel Spaces

Sebastian Mika
Gunnar Rätsch
Jason Weston
Bernhard Schölkopf
Alex Smola
Klaus-Robert Müller

In hyperspectral imagery one pixel typically consists of a mixture of the reflectance spectra of several materials, where the mixture coefficients correspond to the abundances of the constituting ma(cid: 173) terials. We assume linear combinations of reflectance spectra with some additive normal sensor noise and derive a probabilistic MAP framework for analyzing hyperspectral data. As the material re(cid: 173) flectance characteristics are not know a priori, we face the problem of unsupervised linear unmixing. The incorporation of different prior information (e. g. positivity and normalization of the abun(cid: 173) dances) naturally leads to a family of interesting algorithms, for example in the noise-free case yielding an algorithm that can be understood as constrained independent component analysis (ICA). Simulations underline the usefulness of our theory.

PDF Details

NeurIPS Conference 1999 Conference Paper

v-Arc: Ensemble Learning in the Presence of Outliers

Gunnar Rätsch
Bernhard Schölkopf
Alex Smola
Klaus-Robert Müller
Takashi Onoda
Sebastian Mika

AdaBoost and other ensemble methods have successfully been ap(cid: 173) plied to a number of classification tasks, seemingly defying prob(cid: 173) lems of overfitting. AdaBoost performs gradient descent in an error function with respect to the margin, asymptotically concentrating on the patterns which are hardest to learn. For very noisy prob(cid: 173) lems, however, this can be disadvantageous. Indeed, theoretical analysis has shown that the margin distribution, as opposed to just the minimal margin, plays a crucial role in understanding this phe(cid: 173) nomenon. Loosely speaking, some outliers should be tolerated if this has the benefit of substantially increasing the margin on the remaining points. We propose a new boosting algorithm which al(cid: 173) lows for the possibility of a pre-specified fraction of points to lie in the margin area Or even on the wrong side of the decision boundary.

PDF Details

NeurIPS Conference 1998 Conference Paper

Kernel PCA and De-Noising in Feature Spaces

Sebastian Mika
Bernhard Schölkopf
Alex Smola
Klaus-Robert Müller
Matthias Scholz
Gunnar Rätsch

Kernel PCA as a nonlinear feature extractor has proven powerful as a preprocessing step for classification algorithms. But it can also be con(cid: 173) sidered as a natural generalization of linear principal component anal(cid: 173) ysis. This gives rise to the question how to use nonlinear features for data compression, reconstruction, and de-noising, applications common in linear PCA. This is a nontrivial task, as the results provided by ker(cid: 173) nel PCA live in some high dimensional feature space and need not have pre-images in input space. This work presents ideas for finding approxi(cid: 173) mate pre-images, focusing on Gaussian kernels, and shows experimental results using these pre-images in data reconstruction and de-noising on toy examples as well as on real world data. 1 peA and Feature Spaces Principal Component Analysis (PC A) (e. g. [3]) is an orthogonal basis transformation. The new basis is found by diagonalizing the centered covariance matrix of a data set {Xk E RNlk = 1, .. ., f}, defined by C = ((Xi - nates in the Eigenvector basis are called principal components. The size of an Eigenvalue >. corresponding to an Eigenvector v of C equals the amount of variance in the direction of v. Furthermore, the directions of the first n Eigenvectors corresponding to the biggest n Eigenvalues cover as much variance as possible by n orthogonal directions. In many ap(cid: 173) plications they contain the most interesting information: for instance, in data compression, where we project onto the directions with biggest variance to retain as much information as possible, or in de-noising, where we deliberately drop directions with small variance. (Xk))T). The coordi(cid: 173)

PDF Details