Author name cluster

Robert Tibshirani

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

10 papers

1 author row

JMLR Journal 2026 Journal Article

Adaptive Forward Stepwise: A Method for High Sparsity Regression

Ivy Zhang
Robert Tibshirani

This paper proposes a sparse regression method that continuously interpolates between Forward Stepwise selection (FS) and the LASSO. When tuned appropriately, our solutions are much sparser than typical LASSO fits but, unlike FS fits, benefit from the stabilizing effect of shrinkage. Our method, Adaptive Forward Stepwise Regression (AFS) addresses the need for sparser models with shrinkage. We show its connection with boosting via a soft-thresholding viewpoint and demonstrate the ease of adapting the method to classification tasks. In both simulations and real data, our method has lower mean squared error and fewer selected features across multiple settings compared to popular sparse modeling procedures. [abs] [ pdf ][ bib ] &copy JMLR 2026. ( edit, beta )

PDF Details

JMLR Journal 2021 Journal Article

LassoNet: A Neural Network with Feature Sparsity

Ismael Lemhadri
Feng Ruan
Louis Abraham
Robert Tibshirani

Much work has been done recently to make neural networks more interpretable, and one approach is to arrange for the network to use only a subset of the available features. In linear models, Lasso (or $\ell_1$-regularized) regression assigns zero weights to the most irrelevant or redundant features, and is widely used in data science. However the Lasso only applies to linear models. Here we introduce LassoNet, a neural network framework with global feature selection. Our approach achieves feature sparsity by adding a skip (residual) layer and allowing a feature to participate in any hidden layer only if its skip-layer representative is active. Unlike other approaches to feature selection for neural nets, our method uses a modified objective function with constraints, and so integrates feature selection with the parameter learning directly. As a result, it delivers an entire regularization path of solutions with a range of feature sparsity. We apply LassoNet to a number of real-data problems and find that it significantly outperforms state-of-the-art methods for feature selection and regression. LassoNet uses projected proximal gradient descent, and generalizes directly to deep networks. It can be implemented by adding just a few lines of code to a standard neural network. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2021. ( edit, beta )

PDF Details

JMLR Journal 2010 Journal Article

Spectral Regularization Algorithms for Learning Large Incomplete Matrices

Rahul Mazumder
Trevor Hastie
Robert Tibshirani

We use convex relaxation techniques to provide a sequence of regularized low-rank solutions for large-scale matrix completion problems. Using the nuclear norm as a regularizer, we provide a simple and very efficient convex algorithm for minimizing the reconstruction error subject to a bound on the nuclear norm. Our algorithm SOFT-IMPUTE iteratively replaces the missing elements with those obtained from a soft-thresholded SVD. With warm starts this allows us to efficiently compute an entire regularization path of solutions on a grid of values of the regularization parameter. The computationally intensive part of our algorithm is in computing a low-rank SVD of a dense matrix. Exploiting the problem structure, we show that the task can be performed with a complexity of order linear in the matrix dimensions. Our semidefinite-programming algorithm is readily scalable to large matrices; for example SOFT-IMPUTE takes a few hours to compute low-rank approximations of a 10 6 X 10 6 incomplete matrix with 10 7 observed entries, and fits a rank- 95 approximation to the full Netflix training set in 3.3 hours. Our methods achieve good training and test errors and exhibit superior timings when compared to other competitive state-of-the-art techniques. [abs] [ pdf ][ bib ] &copy JMLR 2010. ( edit, beta )

PDF Details

JMLR Journal 2009 Journal Article

Estimation of Sparse Binary Pairwise Markov Networks using Pseudo-likelihoods

Holger Höfling
Robert Tibshirani

We consider the problems of estimating the parameters as well as the structure of binary-valued Markov networks. For maximizing the penalized log-likelihood, we implement an approximate procedure based on the pseudo-likelihood of Besag (1975) and generalize it to a fast exact algorithm. The exact algorithm starts with the pseudo-likelihood solution and then adjusts the pseudo-likelihood criterion so that each additional iterations moves it closer to the exact solution. Our results show that this procedure is faster than the competing exact method proposed by Lee, Ganapathi, and Koller (2006a). However, we also find that the approximate pseudo-likelihood as well as the approaches of Wainwright et al. (2006), when implemented using the coordinate descent procedure of Friedman, Hastie, and Tibshirani (2008b), are much faster than the exact methods, and only slightly less accurate. [abs] [ pdf ][ bib ] &copy JMLR 2009. ( edit, beta )

PDF Details

JMLR Journal 2007 Journal Article

Margin Trees for High-dimensional Classification

Robert Tibshirani
Trevor Hastie

We propose a method for the classification of more than two classes, from high-dimensional features. Our approach is to build a binary decision tree in a top-down manner, using the optimal margin classifier at each split. We implement an exact greedy algorithm for this task, and compare its performance to less greedy procedures based on clustering of the matrix of pairwise margins. We compare the performance of the "margin tree" to the closely related "all-pairs" (one versus one) support vector machine, and nearest centroids on a number of cancer microarray data sets. We also develop a simple method for feature selection. We find that the margin tree has accuracy that is competitive with other methods and offers additional interpretability in its putative grouping of the classes. [abs] [ pdf ][ bib ] &copy JMLR 2007. ( edit, beta )

PDF Details

NeurIPS Conference 2004 Conference Paper

The Entire Regularization Path for the Support Vector Machine

Saharon Rosset
Robert Tibshirani
Ji Zhu
Trevor Hastie

In this paper we argue that the choice of the SVM cost parameter can be critical. We then derive an algorithm that can fit the entire path of SVM solutions for every value of the cost parameter, with essentially the same computational cost as fitting one SVM model.

PDF Details

JMLR Journal 2004 Journal Article

The Entire Regularization Path for the Support Vector Machine

Trevor Hastie
Saharon Rosset
Robert Tibshirani
Ji Zhu

The support vector machine (SVM) is a widely used tool for classification. Many efficient implementations exist for fitting a two-class SVM model. The user has to supply values for the tuning parameters: the regularization cost parameter, and the kernel parameters. It seems a common practice is to use a default value for the cost parameter, often leading to the least restrictive model. In this paper we argue that the choice of the cost parameter can be critical. We then derive an algorithm that can fit the entire path of SVM solutions for every value of the cost parameter, with essentially the same computational cost as fitting one SVM model. We illustrate our algorithm on some examples, and use our representation to give further insight into the range of SVM solutions. [abs] [ pdf ]

PDF Details

NeurIPS Conference 2003 Conference Paper

1-norm Support Vector Machines

Ji Zhu
Saharon Rosset
Robert Tibshirani
Trevor Hastie

The standard 2-norm SVM is known for its good performance in two- In this paper, we consider the 1-norm SVM. We class classi£cation. argue that the 1-norm SVM may have some advantage over the standard 2-norm SVM, especially when there are redundant noise features. We also propose an ef£cient algorithm that computes the whole solution path of the 1-norm SVM, hence facilitates adaptive selection of the tuning parameter for the 1-norm SVM.

PDF Details

NeurIPS Conference 1997 Conference Paper

Classification by Pairwise Coupling

Trevor Hastie
Robert Tibshirani

We discuss a strategy for polychotomous classification that involves estimating class probabilities for each pair of classes, and then cou(cid: 173) pling the estimates together. The coupling model is similar to the Bradley-Terry method for paired comparisons. We study the na(cid: 173) ture of the class probability estimates that arise, and examine the performance of the procedure in simulated datasets. The classifiers used include linear discriminants and nearest neighbors: applica(cid: 173) tion to support vector machines is also briefly described.

PDF Details

NeurIPS Conference 1995 Conference Paper

Discriminant Adaptive Nearest Neighbor Classification and Regression

Trevor Hastie
Robert Tibshirani

Nearest neighbor classification expects the class conditional prob(cid: 173) abilities to be locally constant, and suffers from bias in high di(cid: 173) mensions We propose a locally adaptive form of nearest neighbor classification to try to finesse this curse of dimensionality. We use a local linear discriminant analysis to estimate an effective met(cid: 173) ric for computing neighborhoods. We determine the local decision boundaries from centroid information, and then shrink neighbor(cid: 173) hoods in directions orthogonal to these local decision boundaries, and elongate them parallel to the boundaries. Thereafter, any neighborhood-based classifier can be employed, using the modified neighborhoods. We also propose a method for global dimension reduction, that combines local dimension information. We indicate how these techniques can be extended to the regression problem.

PDF Details