Sparse supervised principal component analysis (SSPCA) for dimension reduction and variable selection

Principal component analysis (PCA) 1 1 PCA: principal component analysis, SPCA: sparse PCA, SSPCA: sparse supervised PCA, SPLS: sparse partial least squares, PMD: penalized matrix decomposition, SVD: singular value decomposition, HSIC: Hilbert Schmidt independence criterion, RKHS: reproducing kernel Hilbert space, SIMPLS: statistically inspired modification of PLS, SVM: support vector machine, CV: cross validation, RBF: radial basis function, RMSE: root mean square error, ROI: region of interest, NIR: near infrared, SSC: solvable solid content, KNN: K nearest neighbour. is one of the main unsupervised pre-processing methods for dimension reduction. When the training labels are available, it is worth using a supervised PCA strategy. In cases that both dimension reduction and variable selection are required, sparse PCA (SPCA) methods are preferred. In this paper, a sparse supervised PCA (SSPCA) method is proposed for pre-processing. This method is appropriate especially in problems where, a high dimensional input necessitates the use of a sparse method and a target label is also available to guide the variable selection strategy. Such a method is valuable in many Engineering and scientific problems, when the number of training samples is also limited. The Hilbert Schmidt Independence Criteria (HSIC) is used to form an objective based on minimization of a loss function and an L 1 norm is used for regularization of the Eigen vectors. While the proposed objective function allows a sparse low rank solution for both linear and non-linear relationships between the input and response matrices, other similar methods in this case are only based on a linear model. The objective is solved based on penalized matrix decomposition (PMD) algorithm. We compare the proposed method with PCA, PMD-based SPCA and supervised PCA. In addition, SSPCA is also compared with sparse partial least squares (SPLS), due to the similarity between the two objective functions. Experimental results from the simulated as well as real data sets show that, SSPCA provides an appropriate trade-off between accuracy and sparsity. Comparisons show that, in terms of sparsity, SSPCA performs the highest level of variable reduction and also, in terms of accuracy it is one of the most successful methods. Therefore, the Eigen vectors found by SSPCA can be used for feature selection in various high dimensional problems.

Authors

• Sara Sharifzadeh
• Ali Ghodsi University of Waterloo
• Line H. Clemmensen
• Bjarne K. Ersbøll

Keywords

• Variable selection
• Dimension reduction
• Sparse PCA
• Supervised PCA
• Sparse supervised PCA
• Penalized matrix decomposition