Sublinear Optimization for Machine Learning

We give sub linear-time approximation algorithms for some optimization problems arising in machine learning, such as training linear classifiers and finding minimum enclosing balls. Our algorithms can be extended to some kernelized versions of these problems, such as SVDD, hard margin SVM, and L 2 -SVM, for which sub linear-time algorithms were not known before. These new algorithms use a combination of a novel sampling techniques and a new multiplicative update algorithm. We give lower bounds which show the running times of many of our algorithms to be nearly best possible in the unit-cost RAM model. We also give implementations of our algorithms in the semi-streaming setting, obtaining the first low pass polylogarithmic space and sub linear time algorithms achieving arbitrary approximation factor.

Authors

• Kenneth L. Clarkson
• Elad Hazan
• David P. Woodruff

Keywords

• Approximation algorithms
• Approximation methods
• Vectors
• Optimization
• Support vector machines
• Machine learning algorithms
• Classification algorithms
• Machine Learning
• Lower Bound
• Support Vector Machine
• Running Time
• Linear Classifier
• Polylogarithmic
• High Probability
• Time Constant
• Kernel Function
• Dimensional Vector
• Generation Algorithm
• Hyperplane
• Hilbert Space
• Euclidean Norm
• Hash Function
• Convex Optimization Problem
• Algorithm For Problem
• Additional Error
• Generalization Error
• Las Vegas
• Online Optimization
• High Probability Of Success
• Leading Order
• Coordinate Vector
• Polynomial Kernel
• Primal-dual Algorithm
• sublinear algorithms
• classification