Random Barzilai–Borwein step size for mini-batch algorithms

Mini-batch algorithms, a well-studied, highly popular approach in stochastic optimization methods, are used by practitioners because of their ability to accelerate training through better use of parallel processing power and reduction of stochastic variance. However, mini-batch algorithms often employ either a diminishing step size or a tuning step size by hand, which, in practice, can be time consuming. In this paper, we propose using the improved Barzilai–Borwein (BB) method to automatically compute step sizes for the state of the art mini-batch algorithm (mini-batch semi-stochastic gradient descent (mS2GD) method), which leads to a new algorithm: mS2GD-RBB. We theoretically prove that mS2GD-RBB converges with a linear convergence rate for strongly convex objective functions. To further validate the efficacy and scalability of the improved BB method, we introduce it into another modern mini-batch algorithm, Accelerated Mini-Batch Prox SVRG (Acc-Prox-SVRG) method. In a machine learning context, numerical experiments on three benchmark data sets indicate that the proposed methods outperform some advanced stochastic optimization methods.

Authors

• Zhuang Yang
• Cheng Wang
• Zhemin Zhang School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, Sichuan, China
• Jonathan Li University of Waterloo

Keywords

• Stochastic gradient descent
• Mini batches
• Barzilai–Borwein method
• Variance reduction
• Convex optimization