MLlib: Machine Learning in Apache Spark

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open- source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed. [abs] [ pdf ][ bib ] [ code ] [ webpage ] &copy JMLR 2016. ( edit, beta )

Authors

• Xiangrui Meng School of Intelligence Science and Technology, Peking University Key Laboratory of Machine Perceptron (MOE), Peking University
• Joseph Bradley
• Burak Yavuz
• Evan Sparks
• Shivaram Venkataraman
• Davies Liu
• Jeremy Freeman
• DB Tsai
• Manish Amde
• Sean Owen
• Doris Xin
• Reynold Xin
• Michael J. Franklin
• Reza Zadeh
• Matei Zaharia Stanford University
• Ameet Talwalkar Carnegie Mellon University

Keywords

No keywords are indexed for this paper.