Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction

Embedding tables are usually huge in click-through rate (CTR) prediction models. To train and deploy the CTR models efficiently and economically, it is necessary to compress their embedding tables. To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed low-precision training (LPT). Also, we provide theoretical analysis on its convergence. The results show that stochastic weight quantization has a faster convergence rate and a smaller convergence error than deterministic weight quantization in LPT. Further, to reduce accuracy degradation, we propose adaptive low-precision training (ALPT) which learns the step size (i.e., the quantization resolution). Experiments on two real-world datasets confirm our analysis and show that ALPT can significantly improve the prediction accuracy, especially at extremely low bit width. For the first time in CTR models, we successfully train 8-bit embeddings without sacrificing prediction accuracy.

Authors

• Shiwei Li Huazhong University of Science and Technology
• Huifeng Guo Huawei Noah's Ark Lab
• Lu Hou Huawei Noah's Ark Lab
• Wei Zhang
• Xing Tang Huawei Noah's Ark Lab
• Ruiming Tang Huawei Noah's Ark Lab
• Rui Zhang
• Ruixuan Li Huazhong University of Science and Technology

Keywords

• DMKM: Data Compression
• DMKM: Recommender Systems