Enable importance-aware model cacheability for inference serving

Inference serving systems are leveraged to deploy deep learning (DL) models as services. Accelerators such as Graphics Processing Units (GPUs) have been extensively used in these systems to reduce model execution time. As accelerators become more powerful and expensive, GPU sharing among DL models across different inference requests is a common practice. However, GPU memory capacity becomes a bottleneck when the number of collocated models increases, making this approach unsustainable. At the same time, collocated models may vary in popularity levels — some are accessed frequently and others are not, leading to low resource efficiency and system performance. While some existing inference serving systems offer the capability to dynamically load and cache model in memory, they are typically locality-aware and exhibit poor performance for DL inference serving. To this end, we present mCache, a novel inference serving-oriented caching system to dynamically manage a set of collocated models with diverse popularity for efficient use of memory. mCache treats each DL model as a cacheable object, and loads model on demand and unloads models when not in use. Rather than using recency or frequency, we manage models in GPU memory based on rank-based importance scores, which jointly consider cache access patterns and model-specific factors, serving as a unified metric to compare different models. During model serving, the importance scores of cached models are dynamically updated, and the least important model is evicted to make room for a newly targeted model when the cache is full. Evaluation with representative DL models shows that mCache reduces memory footprint by nearly a half with a modest inference latency increase. Compared to existing serving system using Least Frequently Used (LFU) caching algorithm, mCache improves throughput by up to 1. 5 × and 2. 39 × given the 40% and 80% GPU memory capacity.

Authors

• Hao Mo
• Didier El Baz
• Ligu Zhu
• Suping Wang
• Songfu Tan
• Hongning Zhao
• Lei Shi State Key Laboratory of Media Convergence and Communication, CUC, Beijing 100024, China State Key Laboratory of Intelligent Game, Yangtze River Delta Research Institute of NPU, Taicang 215400, China

Keywords

• Deep learning
• Inference serving
• Replacement algorithm
• Memory cache