Efficient Device-Cloud Collaborative Offline-to-Online Reinforcement Learning

Federated reinforcement learning in device-cloud architectures suffers from high interaction costs, low sample efficiency, and slow convergence due to device constraints. We propose a data-centric device-cloud collaborative training method that pre-trains a global model on the cloud to provide a high-quality initial policy and selects high-value samples to guide safe and efficient local policy finetuning on devices. Experiments on public benchmarks show our method achieves significantly faster convergence, higher training efficiency, and improved policy stability, outperforming baselines.

Authors

• Honglan Huang Rocket Force University of Engineering
• Wei Shi Rocket Force University of Engineering
• Chaoyue Niu Shanghai Jiao Tong University

Keywords

• Device-cloudcollaborative
• Federatedreinforcementlearning
• Offlineto-online reinforcement learning
• Prioritized experience replay