UAI Conference 2021 Conference Paper
Time-variant variational transfer for value functions
- Giuseppe Canonaco
- Andrea Soprani
- Matteo Giuliani
- Andrea Castelletti
- Manuel Roveri
- Marcello Restelli
In most of the transfer learning approaches to reinforcement learning (RL) the distribution over the tasks is assumed to be stationary. Therefore, the target and source tasks are i. i. d. samples of the same distribution. Unfortunately, this assumption rarely holds in real-world conditions, e. g. , due to seasonality or periodicity, evolution in the environment or faults in the sensors/actuators. In the context of this work, we consider the problem of transferring value functions through a variational method when the distribution that generates the tasks is time-variant, proposing a solution that leverages this temporal structure inherent in the task generating process. Furthermore, by means of a finite-sample analysis, the previously mentioned solution is theoretically compared to its time-invariant version. Finally, the experimental evaluation of the proposed technique is carried out on the lake Como water system representing a real-world scenario and on three different RL environments with three distinct temporal dynamics.