Author name cluster

Elena Smirnova

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

3 papers

1 author row

EWRL Workshop 2022 Workshop Paper

On Convergence of Neural asynchronous Q-iteration

Elena Smirnova

Deep Q-network algorithm is a successful and scalable algorithm on a variety of application domains. The algorithm incrementally trains a Q-network using one-step gradient descent over a mini-batch of randomly sampled transitions in a replay buffer. In this work, we formalize the Deep Q-network algorithm using an abstract algorithm, Neural asynchronous Q-iteration. We show convergence at a rate O(γ̃N ), where N is the number of iterations, γ̃ is a contraction coefficient that depends on a neural network and a rate of a mini-batch update. In particular, we show that Neural Tangent Kernel of a multi-layer ReLU neural network is a non-expansive operator and asynchronous Bellman operator is a smooth Bellman operator on average. Compared to previous work, our analysis is carried in the infinite-dimensional value function space w. r. t. kernel-based value functions. In this space, the algorithm takes a simple form, called Kernel Value Iteration.

PDF Details

AAAI Conference 2020 Conference Paper

Distributionally Robust Counterfactual Risk Minimization

Louis Faury
Ugo Tanielian
Elvis Dohmatob
Elena Smirnova
Flavian Vasile

This manuscript introduces the idea of using Distributionally Robust Optimization (DRO) for the Counterfactual Risk Minimization (CRM) problem. Tapping into a rich existing literature, we show that DRO is a principled tool for counterfactual decision making. We also show that well-established solutions to the CRM problem like sample variance penalization schemes are special instances of a more general DRO problem. In this unifying framework, a variety of distributionally robust counterfactual risk estimators can be constructed using various probability distances and divergences as uncertainty measures. We propose the use of Kullback-Leibler divergence as an alternative way to model uncertainty in CRM and derive a new robust counterfactual objective. In our experiments, we show that this approach outperforms the state-of-the-art on four benchmark datasets, validating the relevance of using other uncertainty measures in practical applications.

PDF Details

NeurIPS Conference 2020 Conference Paper

On the Convergence of Smooth Regularized Approximate Value Iteration Schemes

Elena Smirnova
Elvis Dohmatob

Entropy regularization, smoothing of Q-values and neural network function approximator are key components of the state-of-the-art reinforcement learning (RL) algorithms, such as Soft Actor-Critic~\cite{haarnoja2018soft}. Despite the widespread use, the impact of these core techniques on the convergence of RL algorithms is not yet fully understood. In this work, we analyse these techniques from error propagation perspective using the approximate dynamic programming framework. In particular, our analysis shows that (1) value smoothing results in increased stability of the algorithm in exchange for slower convergence, (2) entropy regularization reduces overestimation errors at the cost of modifying the original problem, (3) we study a combination of these techniques that describes the Soft Actor-Critic algorithm.

PDF Details