Distributional Bellman Operators over Mean Embeddings

We propose a novel algorithmic framework for distributional reinforcementlearning, based on learning finite-dimensional mean embeddings of returndistributions. We derive several new algorithms for dynamic programming andtemporal-difference learning based on this framework, provide asymptoticconvergence theory, and examine the empirical performance of the algorithms ona suite of tabular tasks. Further, we show that this approach can bestraightforwardly combined with deep reinforcement learning, and obtain a newdeep RL agent that improves over baseline distributional approaches on theArcade Learning Environment.