A Distributional Analogue to the Successor Representation

This paper contributes a new approach for distributional reinforcementlearning which elucidates a clean separation of transition structure and rewardin the learning process. Analogous to how the successor representation (SR)describes the expected consequences of behaving according to a given policy,our distributional successor measure (SM) describes the distributionalconsequences of this behaviour. We formulate the distributional SM as adistribution over distributions and provide theory connecting it withdistributional and model-based reinforcement learning. Moreover, we propose analgorithm that learns the distributional SM from data by minimizing a two-levelmaximum mean discrepancy. Key to our method are a number of algorithmictechniques that are independently valuable for learning generative models ofstate. As an illustration of the usefulness of the distributional SM, we showthat it enables zero-shot risk-sensitive policy evaluation in a way that wasnot previously possible.