Off-policy Distributional Q(λ): Distributional RL Without Importance Sampling

We introduce off-policy distributional Q(λ), a new addition to thefamily of off-policy distributional evaluation algorithms. Off-policydistributional Q(λ) does not apply importance sampling for off-policylearning, which introduces intriguing interactions with signed measures. Suchunique properties distributional Q(λ) from other existing alternativessuch as distributional Retrace. We characterize the algorithmic properties ofdistributional Q(λ) and validate theoretical insights with tabularexperiments. We show how distributional Q(λ)-C51, a combination ofQ(λ) with the C51 agent, exhibits promising results on deep RLbenchmarks.