Papers Of Distributional RL

Related papers for Distributional Reinforcement Learning (DistRL). Since there are tons of new papers on distributional RL with various applications in each conference, we are only able to update those we just read and consider as insightful in our subjective opinion. Please feel free to let us know if you feel we have missed some important papers; we would appreciate your kindness.

Contact : Ke Sun, ksun6@ualberta.ca

Onging

2025

• Limit Theorems for Entropy-Regularized and Distributional Reinforcement Learning (NeurIPS 2025)

• The Benefits of Being Categorical Distribution: Uncertainty-Aware Entropy Regularized Exploration in Reinforcement Learning (NeurIPS 2025)

• Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation (ICML 2025)

• Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning (ICML 2025)

• Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025)

• AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning (ICML 2025)

2024

• Foundations of Multivariate Distributional Reinforcement Learning (NeurIPS 2024)

• Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024)

• Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning (NeurIPS 2024)

• Distributional Reinforcement Learning via Regularized Wasserstein Loss (NeurIPS 2024)

• A Distributional Analogue to the Successor Representation (ICML 2024)

• More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning (ICML 2024)

• Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024)

• Distributional Bellman Operators over Mean Embeddings (ICML 2024)

• Diverse Projection Ensembles for Distributional Reinforcement Learning (ICLR 2024)

• How Does Return Distribution in Distributional Reinforcement Learning Help Optimization (ICML2024 workshop)

2023

• An Analysis of Quantile Temporal-Difference Learning (JMLR)

This paper gives a comprehensive analysis of quantile TD, with a particular focus on the convergence of sample-based quantile TD by leveraging the stochastic approximation techniques instead of the already existing contraction analysis in the dynamic programming scenario.

• Distributional Reinforcement Learning (Book, MIT Press)

• The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation (ICML 2023)

The authors argue that quantile TD is also fundamental akin to the clascial TD, as it can offer better value estimation even without directly do the return distribution learning. Specifically, the analysis is mainly in the tabular setting.

• Distributional Offline Policy Evaluation with Predictive Error Guarantees (ICML 2023)

This paper extends the fitted Q evaluation to its distributional version. Under MLE estimation with a probabilistic model, e.g., generative models, some prediction guarantees are provided by considering the TV and Wasserstein distance. Note that the prediction guarantee is based on the small in-distribution generalization error and the analysis is not directly related to practical distributional RL algorithms. Some underlying connection with categorical distributional RL exists as MLE is equivalent to KL divergence, but the authors did not clearly state that.

• Variance Control for Distributional Reinforcement Learning (ICML 2023)

• Adversarial Learning of Distributional Reinforcement Learning (ICML 2023)

• The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning (NeurIPS 2023)

• Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion (NeurIPS 2023)

• Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observations (ECML-PKDD 2023)

2022

• The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning (NeurIPS 2022)

• Distributional Reinforcement Learning for Risk-Sensitive Policies (NeurIPS 2022)

2021

• Conservative Offline Distributional Reinforcement Learning (NeurIPS 2021)

• Distributional Reinforcement Learning for Multi-Dimensional Reward Functions (NeurIPS 2021)

• Distributional Reinforcement Learning via Moment Matching (AAAI 2021)

• Bayesian Distributional Policy Gradients (AAAI 2021)

• Dynamic Programming in Distributional Reinforcement Learning (Research Report)

2020

• Non-crossing quantile regression for deep reinforcement learning (NeurIPS 2020)

2019

• Fully Parameterized Quantile Function for Distributional Reinforcement Learning (NeurIPS 2019)

• Statistics and Samples in Distributional Reinforcement Learning (ICML 2019)

• Distributional Reinforcement Learning for Efficient Exploration (ICML 2019)

• QUOTA: The Quantile Option Architecture for Reinforcement Learning (AAAI 2019)

• A Comparative Analysis of Expected and Distributional Reinforcement Learning (AAAI 2019)

• Distributional Reinforcement Learning with Linear Function approximation (AISTATS 2019)

2018

• Distributional Reinforcement Learning with Quantile Regression (AAAI 2018)

• Implicit Quantile Networks for Distributional Reinforcement Learning (ICML 2018)

• An Analysis of Categorical Distributional Reinforcement Learning (AISTATS 2018)

• Distributed Distributional Deterministic Policy Gradients (ICLR 2018)

2017

• A Distributional Perspective on Reinforcement Learning (ICML 2017)

• The Cramer Distance as a Solution to Biased Wasserstein Gradients (Research Report)