Hello! I am Haoran Xu. I am a 2nd-year Ph.D. student at UT Austin, advised by Prof. Amy Zhang. I used to be in AIR, Tsinghua University, JD.com and MSRA, working with Xianyuan Zhan and Zheng Yu.

My research aims towards super-human AGI using reinforcement learning, currently I am interested in:

  • (Dual-RL) Rethinking the dual formulation of reinforcement learning: is this in-sample learning paradigm all we need?
  • (GenRL) How to leverage the strong power of generative models to decision making?
  • (LLM+RL) How to enable better training and finetuning of LLMs using RL?
I am open to collaboration, feel free to reach me out!

Some links: Github / Twitter / Google Scholar / haoran.xu@utexas.edu

News

  • One paper (Diffusion-DICE) on Dual-RL and GenRL is accepted to NeurIPS 2024.
  • I will start my summer internship at MSR (New York), working with Alex Lamb and John Langford.
  • πŸ‡¦πŸ‡Ή I am attending ICLR 2024 in-person at Vienna.
  • One paper (ODICE) on Dual-RL is accepted to ICLR 2024 as spotlight.
  • πŸ‡ΊπŸ‡Έ I am attending NeurIPS 2023 in-person at New Orleans.
  • One paper (OMIGA) on offline multi-agent RL is accepted to NeurIPS 2023.
  • πŸŽ“ Starting my PhD at UT Austin.
  • πŸ‡·πŸ‡Ό I am attending ICLR 2023 in-person at Kigali.
  • 🌟 One paper (IVR) on offline RL is accepted to ICLR 2023 as oral.
  • Honored to be selected as Top Reviewers in NeurIPS 2022.
  • One paper (POR) on offline RL is accepted to NeurIPS 2022 as oral.
  • One paper (DWBC) on offline IL is accepted to ICML 2022.

Publications (* marks equal contribution)

  • ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update Liyuan Mao*, Haoran Xu*, Weinan Zhang, Xianyuan Zhan ICLR 2024 (Spotlight, Top 5%) 2024 Paper | Code | Thread
  • Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, Xianyuan Zhan ICLR 2023 (Oral, Top 5%) 2023 Paper | Code | Slide
  • A Policy-Guided Imitation Approach for Offline Reinforcement Learning Haoran Xu*, Li Jiang*, Jianxiong Li, Xianyuan Zhan NeurIPS 2022 (Oral, Top 2%) 2023 Paper | Code | Thread | Media
  • Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations Haoran Xu, Xianyuan Zhan, Honglei Yin, Huiling Qin ICML 2022 2022 Paper | Code | Thread
  • Constraints Penalized Q-Learning for Safe Offline Reinforcement Learning Haoran Xu, Xianyuan Zhan, Xiangyu Zhu AAAI 2022 (Spotlight @ ICML 2021 RL4RealLife workshop) 2022 Paper | Code | Slides
  • DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning Xianyuan Zhan*, Haoran Xu*, Yue Zhang, Yusen Huo, Xiangyu Zhu, Honglei Yin, Yu Zheng AAAI 2022 (Spotlight @ ICML 2021 RL4RealLife workshop) 2022 Paper | Code
  • Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning Liyuan Mao*, Haoran Xu*, Weinan Zhang, Xianyuan Zhan, Amy Zhang NeurIPS 2024 2024 Paper | Code | Website |
  • ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update Liyuan Mao*, Haoran Xu*, Weinan Zhang, Xianyuan Zhan ICLR 2024 (Spotlight, Top 5%) 2024 Paper | Code | Thread
  • Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, Xianyuan Zhan ICLR 2023 (Oral, Top 5%) 2023 Paper | Code | Slide
  • A Policy-Guided Imitation Approach for Offline Reinforcement Learning Haoran Xu*, Li Jiang*, Jianxiong Li, Xianyuan Zhan NeurIPS 2022 (Oral, Top 2%) 2023 Paper | Code | Thread | Media
  • Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning Liyuan Mao*, Haoran Xu*, Weinan Zhang, Xianyuan Zhan, Amy Zhang NeurIPS 2024 2024 Paper | Code | Website |
  • PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Ya-Qin Zhang Preprint 2024 Paper | Code
  • ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update Liyuan Mao*, Haoran Xu*, Weinan Zhang, Xianyuan Zhan ICLR 2024 (Spotlight, Top 5%) 2024 Paper | Code | Thread
  • Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization Xiangsen Wang, Haoran Xu, Yinan Zheng, Xianyuan Zhan NeurIPS 2023 2023 Paper | Code
  • SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning Qin Zhang*, Linrui Zhang*, Haoran Xu, Li Shen, Bowen Wang, Xueqian Wang, Bo Yuan, Yongzhe Chang, Dacheng Tao ICLR 2023 SR4AD Workshop 2023 Paper
  • Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, Xianyuan Zhan ICLR 2023 (Notable Top 5%) 2023 Paper | Code
  • Mind the Gap: Offline Policy Optimizaiton for Imperfect Rewards Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, Ya-Qin Zhang ICLR 2023 2023 Paper | Code
  • When data geometry meets deep function: Generalizing offline reinforcement learning Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, Ya-Qin Zhang ICLR 2023 2023 Paper | Code
  • A Policy-Guided Imitation Approach for Offline Reinforcement Learning Haoran Xu*, Li Jiang*, Jianxiong Li, Xianyuan Zhan NeurIPS 2022 (Oral, Top 2%) 2022 Paper | Code | Slides | Media
  • Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations Haoran Xu, Xianyuan Zhan, Honglei Yin, Huiling Qin ICML 2022 2022 Paper | Code | Slides
  • Constraints Penalized Q-Learning for Safe Offline Reinforcement Learning Haoran Xu, Xianyuan Zhan, Xiangyu Zhu AAAI 2022 (Spotlight @ ICML 2021 RL4RealLife workshop) 2022 Paper | Code | Slides
  • DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning Xianyuan Zhan*, Haoran Xu*, Yue Zhang, Yusen Huo, Xiangyu Zhu, Honglei Yin, Yu Zheng AAAI 2022 (Spotlight @ ICML 2021 RL4RealLife workshop) 2022 Paper | Code
  • Discriminator-Guided Model-Based Offline Imitation Learning Wenjia Zhang, Haoran Xu, Haoyi Niu, Peng Cheng, Ming Li, Heming Zhang, Guyue Zhou, Xianyuan Zhan CoRL 2022 2022 Paper
  • Model-Based Offline Planning with Trajectory Pruning Xianyuan Zhan, Xiangyu Zhu, Haoran Xu IJCAI 2022 2022 Paper | Code
  • ECoalVis: Visual Analysis of Control Strategies in Coal-fired Power Plants Shuhan Liu, Di Weng, Yuan Tian, Zikun Deng, Haoran Xu, Xiangyu Zhu, Honglei Yin, Xianyuan Zhan, Yingcai Wu IEEE VIS 2022 2022 Paper | Code
  • Multi-Memory enhanced Separation Network for Indoor Temperature Prediction Zhewen Duan, Xiuwen Yi, Peng Li, Dekang Qi, Yexin Li, Haoran Xu, Yanyong Huang, Junbo Zhang, Yu Zheng DASFAA 2022 2022 Paper
  • Offline Reinforcement Learning with Soft Behavioral Regularization Haoran Xu, Xianyuan Zhan, Jianxiong Li, Honglei Yin NeurIPS 2021 Offline RL Workshop 2021 Paper | Code
  • Robust Spatio-Temporal Purchase Prediction via Deep Meta Learning Huiling Qin, Songyu Ke, Xiaodu Yang, Haoran Xu, Xianyuan Zhan, Yu Zheng AAAI 2021 2021 Paper

Professional Services

Reviewer for ICLR, ICML, NeurIPS