Awesome Decision Transformer

This is a collection of research papers for Decision Transformer (DT). And the repository will be continuously updated to track the frontier of DT.

Welcome to follow and star!

Table of Contents

• A Taxonomy of DT Algorithms

• Papers

  • Arxiv
  • NeurIPS 2023 (New!!!)
  • CoRL 2023
  • ICLR 2023
  • NeurIPS 2022
  • CoRL 2022
  • ICML 2022
  • AAAI 2022
  • ICLR 2022
  • NeurIPS 2021
  • ICML 2021

• Contributing

Overview of Transformer

The Decision Transformer was proposed by “Decision Transformer: Reinforcement Learning via Sequence Modeling” by Chen L. et al. It casts (offline) Reinforcement Learning as a conditional-sequence modeling problem.

Specifically, DT model is a causal transformer model conditioned on the desired return, (past) states, and actions to generate future actions in an autoregressive manner.

Advantage

1. Bypass the need for bootstrapping for long term credit assignment
2. Avoid undesirable short-sighted behaviors due to the discounting future rewards.
3. Enjoy the transformer models widely used in language and vision, which are easy to scale and adapt to multi-modal data.

Papers

format:
- [title](paper link) [links]
  - author1, author2, and author3...
  - publisher
  - key 
  - code 
  - experiment environment

Arxiv

• Context-Former: Stitching via Latent Conditioned Sequence Modeling

  • Ziqi Zhang, Jingzehua Xu, Zifeng Zhuang, Jinxin Liu, Donglin wang
  • Key: DT, Latent Conditioned Sequence Modeling
  • ExpEnv: D4RL

• Real-time Network Intrusion Detection via Decision Transformers

  • Jingdi Chen, Hanhan Zhou, Yongsheng Mei, Gina Adam, Nathaniel D. Bastian, Tian Lan
  • Key: DT, Network Intrusion Detection
  • ExpEnv: UNSW-NB15

• Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning

  • Sabrina McCallum, Max Taylor-Davies, Stefano V. Albrecht, Alessandro Suglia
  • Key: DT, language feedback
  • ExpEnv: BabyAI

• Rethinking Decision Transformer via Hierarchical Reinforcement Learning

  • Yi Ma, Chenjun Xiao, Hebin Liang, Jianye Hao
  • Key: DT, Hierarchical Reinforcement Learning
  • ExpEnv: D4RL

• Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making

  • Jeonghye Kim, Suyoung Lee, Woojun Kim, Youngchul Sung
  • Key: MetaFormer, Decision ConvFormer
  • ExpEnv: D4RL, Atari

• Prompt-Tuning Decision Transformer with Preference Ranking

  • Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
  • Key: Prompt-Tuning
  • ExpEnv: D4RL

• Graph Decision Transformer

  • Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
  • Key: graph transformer
  • ExpEnv: Atari

• A Survey on Transformers in Reinforcement Learning

  • Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, Deheng Ye
  • Key: survey

• Can Offline Reinforcement Learning Help Natural Language Understanding?

  • Ziqi Zhang, Yile Wang, Yue Zhang, Donglin Wang
  • Key: Language model
  • ExpEnv: MuJoco, Maze 2D

• Hierarchical Decision Transformer

  • André Correia, Luís A. Alexandre
  • Key: Hierarchical Learning, Imitation Learning
  • ExpEnv: MuJoco, D4RL, RoboMimic, Maze 2D

• PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training

  • Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, Ashish Kapoor
  • Key: Robotics, Pretrain, Multitask, Representation
  • ExpEnv: MuSHR car, Habitat

• LATTE: LAnguage Trajectory TransformEr

  • Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, Rogerio Bonatti
  • Key: MultiModal, Robotics
  • Code: official, official
  • ExpEnv: CoppeliaSim

• Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL

  • Taku Yamagata, Ahmed Khalil, Raul Santos-Rodriguez
  • Key: Q-Learning
  • ExpEnv: D4RL

• Multi-Game Decision Transformers

  • Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, Igor Mordatch
  • Key: Multi-Task, Finetuning
  • Code: official
  • ExpEnv: Atari, REM

• Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

  • Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Bo Xu
  • Key: Multi-Agent RL
  • Code: official
  • ExpEnv: SMAC

• Transfer learning with causal counterfactual reasoning in Decision Transformers

  • Ayman Boustati, Hana Chockler, Daniel C. McNamee
  • Key: Causal reasoning, Transfer Learning
  • ExpEnv: MINIGRID

• Pretraining for Language Conditioned Imitation with Transformers

  • Aaron L Putterman, Kevin Lu, Igor Mordatch, Pieter Abbeel
  • Key: Text-Conditioned Decision
  • ExpEnv: Text-Conditioned Frostbite (MultiModal Benchmark)

• An Offline Deep Reinforcement Learning for Maintenance Decision-Making

  • Hamed Khorasgani, Haiyan Wang, Chetan Gupta, Ahmed Farahat
  • Publisher: Annual Conference of the PHM Society 2021
  • Key: Offline Supervised RL, Remaining Useful Life Estimation
  • ExpEnv: NASA C-MAPSS

• A Sequence Modelling Approach to Question Answering in Text-Based Games

  • Gregory Furman, Edan Toledo, Jonathan Shock, Jan Buys
  • Publisher: Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022)
  • Key: VQA
  • ExpEnv: QAIT

• Can Wikipedia Help Offline Reinforcement Learning?

  • Machel Reid, Yutaro Yamada, Shixiang Shane Gu
  • Key: VLN, Transfer Learning
  • Code: official
  • ExpEnv: MuJoco, D4RL, Atari

• Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learning

  • Qinjie Lin, Han Liu, Biswa Sengupta
  • Key: Multi-Task RL, Sparse Reward
  • ExpEnv: MINIGRID

• Bootstrapped Transformer for Offline Reinforcement Learning

  • Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, Dongsheng Li
  • Key: Generation model
  • Code: official
  • ExpEnv: MuJoco, D4RL, Adroit

• Deep Transformer Q-Networks for Partially Observable Reinforcement Learning

  • Kevin Esslinger, Robert Platt, Christopher Amato
  • Key: POMDP, Transformer Q-Learning
  • ExpEnv: GV, Car Flag

• Multi-Agent Reinforcement Learning is a Sequence Modeling Problem

  • Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, Yaodong Yang
  • Key: Multi-Agent RL
  • ExpEnv: SMAC, MA MuJoco

• Transformers are Adaptable Task Planners

  • Vidhi Jain, Yixin Lin, Eric Undersander, Yonatan Bisk, Akshara Rai
  • Key: Task Planning, Prompt, Control, Generalization
  • Code: official
  • ExpEnv: Dishwasher Loading

• You Can't Count on Luck: Why Decision Transformers Fail in Stochastic Environments

  • Keiran Paster, Sheila McIlraith, Jimmy Ba
  • Key: Stochastic Environments
  • ExpEnv: Gambling, Connect Four, 2048

• When does return-conditioned supervised learning work for offline reinforcement learning?

  • David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, Joan Bruna
  • Key: Theoretical analysis
  • ExpEnv: MuJoco, D4RL

• SimStu-Transformer: A Transformer-Based Approach to Simulating Student Behaviour

  • Zhaoxing Li, Lei Shi, Alexandra Cristea, Yunzhan Zhou, Chenghao Xiao, Ziqi Pan
  • Key: Intelligent Tutoring System

• Attention-Based Learning for Combinatorial Optimization

  • Carson Smith
  • Key: Combinatorial Optimization

NeurIPS 2023

• Elastic Decision Transformer

  • Yueh-Hua Wu, Xiaolong Wang, Masashi Hamaya
  • Publisher: NeurIPS 2023
  • Key: Offline RL, stitch trajectory, Multi-Task
  • ExpEnv: D4RL

• Learning to Modulate pre-trained Models in RL

  • Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, Sepp Hochreiter
  • Publisher: NeurIPS 2023 (Poster)
  • Key: reinforcement learning, multi-task learning, continual learning, fine-tuning
  • ExpEnv: MetaWorld, DMControl

CoRL 2023

• Q-Transformer
  • Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
  • Publisher: CoRL 2023
  • Key: Reinforcement Learning, Offline RL, Transformers, Q-Learning, Robotic Manipulation
  • Code: Unofficial
  • ExpEnv: None

ICLR 2023

• EDGI: Equivariant Diffusion for Planning with Embodied Agents

  • Johann Brehmer, Joey Bose, Pim de Haan, Taco Cohen
  • Publisher: ICLR 2023 Reincarnating RL workshop
  • Key: rich geometric structure, equivariant, conditional generative modeling, representation
  • ExpEnv: None

• Learning to Modulate pre-trained Models in RL

  • Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, Sepp Hochreiter
  • Publisher: ICLR 2023 Reincarnating RL workshop
  • Key: reinforcement learning, multi-task learning, continual learning, fine-tuning
  • ExpEnv: MetaWorld, DMControl

• DeFog: Decision Transformer under Random Frame Dropping

  • Kaizhe Hu*, Ray Chen Zheng*, Yang Gao, Huazhe Xu
  • Publisher: ICLR 2023
  • Key: Offline RL, POMDP, Frame-Dropping, Practical Application
  • Code: official
  • ExpEnv: MuJoco, D4RL, Atari

NeurIPS 2022

• Decision making as language generation
  • Roland Memisevic, Sunny Panchal, Mingu Lee
  • Publisher: NeurIPS 2022 Workshop FMDM
  • Key: Generation
  • ExpEnv: Traversals (Toy experiment)

CoRL 2022

• Offline Reinforcement Learning for Customizable Visual Navigation

  • Dhruv Shah, Arjun Bhorkar, Hrishit Leen, Ilya Kostrikov, Nicholas Rhinehart, Sergey Levine
  • Publisher: CoRL 2022 (Oral)
  • Key: Visual Navigation
  • ExpEnv: RECON

• Instruction-driven history-aware policies for robotic manipulations

  • Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia, Makarand Tapaswi, Ivan Laptev, Cordelia Schmid
  • Publisher: CoRL 2022 (Oral)
  • Key: Robotics, Language Instruction
  • Code: official
  • ExpEnv: RLBench

• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

  • Mohit Shridhar, Lucas Manuelli, Dieter Fox
  • Publisher: CoRL 2022
  • Key: Robotics, Language Grounding, Behavior Cloning
  • Code: official
  • ExpEnv: RLBench

ICML 2022

• Online Decision Transformer

  • Qinqing Zheng, Amy Zhang, Aditya Grover
  • Publisher: ICML 2022 (Oral)
  • Key: Online finetuning, Max-entropy, Exploration
  • Code: unofficial
  • ExpEnv: MuJoco, D4RL

• Prompting Decision Transformer for Few-Shot Policy Generalization

  • Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, Chuang Gan
  • Publisher: ICML 2022 (Poster)
  • Key: Prompt, Few-shot, Generalization
  • Code: official (released soon)
  • ExpEnv: DMC

• Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning

  • Adam R Villaflor, Zhe Huang, Swapnil Pande, John M Dolan, Jeff Schneider
  • Publisher: ICML 2022 (Poster)
  • Key: World model
  • Code: official (released soon)
  • ExpEnv: CARLA

• AnyMorph: Learning Transferable Polices By Inferring Agent Morphology

  • Brandon Trabucco, Mariano Phielipp, Glen Berseth
  • Publisher: ICML 2022 (Poster)
  • Key: Morphology, Transfer Learning, Zero Shot
  • ExpEnv: Modular-RL

AAAI 2022

• Dreaming with Transformers
  • Catherine Zeng, Jordan Docter, Alexander Amini, Igor Gilitschenski, Ramin Hasani, Daniela Rus
  • Publisher: AAAI 2022 (RLG Workshop)
  • Key: Dreamer, World Model
  • ExpEnv: Deepmind Lab, VISTA

ICLR 2022

• Learning Transferable Policies By Inferring Agent Morphology

  • Brandon Trabucco, Mariano Phielipp, Glen Berseth
  • Publisher: ICLR 2022 (GPL Workshop Poster)
  • Key: Morphology, Transfer Learning, Zero Shot
  • ExpEnv: Modular-RL

• Silver-Bullet-3D at ManiSkill 2021: Learning-from-Demonstrations and Heuristic Rule-based Methods for Object Manipulation

  • Yingwei Pan, Yehao Li, Yiheng Zhang, Qi Cai, Fuchen Long, Zhaofan Qiu, Ting Yao, Tao Mei
  • Publisher: ICLR 2022 (GPL Workshop Poster)
  • Key: Object Manipulation
  • Code: official
  • ExpEnv: ManiSkill

• Generalized Decision Transformer for Offline Hindsight Information Matching

  • Hiroki Furuta, Yutaka Matsuo, Shixiang Shane Gu
  • Publisher: ICLR 2021 (Spotlight)
  • Key: HIM, SMM
  • Code: official
  • ExpEnv: MuJoco, D4RL

NeurIPS 2021

• Decision Transformer: Reinforcement Learning via Sequence Modeling

  • Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch
  • Publisher: NeurIPS 2021 (Poster)
  • Key: Conditional sequence modeling
  • Code: official, DI-engine
  • ExpEnv: MuJoco, D4RL, Atari

• Offline Reinforcement Learning as One Big Sequence Modeling Problem

  • Michael Janner, Qiyang Li, Sergey Levine
  • Publisher: NeurIPS 2021 (Spotlight)
  • Key: Conditional sequence modeling, Discretization
  • Code: official
  • ExpEnv: MuJoco, D4RL

• TransDreamer: Reinforcement Learning with Transformer World Models

  • Chang Chen, Yi-Fu Wu, Jaesik Yoon, Sungjin Ahn
  • Publisher: NeurIPS 2021 (Deep RL Workshop)
  • Key: Dreamer, World Model
  • ExpEnv: Hidden Order Discovery, DMC, Atari

ICML 2021

• Reinforcement learning as one big sequence modeling problem
  • Michael Janner, Qiyang Li, Sergey Levine
  • Publisher: ICML workshop
  • Key: Conditional sequence modeling, Discretization
  • Code: official
  • ExpEnv: MuJoco, D4RL

Contributing

Our purpose is to make this repo even better. If you are interested in contributing, please refer to HERE for instructions in contribution.

License

Awesome Decision Transformer is released under the Apache 2.0 license.