This is a collection of research papers for Decision Transformer (DT). And the repository will be continuously updated to track the frontier of DT.
Welcome to follow and star!
The Decision Transformer was proposed by “Decision Transformer: Reinforcement Learning via Sequence Modeling” by Chen L. et al. It casts (offline) Reinforcement Learning as a conditional-sequence modeling problem.
Specifically, DT model is a causal transformer model conditioned on the desired return, (past) states, and actions to generate future actions in an autoregressive manner.
- Bypass the need for bootstrapping for long term credit assignment
- Avoid undesirable short-sighted behaviors due to the discounting future rewards.
- Enjoy the transformer models widely used in language and vision, which are easy to scale and adapt to multi-modal data.
-
On Transforming Reinforcement Learning With Transformers: The Development Trajectory
- Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, Dacheng Tao
- Publisher: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
-
Large sequence models for sequential decision-making: a survey
- Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo Mai, Jun Wang, Haifeng Zhang, Weinan Zhang
- Publisher: Frontiers of Computer Science
-
A Survey on Transformers in Reinforcement Learning
- Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, Deheng Ye
- Publisher: Transactions on Machine Learning Research (TMLR)
-
Transformers in Reinforcement Learning: A Survey
- Pranav Agarwal, Aamer Abdul Rahman, Pierre-Luc St-Charles, Simon J.D. Prince, Samira Ebrahimi Kahou
format:
- [title](paper link) [links]
- author1, author2, and author3...
- publisher
- key
- code
- experiment environment
-
Decision MetaMamba: Integrating Multi-Modal Input Token Mixer into Mamba-Based Decision Models
- Wall Kim
- Key: SSM, Mamba, Sequential Decision Making
- ExpEnv: D4RL
-
Context-Former: Stitching via Latent Conditioned Sequence Modeling
- Ziqi Zhang, Jingzehua Xu, Zifeng Zhuang, Jinxin Liu, Donglin wang
- Key: DT, Latent Conditioned Sequence Modeling
- ExpEnv: D4RL
-
Real-time Network Intrusion Detection via Decision Transformers
- Jingdi Chen, Hanhan Zhou, Yongsheng Mei, Gina Adam, Nathaniel D. Bastian, Tian Lan
- Key: DT, Network Intrusion Detection
- ExpEnv: UNSW-NB15
-
Prompt-Tuning Decision Transformer with Preference Ranking
- Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
- Key: Prompt-Tuning
- ExpEnv: D4RL
-
- Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
- Key: graph transformer
- ExpEnv: Atari
-
Can Offline Reinforcement Learning Help Natural Language Understanding?
- Ziqi Zhang, Yile Wang, Yue Zhang, Donglin Wang
- Key: Language model
- ExpEnv: MuJoco, Maze 2D
-
SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
- Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, Dacheng Tao
- Key: Offline Safe RL, DT
- ExpEnv: D4RL
-
Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
-
Transfer learning with causal counterfactual reasoning in Decision Transformers
- Ayman Boustati, Hana Chockler, Daniel C. McNamee
- Key: Causal reasoning, Transfer Learning
- ExpEnv: MINIGRID
-
Pretraining for Language Conditioned Imitation with Transformers
- Aaron L Putterman, Kevin Lu, Igor Mordatch, Pieter Abbeel
- Key: Text-Conditioned Decision
- ExpEnv: Text-Conditioned Frostbite (MultiModal Benchmark)
-
An Offline Deep Reinforcement Learning for Maintenance Decision-Making
- Hamed Khorasgani, Haiyan Wang, Chetan Gupta, Ahmed Farahat
- Publisher: Annual Conference of the PHM Society 2021
- Key: Offline Supervised RL, Remaining Useful Life Estimation
- ExpEnv: NASA C-MAPSS
-
A Sequence Modelling Approach to Question Answering in Text-Based Games
- Gregory Furman, Edan Toledo, Jonathan Shock, Jan Buys
- Publisher: Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022)
- Key: VQA
- ExpEnv: QAIT
-
- Qinjie Lin, Han Liu, Biswa Sengupta
- Key: Multi-Task RL, Sparse Reward
- ExpEnv: MINIGRID
-
Deep Transformer Q-Networks for Partially Observable Reinforcement Learning
-
SimStu-Transformer: A Transformer-Based Approach to Simulating Student Behaviour
- Zhaoxing Li, Lei Shi, Alexandra Cristea, Yunzhan Zhou, Chenghao Xiao, Ziqi Pan
- Key: Intelligent Tutoring System
-
Attention-Based Learning for Combinatorial Optimization
- Carson Smith
- Key: Combinatorial Optimization
- Decomposed Prompt Decision Transformer for Efficient Unseen Task Generalization
- Hongling Zheng, Li Shen, Yong Luo, Tongliang Liu, Jialie Shen, Dacheng Tao
- Publisher: NeurIPS 2024
- Key: Offline Reinforcement Learning, Prompt Tuning
- ExpEnv: MuJoco
-
HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
- Shengchao Hu, Ziqing Fan, Li Shen, Ya Zhang, Yanfeng Wang, Dacheng Tao
- Publisher: ICML 2024
- Key: Multi-task, DT
- ExpEnv: MetaWorld
-
Q-value Regularized Transformer for Offline Reinforcement Learning
- Shengchao Hu, Ziqing Fan, Chaoqin Huang, Li Shen, Ya Zhang, Yanfeng Wang, Dacheng Tao
- Publisher: ICML 2024
- Key: Q-learning, DT
- ExpEnv: D4RL
-
- Zijian Guo, Weichao Zhou, Wenchao Li
- Publisher: ICML 2024
- Key: Signal Temporal Logic (STL), DT
- ExpEnv: DSRL
-
Think Before You Act: Decision Transformers with Working Memory
-
In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
- Sili Huang, Jifeng Hu, Hechang Chen, Lichao Sun, Bo Yang
- Publisher: ICML 2024
- Key: Hierarchical Structure, DT
- ExpEnv: D4RL
-
Rethinking Decision Transformer via Hierarchical Reinforcement Learning
- Yi Ma, Jianye Hao, Hebin Liang, Chenjun Xiao,
- Publisher: ICML 2024
- Key: DT, Hierarchical Reinforcement Learning
- ExpEnv: D4RL
-
Learning Multi-Agent Communication from Graph Modeling Perspective
- Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
- Publisher: ICLR 2024
- Key: Communication, Sequence Modeling
- ExpEnv: SMAC
-
Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
-
When should we prefer Decision Transformers for Offline Reinforcement Learning?
- Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, Amy Zhang
- Key: offline reinforcement learning, sequence modeling, reinforcement learning
- ExpEnv: D4RL
-
- Licong Lin, Yu Bai, Song Mei
- Key: transformers, in-context learning, reinforcement learning, learning theory
- ExpEnv: stochastic linear bandit
-
Searching for High-Value Molecules Using Reinforcement Learning and Transformers
- Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phielipp, Glen Berseth
- Key: chemistry, reinforcement learning, language models
- ExpEnv: docking and pytdc tasks
-
SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks
- Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, Xiang Ren
- Publisher: NeurIPS 2023
- Key: Dual-Process Theory
- ExpEnv: ScienceWorld
-
HIQL: Offline Goal-Conditioned RL with Latent States as Actions
- Seohong Park, Dibya Ghosh, Benjamin Eysenbach, Sergey Levine
- Publisher: NeurIPS 2023
- Key: Hierarchical Goal-Conditioned RL, Offline Reinforcement Learning, Value Function Estimation
- ExpEnv: D4RL
-
Structured state space models for in-context reinforcement learning
- Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, Feryal Behbahani
- Publisher: NeurIPS 2023
- Key: state space model, in-context RL
- ExpEnv: t-maze
-
Supervised Pretraining Can Learn In-Context Reinforcement Learning
- Jonathan N. Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, Emma Brunskill
- Publisher: NeurIPS 2023
- Key: Decision Pretrained Transformer, in-context learning
- ExpEnv: Dark Room
-
- Sabrina McCallum, Max Taylor-Davies, Stefano V. Albrecht, Alessandro Suglia
- Publisher: NeurIPS 2023 Workshop
- Key: DT, language feedback
- ExpEnv: BabyAI
-
STEVE-1: A Generative Model for Text-to-Behavior in Minecraft
- Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, Sheila McIlraith
- Publisher: NeurIPS 2023
- Key: instruction-tuned Video Pretraining
- ExpEnv: Minecraft
-
Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
- Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, Song Mei
- Publisher: NeurIPS 2023
- Key: in-context learning, transformers, deep learning theory, learning theory
- ExpEnv: in-context regression problems
-
- Yueh-Hua Wu, Xiaolong Wang, Masashi Hamaya
- Publisher: NeurIPS 2023
- Key: Offline RL, stitch trajectory, Multi-Task
- ExpEnv: D4RL
-
Transformers are Adaptable Task Planners
- Vidhi Jain, Yixin Lin, Eric Undersander, Yonatan Bisk, Akshara Rai
- Publisher: CoRL 2023
- Key: Task Planning, Prompt, Control, Generalization
- Code: official
- ExpEnv: Dishwasher Loading
-
- Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
- Publisher: CoRL 2023
- Key: Reinforcement Learning, Offline RL, Transformers, Q-Learning, Robotic Manipulation
- Code: Unofficial
- ExpEnv: None
-
PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
- Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, Ashish Kapoor
- Publisher: IROS 2023
- Key: Robotics, Pretrain, Multitask, Representation
- ExpEnv: MuSHR car, Habitat
-
Constrained Decision Transformer for Offline Safe Reinforcement Learning
- Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, Ding Zhao
- Publisher: ICML 2023
- Key: Offline Safe RL, DT
- ExpEnv: Bullet-Safety-Gym
-
- Taku Yamagata, Ahmed Khalil, Raul Santos-Rodriguez
- Publisher: ICML 2023
- Key: Q-Learning
- ExpEnv: D4RL
- LATTE: LAnguage Trajectory TransformEr
- Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, Rogerio Bonatti
- Publisher: ICRA 2023
- Key: MultiModal, Robotics
- Code: official, official
- ExpEnv: CoppeliaSim
-
EDGI: Equivariant Diffusion for Planning with Embodied Agents
- Johann Brehmer, Joey Bose, Pim de Haan, Taco Cohen
- Publisher: ICLR 2023 Reincarnating RL workshop
- Key: rich geometric structure, equivariant, conditional generative modeling, representation
- ExpEnv: None
-
When does return-conditioned supervised learning work for offline reinforcement learning?
-
You Can't Count on Luck: Why Decision Transformers Fail in Stochastic Environments
- Keiran Paster, Sheila McIlraith, Jimmy Ba
- Publisher: NeurIPS 2022
- Key: Stochastic Environments
- ExpEnv: Gambling, Connect Four, 2048
-
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
-
Decision making as language generation
- Roland Memisevic, Sunny Panchal, Mingu Lee
- Publisher: NeurIPS 2022 Workshop FMDM
- Key: Generation
- ExpEnv: Traversals (Toy experiment)
-
Offline Reinforcement Learning for Customizable Visual Navigation
- Dhruv Shah, Arjun Bhorkar, Hrishit Leen, Ilya Kostrikov, Nicholas Rhinehart, Sergey Levine
- Publisher: CoRL 2022 (Oral)
- Key: Visual Navigation
- ExpEnv: RECON
-
Instruction-driven history-aware policies for robotic manipulations
-
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
-
- Qinqing Zheng, Amy Zhang, Aditya Grover
- Publisher: ICML 2022 (Oral)
- Key: Online finetuning, Max-entropy, Exploration
- Code: unofficial
- ExpEnv: MuJoco, D4RL
-
Prompting Decision Transformer for Few-Shot Policy Generalization
-
Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning
-
AnyMorph: Learning Transferable Polices By Inferring Agent Morphology
- Brandon Trabucco, Mariano Phielipp, Glen Berseth
- Publisher: ICML 2022 (Poster)
- Key: Morphology, Transfer Learning, Zero Shot
- ExpEnv: Modular-RL
- Dreaming with Transformers
- Catherine Zeng, Jordan Docter, Alexander Amini, Igor Gilitschenski, Ramin Hasani, Daniela Rus
- Publisher: AAAI 2022 (RLG Workshop)
- Key: Dreamer, World Model
- ExpEnv: Deepmind Lab, VISTA
-
Learning Transferable Policies By Inferring Agent Morphology
- Brandon Trabucco, Mariano Phielipp, Glen Berseth
- Publisher: ICLR 2022 (GPL Workshop Poster)
- Key: Morphology, Transfer Learning, Zero Shot
- ExpEnv: Modular-RL
-
Generalized Decision Transformer for Offline Hindsight Information Matching
-
Decision Transformer: Reinforcement Learning via Sequence Modeling
-
Offline Reinforcement Learning as One Big Sequence Modeling Problem
-
TransDreamer: Reinforcement Learning with Transformer World Models
Our purpose is to make this repo even better. If you are interested in contributing, please refer to HERE for instructions in contribution.
Awesome Decision Transformer is released under the Apache 2.0 license.