Initial public release

README.md

@@ -2,7 +2,7 @@
 <p align="center">
   <a href="https://InhwanBae.github.io/"><strong>Inhwan Bae</strong></a>
   ·  
-  <a href="https://scholar.google.com/citations?user=OZcCxNQAAAAJ"><strong>Junoh Lee</strong></a>
+  <a href="https://leejunoh.com/"><strong>Junoh Lee</strong></a>
   ·
   <a href="https://scholar.google.com/citations?user=Ei00xroAAAAJ"><strong>Hae-Gon Jeon</strong></a>
   <br>
@@ -21,16 +21,108 @@
   <br>Traditional vs. Our language-based trajectory prediction, LMTraj.
 </div>
 
-[//]: # (<br>This repository contains the code for the EigenTrajectory&#40;𝔼𝕋&#41; space applied to the 10 traditional Euclidean-based trajectory predictors.)
-[//]: # (<br>EigenTrajectory-LB-EBM achieves ADE/FDE of 0.21/0.34 while requiring only 1 hour for training! )
+<br>This repository contains the code for the LMTrajectory framework.
+<br>**TL;DR**: Language model-based, Multimodal input, Multimodal output, Multi-task training approach for Zero-shot and Supervised human trajectory prediction. 
 
 <br>
 
 ## 💬 LMTrajectory Pipeline 🗨️
+* **Prompt-Based Approach**: Moving away from conventional numerical regression models, we reframe the task into a prompt-based question-answering perspective.
+* **Social Reasoning**: Beyond physics-based mathematical interaction modeling, our approach leverages language models to incorporate social reasoning.
+* **Multi-Task Training**: Supplementary tasks enhance the model's ability to grasp higher-level context through multi-task training.
+* **Numerical Tokenizer**: Our numerical tokenizer effectively separates text and numbers, enabling the model to learn correlations in sequential data.
+* **SOTA Performance**: Our holistic solution achieves state-of-the-art results on trajectory prediction benchmarks traditionally dominated by numerical regressors.
+
+<br>
+
+## Zero-Shot Evaluation
+### Setup
+**Environment**
+<br>All models were tested on Ubuntu 20.04 with Python 3.10 and PyTorch 2.0.1 with CUDA 11.7.
+Dependencies include Python packages such as `scipy`, `simdkalman` and `openai==0.28.0`.
+
+**Dataset**
+<br>Preprocessed [ETH](https://data.vision.ee.ethz.ch/cvl/aem/ewap_dataset_full.tgz) and [UCY](https://graphics.cs.ucy.ac.cy/research/downloads/crowd-data) datasets are released in this repository. 
+The train/validation/test splits are the same as those fond in [Social-GAN](https://github.com/agrimgupta92/sgan).
+
+**Sample**
+<br>We provide our zero-shot prediction results in the [release section](https://github.com/InhwanBae/LMTrajectory/releases). 
+These results include all multimodal trajectories and are available for use in future zero-shot research.
+
+### Evaluate LMTraj-ZERO
+**Preliminary**
+<br>To evaluate our LMTraj-ZERO model, you will need an `OPENAI_API_KEY` to access the OpenAI API. Create the API key using the [instruction](https://platform.openai.com/docs/api-reference/authentication) provided by OpenAI, and then paste the key into `./zero-shot/chatgpt_trajectory_predictor_v3` [line 25](https://github.com/InhwanBae/LMTrajectory/blob/main/zero-shot/chatgpt_trajectory_predictor_v3.py#L25).
+
+**Prediction**
+<br>We provide scripts to evaluate our LMTraj-ZERO model for all datasets simultaneously.
+Two scripts are provided in `./zero-shot/chatgpt_sequential_v3.sh` and `./zero-shot/chatgpt_multi_v3.sh`. The former script is used to evaluate our model step-by-step, and the latter script is used to evaluate our model with a thread pool for faster inference.
+```bash
+# Choose one of the following scripts to evaluate our LMTraj-ZERO model.
+./chatgpt_sequential_v3.sh -d <DATASET_ID> -m <LLM_MODEL_ID>
+./chatgpt_multi_v3.sh -d <DATASET_ID> -m <LLM_MODEL_ID>
+
+# Supported dataset id: 1 (ETH), 2 (HOTEL), 3 (UNIV), 4 (ZARA1), 5 (ZARA2)
+# Supported llm model id: 1 (gpt-3.5-turbo-0301), 2 (gpt-4-0314), 3 (gpt-3.5-turbo-1106), 4 (gpt-4-1106-preview)
+
+# Examples
+cd zero-shot
+./chatgpt_multi_v3.sh -d 1 -m 4
+./chatgpt_multi_v3.sh -d 2 -m 4
+```
+If an error is encountered, your progress will be saved. When you rerun the same script, it will skip the parts that were successfully executed and only regenerate the paths where issues occurred.
+
+If you want to run the model with custom hyperparameters or other models available by [OpenAI](https://platform.openai.com/docs/models), use `./zero-shot/chatgpt_trajectory_predictor_v3.py` instead of the script file. 
+<br>*Warning: A misclick could upgrade you to OpenAI Tier 5, as it did for me :(* 
+
+**Evaluation**
+As the final step, we provide code to evaluate the trajectories generated by our LMTraj-ZERO. To evaluate, first combine the predicted trajectories into a single JSON file.
+```bash
+python ./zero-shot/chatgpt-fragmented_dump_combiner.py --dataset <DATASET_ID> --model <LLM_MODEL_ID>
+
+# Supported dataset id: 1 (ETH), 2 (HOTEL), 3 (UNIV), 4 (ZARA1), 5 (ZARA2)
+# Supported llm model id: 1 (gpt-3.5-turbo-0301), 2 (gpt-4-0314), 3 (gpt-3.5-turbo-1106), 4 (gpt-4-1106-preview)
+
+# Examples
+python ./zero-shot/chatgpt-fragmented_dump_combiner.py --dataset 1 --model 4
+python ./zero-shot/chatgpt-fragmented_dump_combiner.py --dataset 2 --model 4
+```
+
+Next, evaluate the combined trajectories using ADE and FDE metrics.
+```bash
+python ./zero-shot/compute_ade_fde_from_dump.py
+
+# Supported dataset id: 1 (ETH), 2 (HOTEL), 3 (UNIV), 4 (ZARA1), 5 (ZARA2)
+# Supported llm model id: 1 (gpt-3.5-turbo-0301), 2 (gpt-4-0314), 3 (gpt-3.5-turbo-1106), 4 (gpt-4-1106-preview)
+
+# Examples
+python ./zero-shot/compute_ade_fde_from_dump.py --dataset 1 --model 4
+python ./zero-shot/compute_ade_fde_from_dump.py --dataset 2 --model 4
+```
+
+### Evaluate Algorithmic Models
+We provide four algorithmic models for comparison in zero-shot trajectory prediction task, available in `./zero-shot/algorithmic_model_benchmark.py`.
+The source code supports four extrapolation methods: stop, linear extrapolation, cubic extrapolation and Kalman filter.
+```bash
+python ./zero-shot/algorithmic_model_benchmark.py --model <MODEL_TYPE>
+
+# Examples
+python ./zero-shot/algorithmic_model_benchmark.py --model stop
+python ./zero-shot/algorithmic_model_benchmark.py --model linear
+python ./zero-shot/algorithmic_model_benchmark.py --model cubic
+python ./zero-shot/algorithmic_model_benchmark.py --model kalman
+```
+
+<br>
+
+[//]: # (## Supervised Training)
+[//]: # (## Supervised Evaluation)
+## Supervised Training & Evaluation
 
 - [x] 3/27 Paper [released](https://arxiv.org/abs/2403.18447)!
-- [ ] Source code will be released shortly. (Please check our [project page](https://inhwanbae.github.io/publication/lmtrajectory/) :)
+- [x] LMTraj-ZERO released!
+- [ ] LMTraj-SUP will be released shortly (before conference) :) 
 
+<br>
 
 ## 📖 Citation
 If you find this code useful for your research, please cite our trajectory prediction papers :)