From 8ab3178ffd64a62d9dfec070ffb7d6eac0771e91 Mon Sep 17 00:00:00 2001 From: Arindam Jati Date: Fri, 8 Nov 2024 06:31:07 -0500 Subject: [PATCH 1/2] readme update with data download instructions --- README.md | 2 +- notebooks/hfdemo/tinytimemixer/README.md | 4 ++++ .../tinytimemixer/full_benchmarking/README.md | 14 +++++++++++++- 3 files changed, 18 insertions(+), 2 deletions(-) create mode 100644 notebooks/hfdemo/tinytimemixer/README.md diff --git a/README.md b/README.md index 9bb1f76f..ca22c3b6 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,6 @@ Public notebooks, utilities, and serving components for working with Time Series The core TSFM time series models have been made available on Hugging Face -- details can be found [here](https://github.com/ibm-granite/granite-tsfm/wiki). Information on the services component can be found [here](services/inference/README.md). - ## Python Version The current Python versions supported are 3.9, 3.10, 3.11, 3.12. @@ -28,6 +27,7 @@ pip install ".[notebooks]" - Transfer learning with `PatchTSMixer` [[Try it out]](https://github.com/ibm-granite/granite-tsfm/blob/main/notebooks/hfdemo/patch_tsmixer_transfer.ipynb) - Transfer learning with `PatchTST` [[Try it out]](https://github.com/ibm-granite/granite-tsfm/blob/main/notebooks/hfdemo/patch_tst_transfer.ipynb) - Getting started with `TinyTimeMixer (TTM)` [[Try it out]](https://github.com/ibm-granite/granite-tsfm/blob/main/notebooks/hfdemo/ttm_getting_started.ipynb) +- `TTM` full benchmarking scripts and results are available [[here]](https://github.com/ibm-granite/granite-tsfm/tree/main/notebooks/hfdemo/tinytimemixer/full_benchmarking) ## 📗 Google Colab Tutorials Run the TTM tutorial in Google Colab, and quickly build a forecasting application with the pre-trained TSFM models. diff --git a/notebooks/hfdemo/tinytimemixer/README.md b/notebooks/hfdemo/tinytimemixer/README.md new file mode 100644 index 00000000..9030486b --- /dev/null +++ b/notebooks/hfdemo/tinytimemixer/README.md @@ -0,0 +1,4 @@ +# Steps to run the M4 notebook + +## Fetching the M4 data +The M4 data can be downloaded from the [Time-Series-Library](https://github.com/thuml/Time-Series-Library). The authors of that library have shared the data through this [download link](https://drive.google.com/drive/folders/15zio96o3NK4XOoR5L88oaWcJDVOiqQo9). \ No newline at end of file diff --git a/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md b/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md index f0d26dc5..9ec37434 100644 --- a/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md +++ b/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md @@ -1,13 +1,25 @@ # Steps to run the full benchmarking +## Fetching the data +The evaluation data can be downloaded from the [Time-Series-Library](https://github.com/thuml/Time-Series-Library). The authors of that library have shared the data through this [download link](https://drive.google.com/drive/folders/1vE0ONyqPlym2JaaAoEe0XNDR8FS_d322). The ETT datasets can also be downloaded from [ETT-Github-Repository](https://github.com/zhouhaoyi/ETDataset). + +Download and save the datasets in a directory. For example, in `data_root_path`. + +## Running the scripts + 1. In terminal, the any one of the three bash scripts `granite-r2.sh`, `granite-r1.sh`, or `research-use-r2.sh`. 2. Run `summarize_results.py`. For example, ``` -sh granite-r2.sh +sh granite-r2.sh data_root_path/ python summarize_results.py -rd=results-granite-r2/ ``` It will run all benchmarking and dump the results. The dumped results are available in the CSV files. + + +## Benchmarking Results +Note that, although random seed has been set, the mean squared error (MSE) scores might not match the below scores exactly depending on the runtime environment. The following results were obtained in a Unix-based machine equipped with one NVIDIA A-100 GPU. + 1. TTM-Research-Use model results: - `combined_results-research-use-r2.csv`: Across all datasets, all TTM models, and all forecast horizons. - `combined_avg_results-research-use-r2.csv`: Across all datasets and all TTM models average over forecast horizons. From 70fd77623870f9a4dda335af73dd80bcbdd48d8d Mon Sep 17 00:00:00 2001 From: vijaye12 Date: Mon, 11 Nov 2024 09:38:49 -0500 Subject: [PATCH 2/2] minor changes --- notebooks/hfdemo/tinytimemixer/README.md | 4 ---- notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md | 5 +++-- notebooks/hfdemo/tinytimemixer/ttm_m4_hourly.ipynb | 4 +++- 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/notebooks/hfdemo/tinytimemixer/README.md b/notebooks/hfdemo/tinytimemixer/README.md index 9030486b..e69de29b 100644 --- a/notebooks/hfdemo/tinytimemixer/README.md +++ b/notebooks/hfdemo/tinytimemixer/README.md @@ -1,4 +0,0 @@ -# Steps to run the M4 notebook - -## Fetching the M4 data -The M4 data can be downloaded from the [Time-Series-Library](https://github.com/thuml/Time-Series-Library). The authors of that library have shared the data through this [download link](https://drive.google.com/drive/folders/15zio96o3NK4XOoR5L88oaWcJDVOiqQo9). \ No newline at end of file diff --git a/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md b/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md index 9ec37434..62b77fd7 100644 --- a/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md +++ b/notebooks/hfdemo/tinytimemixer/full_benchmarking/README.md @@ -1,9 +1,10 @@ # Steps to run the full benchmarking ## Fetching the data -The evaluation data can be downloaded from the [Time-Series-Library](https://github.com/thuml/Time-Series-Library). The authors of that library have shared the data through this [download link](https://drive.google.com/drive/folders/1vE0ONyqPlym2JaaAoEe0XNDR8FS_d322). The ETT datasets can also be downloaded from [ETT-Github-Repository](https://github.com/zhouhaoyi/ETDataset). +The evaluation data can be downloaded from any of the previous time-series github repos like autoformer or timesnet or informer. [Sample download link](https://drive.google.com/drive/folders/1vE0ONyqPlym2JaaAoEe0XNDR8FS_d322). The ETT datasets can also be downloaded from [ETT-Github-Repository](https://github.com/zhouhaoyi/ETDataset). -Download and save the datasets in a directory. For example, in `data_root_path`. +Download and save the datasets in a directory. For example, in `data_root_path`. +CSVs of each data should reside in location `data_root_path/$dataset_name/$dataset_name.csv` for our data utils to process them automatically. ## Running the scripts diff --git a/notebooks/hfdemo/tinytimemixer/ttm_m4_hourly.ipynb b/notebooks/hfdemo/tinytimemixer/ttm_m4_hourly.ipynb index df27c367..d5147e3f 100644 --- a/notebooks/hfdemo/tinytimemixer/ttm_m4_hourly.ipynb +++ b/notebooks/hfdemo/tinytimemixer/ttm_m4_hourly.ipynb @@ -27,7 +27,9 @@ "\n", "Note that other subsets of M4 data like (daily, monthly, quarterly, and yearly)\n", "are shorter in length and are not suitable for TTM-512-96 or TTM-1024-96 model.\n", - "Stay tuned for more TTM models!" + "Stay tuned for more TTM models!\n", + "\n", + "Dataset download link: [download link](https://drive.google.com/drive/folders/15zio96o3NK4XOoR5L88oaWcJDVOiqQo9)" ] }, {