diff --git a/README.html b/README.html index 941276f..26e7826 100644 --- a/README.html +++ b/README.html @@ -169,6 +169,7 @@
  • Quickstart
  • Dataset Information
  • Task Overview
    • +
  • Training
  • Evaluation
  • Baseline Models
    • @@ -372,6 +373,7 @@

    Contents

  • Getting Started
  • Build Your Own Model
  • Benchmarking
    • +
  • Motivation
    • @@ -388,10 +390,10 @@

    ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seas

    ChaosBench is a benchmark project to improve long-term forecasting of chaotic systems, in particular subseasonal-to-seasonal (S2S) climate, using ML approaches.

    Homepage 🔗: https://leap-stc.github.io/ChaosBench

    Paper 📚: https://arxiv.org/

    -

    Dataset 🤗: https://huggingface.co/datasets/juannat7/ChaosBench

    +

    Dataset 🤗: https://huggingface.co/datasets/LEAP/ChaosBench

    Features#

    -

    Overview of ChaosBench

    +

    Overview of ChaosBench

    1️⃣ Extended Observations. Spanning over 45 years (1979 - 2023) of ERA5 reanalysis

    2️⃣ Diverse Baselines. Wide selection of physics-based forecasts from leading national agencies in Europe, the UK, America, and Asia

    3️⃣ Differentiable Physics Metrics. Introduces two differentiable physics-based metrics to minimize the decay of power spectra at long forecasting horizon (blurriness)

    @@ -419,6 +421,13 @@

    BenchmarkingLeaderboard

    +
    +

    Motivation#

    +

    1️⃣ Collapse to Climatology. Performing comparable or worse than climatology renders these state-of-the-art-models operationally unusable +Collapse

    +

    2️⃣ Blurring Artifact. Averaged-out forecasts is of little use when one attempts to predict extreme events requiring high-fidelity on the S2S scale (e.g., droughts, hurricanes) +Blurring

    +
    @@ -482,6 +491,7 @@

    BenchmarkingGetting Started
  • Build Your Own Model
  • Benchmarking
    • +
  • Motivation
    • diff --git a/_images/all_rmse_sota.png b/_images/all_rmse_sota.png new file mode 100644 index 0000000..af53a6f Binary files /dev/null and b/_images/all_rmse_sota.png differ diff --git a/_images/chaosbench_scheme.jpg b/_images/chaosbench_scheme.jpg new file mode 100644 index 0000000..3450bde Binary files /dev/null and b/_images/chaosbench_scheme.jpg differ diff --git a/_images/preds_climax_q700_direct_Task1.png b/_images/preds_climax_q700_direct_Task1.png new file mode 100644 index 0000000..07963a2 Binary files /dev/null and b/_images/preds_climax_q700_direct_Task1.png differ diff --git a/_sources/README.md b/_sources/README.md index 4be9024..ee346d4 100644 --- a/_sources/README.md +++ b/_sources/README.md @@ -7,8 +7,7 @@ Homepage 🔗: https://leap-stc.github.io/ChaosBench Paper 📚: https://arxiv.org/ -Dataset 🤗: https://huggingface.co/datasets/juannat7/ChaosBench - +Dataset 🤗: https://huggingface.co/datasets/LEAP/ChaosBench ## Features @@ -22,7 +21,6 @@ Dataset 🤗: https://huggingface.co/datasets/juannat7/ChaosBench 4️⃣ __Large-Scale Benchmarking__. Systematic evaluation for state-of-the-art ML-based weather models like PanguWeather, FourcastNetV2, ViT/ClimaX, and Graphcast - ## Getting Started - [Quickstart](https://leap-stc.github.io/ChaosBench/quickstart.html) - [Dataset Overview](https://leap-stc.github.io/ChaosBench/dataset.html) @@ -35,4 +33,12 @@ Dataset 🤗: https://huggingface.co/datasets/juannat7/ChaosBench ## Benchmarking - [Baseline Models](https://leap-stc.github.io/ChaosBench/baseline.html) -- [Leaderboard](https://leap-stc.github.io/ChaosBench/leaderboard.html) \ No newline at end of file +- [Leaderboard](https://leap-stc.github.io/ChaosBench/leaderboard.html) + + +## Motivation +1️⃣ __Collapse to Climatology__. Performing comparable or worse than climatology renders these state-of-the-art-models operationally unusable +![Collapse](docs/all_rmse_sota.png) + +2️⃣ __Blurring Artifact__. Averaged-out forecasts is of little use when one attempts to predict extreme events requiring high-fidelity on the S2S scale (e.g., droughts, hurricanes) +![Blurring](docs/preds_climax_q700_direct_Task1.png) \ No newline at end of file diff --git a/_sources/baseline.md b/_sources/baseline.md index 9077477..603f824 100644 --- a/_sources/baseline.md +++ b/_sources/baseline.md @@ -23,7 +23,7 @@ We differentiate between physics-based and data-driven models. The former is suc - [x] GraphCast ## Model Checkpoints -Checkpoints for data-driven models are accessible from [here](https://huggingface.co/datasets/juannat7/ChaosBench/tree/main/logs) +Checkpoints for data-driven models are accessible from [here](https://huggingface.co/datasets/LEAP/ChaosBench/tree/main/logs) - Data-driven models are indicated by the `_s2s` suffix (e.g., `unet_s2s`). diff --git a/_sources/evaluation.md b/_sources/evaluation.md index 387e1f0..0ab0267 100644 --- a/_sources/evaluation.md +++ b/_sources/evaluation.md @@ -22,7 +22,7 @@ __For example__, in our `unet_s2s` baseline model, we can run: ## Accessing Baseline Scores -You can access the complete scores (in `.csv` format) for data-driven, physics-based models, climatology, and persistence [here](https://huggingface.co/datasets/juannat7/ChaosBench/tree/main/logs). Below is a snippet from `logs/climatology/eval/rmse_climatology.csv`, where each row represents ``, such as `RMSE`, at each future timestep. +You can access the complete scores (in `.csv` format) for data-driven, physics-based models, climatology, and persistence [here](https://huggingface.co/datasets/LEAP/ChaosBench/tree/main/logs). Below is a snippet from `logs/climatology/eval/rmse_climatology.csv`, where each row represents ``, such as `RMSE`, at each future timestep. | z-10 | z-50 | z-100 | z-200 | z-300 | ... | w-1000 | |----------|----------|----------|----------|----------|-----|----------| diff --git a/_sources/quickstart.md b/_sources/quickstart.md index 3b7c96a..3779cf5 100644 --- a/_sources/quickstart.md +++ b/_sources/quickstart.md @@ -8,11 +8,11 @@ cd ChaosBench mkdir data ``` -**Step 3**: Navigate to `chaosbench/config.py` and change the field `DATA_DIR = //ChaosBench/data` (_Provide absolute path_) +**Step 3**: Navigate to `chaosbench/config.py` and change the field `DATA_DIR = ChaosBench/data` **Step 4**: Initialize the space by running ``` -cd //ChaosBench/data/ +cd ChaosBench/data/ wget https://huggingface.co/datasets/juannat7/ChaosBench/blob/main/process.sh chmod +x process.sh ``` diff --git a/_sources/task.md b/_sources/task.md index 946338a..afb461f 100644 --- a/_sources/task.md +++ b/_sources/task.md @@ -14,6 +14,7 @@ __NOTE__: Before training your own model [instructions here](https://leap-stc.gi - Task 2️⃣: `only_headline: True`. By default, it is going to optimize on {t-850, z-500, q-700}. To change this, modify the `HEADLINE_VARS` field in `chaosbench/config.py` +# Training Strategies In addition, we also provide flags to train the model either __autoregressively__ or __directly__. - Autoregressive: Using current output as the next model input. The number of iterative steps is defined in the `n_step: ` field. For our baselines, we set `N_STEP = 5`. diff --git a/_sources/training.md b/_sources/training.md index 2d062fe..578dfdd 100644 --- a/_sources/training.md +++ b/_sources/training.md @@ -4,11 +4,14 @@ We will outline how one can implement their own data-driven models. Several examples, including ED, FNO, ResNet, and UNet have been provided. -**Step 1**: Define your model class in `chaosbench/models/.py`. At present, we only support models built with `PyTorch` +### Step 1 +Define your model class in `chaosbench/models/.py` -**Step 2**: Initialize your model in `chaosbench/models/model.py` under `__init__` method in `S2SBenchmarkModel` class +### Step 2 +Initialize your model in `chaosbench/models/model.py` under `S2SBenchmarkModel.__init__` -**Step 3**: Write a configuration file in `chaosbench/configs/_s2s.yaml`. We recommend reading the details on the definition of [hyperparameters](https://leap-stc.github.io/ChaosBench/baseline.html) and the different [task]((https://leap-stc.github.io/ChaosBench/task.html) before training. Also change the `model_name: _s2s` to ensure correct pathing +### Step 3 +Write a configuration file in `chaosbench/configs/_s2s.yaml`. Details on the definition of [hyperparameters](https://leap-stc.github.io/ChaosBench/baseline.html) and the different [task](https://leap-stc.github.io/ChaosBench/task.html). Also change the `model_name: _s2s` to ensure correct pathing - Task 1️⃣ (autoregressive): `only_headline: False ; n_step: ` - Task 1️⃣ (direct): `only_headline: False ; n_step: 1 ; lead_time: ` @@ -17,8 +20,8 @@ We will outline how one can implement their own data-driven models. Several exam - Task 2️⃣ (direct): `only_headline: True ; n_step: 1 ; lead_time: ` -**Step 4**: Train by running `python train.py --config_filepath chaosbench/configs/_s2s.yaml` +### Step 4 +Train by running `python train.py --config_filepath chaosbench/configs/_s2s.yaml` -**Step 5**: Done! -__NOTE__: Remember to replace `` with your own model name, e.g., `unet`. Checkpoints and logs would be automatically generated in `logs/_s2s/`. \ No newline at end of file +Remember to replace `` with your own model name, e.g., `unet`. Checkpoints and logs would be automatically generated in `logs/_s2s/`. \ No newline at end of file diff --git a/baseline.html b/baseline.html index 966db77..d98852f 100644 --- a/baseline.html +++ b/baseline.html @@ -170,6 +170,7 @@
  • Quickstart
  • Dataset Information
  • Task Overview
    • +
  • Training
  • Evaluation
  • Baseline Models
    • @@ -415,7 +416,7 @@

    Model Definition

    Model Checkpoints#

    -

    Checkpoints for data-driven models are accessible from here

    +

    Checkpoints for data-driven models are accessible from here

    • Data-driven models are indicated by the _s2s suffix (e.g., unet_s2s).

    • The hyperparameter specifications are located in version_xx/lightning_logs/hparams.yaml. The hyperparameters encode the following:

      diff --git a/dataset.html b/dataset.html index 381ffaa..59ec755 100644 --- a/dataset.html +++ b/dataset.html @@ -170,6 +170,7 @@
    • Quickstart
    • Dataset Information
    • Task Overview
      • +
    • Training
    • Evaluation
    • Baseline Models
      • diff --git a/evaluation.html b/evaluation.html index 1a4dfea..86491be 100644 --- a/evaluation.html +++ b/evaluation.html @@ -168,6 +168,7 @@
    • Quickstart
    • Dataset Information
    • Task Overview
      • +
    • Training
    • Evaluation
    • Baseline Models
      • @@ -402,7 +403,7 @@

      Evaluation

      Accessing Baseline Scores#

      -

      You can access the complete scores (in .csv format) for data-driven, physics-based models, climatology, and persistence here. Below is a snippet from logs/climatology/eval/rmse_climatology.csv, where each row represents <METRIC>, such as RMSE, at each future timestep.

      +

      You can access the complete scores (in .csv format) for data-driven, physics-based models, climatology, and persistence here. Below is a snippet from logs/climatology/eval/rmse_climatology.csv, where each row represents <METRIC>, such as RMSE, at each future timestep.

      diff --git a/genindex.html b/genindex.html index a27f04b..2aa2750 100644 --- a/genindex.html +++ b/genindex.html @@ -167,6 +167,7 @@
    • Quickstart
    • Dataset Information
    • Task Overview
      • +
    • Training
    • Evaluation
    • Baseline Models
      • diff --git a/leaderboard.html b/leaderboard.html index b38c34b..b3253bb 100644 --- a/leaderboard.html +++ b/leaderboard.html @@ -167,6 +167,7 @@
    • Quickstart
    • Dataset Information
    • Task Overview
      • +
    • Training
    • Evaluation
    • Baseline Models
      • diff --git a/quickstart.html b/quickstart.html index 8759f61..20f7d7d 100644 --- a/quickstart.html +++ b/quickstart.html @@ -168,6 +168,7 @@
    • Quickstart
    • Dataset Information
    • Task Overview
      • +
    • Training
    • Evaluation
    • Baseline Models
      • @@ -377,9 +378,9 @@

      Quickstartmkdir data -

      Step 3: Navigate to chaosbench/config.py and change the field DATA_DIR = /<YOUR_WORKING_DIR>/ChaosBench/data (Provide absolute path)

      +

      Step 3: Navigate to chaosbench/config.py and change the field DATA_DIR = ChaosBench/data

      Step 4: Initialize the space by running

      -
      cd /<YOUR_WORKING_DIR>/ChaosBench/data/
+
      cd ChaosBench/data/
 wget https://huggingface.co/datasets/juannat7/ChaosBench/blob/main/process.sh
 chmod +x process.sh
 
      
diff --git a/search.html b/search.html
index 6c42c0d..878c734 100644
--- a/search.html
+++ b/search.html
@@ -169,6 +169,7 @@
 
    • Quickstart
      • 
 
    • Dataset Information
      • 
 
    • Task Overview
      • 
+
 
    • Training
      • 
 
    • Evaluation
      • 
 
    • Baseline Models
      • 
diff --git a/searchindex.js b/searchindex.js
index e90c29d..02c7f35 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["README", "baseline", "dataset", "evaluation", "leaderboard", "quickstart", "task", "training"], "filenames": ["README.md", "baseline.md", "dataset.md", "evaluation.md", "leaderboard.md", "quickstart.md", "task.md", "training.md"], "titles": ["ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction", "Baseline Models", "Dataset Information", "Evaluation", "Leaderboard", "Quickstart", "Task Overview", "Training"], "terms": {"project": 0, "improv": 0, "long": [0, 6], "term": [0, 6], "forecast": [0, 1, 4, 6], "chaotic": 0, "system": 0, "particular": [0, 1], "s2": 0, "us": [0, 1, 4, 6], "ml": [0, 4], "approach": [0, 1], "homepag": 0, "http": [0, 5, 7], "leap": [0, 7], "stc": [0, 7], "github": [0, 5, 7], "io": [0, 7], "paper": [0, 1], "arxiv": 0, "org": 0, "dataset": [0, 5], "huggingfac": [0, 5], "co": [0, 5], "juannat7": [0, 5], "1": [0, 1, 2, 3, 4, 5, 6, 7], "extend": 0, "observ": [0, 2], "span": 0, "over": 0, "45": 0, "year": [0, 1], "1979": [0, 2], "2023": [0, 2, 3], "era5": [0, 2, 5], "reanalysi": [0, 2], "2": [0, 1, 4, 5, 6, 7], "divers": 0, "baselin": [0, 6], "wide": 0, "select": 0, "from": [0, 1, 3, 5], "lead": [0, 4, 6], "nation": [0, 1], "agenc": 0, "europ": 0, "uk": [0, 1], "america": 0, "asia": 0, "3": [0, 1, 5, 7], "differenti": [0, 1], "metric": [0, 3, 4], "introduc": 0, "two": [0, 6], "minim": 0, "decai": 0, "power": 0, "spectra": 0, "horizon": 0, "blurri": 0, "4": [0, 1, 3, 5, 7], "larg": 0, "scale": 0, "systemat": 0, "evalu": [0, 4], "state": 0, "art": 0, "weather": [0, 1, 6], "like": 0, "panguweath": [0, 1], "fourcastnetv2": [0, 1], "vit": [0, 1], "climax": [0, 1], "graphcast": [0, 1], "quickstart": 0, "overview": 0, "task": [0, 1, 4, 7], "train": [0, 1, 3, 6], "leaderboard": 0, "we": [1, 3, 4, 6, 7], "between": 1, "physic": [1, 2, 3, 4], "base": [1, 2, 3, 4], "data": [1, 3, 4, 5, 7], "driven": [1, 3, 4, 7], "The": [1, 2, 6], "former": 1, "succintli": 1, "illustr": 1, "figur": 1, "below": [1, 3], "ukmo": [1, 5], "meteorolog": 1, "offic": 1, "ncep": [1, 5], "center": 1, "environment": 1, "predict": [1, 6], "cma": [1, 5], "china": 1, "administr": 1, "ecmwf": [1, 5], "european": 1, "centr": 1, "medium": 1, "rang": 1, "lag": 1, "autoencod": 1, "fourier": 1, "neural": 1, "oper": 1, "fno": [1, 7], "resnet": [1, 7], "unet": [1, 7], "ar": [1, 2, 4, 6], "access": 1, "here": [1, 3, 6], "indic": [1, 2], "_s2": [1, 3, 6, 7], "suffix": 1, "e": [1, 4, 5, 7], "g": [1, 4, 5, 7], "unet_s2": [1, 3], "hyperparamet": [1, 7], "specif": [1, 2, 6], "locat": 1, "version_xx": 1, "lightning_log": 1, "hparam": 1, "yaml": [1, 6, 7], "encod": 1, "follow": [1, 2], "lead_tim": [1, 6, 7], "default": [1, 6], "arbitrari": [1, 4], "delta_t": 1, "finetun": 1, "direct": [1, 3, 6, 7], "n_step": [1, 6, 7], "number": [1, 6], "autoregress": [1, 3, 6, 7], "step": [1, 5, 6, 7], "s": [1, 3, 6], "only_headlin": [1, 6, 7], "fals": [1, 6, 7], "optim": [1, 6], "true": [1, 6, 7], "batch_siz": 1, "batch": 1, "size": 1, "train_year": 1, "list": 1, "val_year": 1, "valid": 1, "epoch": 1, "input_s": 1, "input": [1, 2, 5, 6], "channel": [1, 2, 4, 6], "learning_r": 1, "updat": 1, "each": [1, 3, 4, 5], "iter": [1, 6], "model_nam": [1, 3, 7], "name": [1, 4, 7], "consist": 1, "num_work": 1, "worker": 1, "dataload": 1, "output_s": 1, "output": [1, 6], "t_max": 1, "cosin": 1, "learn": 1, "rate": 1, "schedul": 1, "cycl": 1, "note": [1, 2, 5, 6, 7], "you": [1, 3, 5, 6], "notic": 1, "version": [1, 3], "0": [1, 3], "up": 1, "dai": [1, 4], "ahead": 1, "5": [1, 3, 5, 6, 7], "onli": [1, 7], "do": 1, "have": [1, 7], "mani": 1, "more": [1, 4], "thi": [1, 4, 6], "check": 1, "effect": 1, "vs": 1, "describ": 1, "In": [1, 6], "6": [1, 3], "7": [1, 3], "8": [1, 3], "9": [1, 3], "10": [1, 2, 3, 6], "11": [1, 3], "12": [1, 3], "13": 1, "14": 1, "15": [1, 6], "16": 1, "17": 1, "18": 1, "19": 1, "20": [1, 6], "21": 1, "element": 1, "arrai": 1, "correspond": [1, 3], "delta": 1, "t": [1, 2, 6], "25": [1, 6], "30": [1, 6], "35": [1, 6], "40": [1, 6], "44": [1, 4, 6], "hand": [2, 7], "explor": 2, "chaosbench": [2, 5, 6, 7], "notebook": [2, 7], "01a_s2s_data_explor": 2, "ipynb": [2, 7], "target": [2, 6], "tabl": 2, "48": [2, 4], "variabl": [2, 6], "avail": 2, "model": [2, 3, 4, 6, 7], "contain": [2, 4], "all": [2, 4, 6], "field": [2, 5, 6], "includ": [2, 7], "uncheck": 2, "box": 2, "paramet": 2, "level": 2, "hpa": 2, "1000": [2, 3, 6], "925": 2, "850": [2, 6], "700": [2, 6], "500": [2, 6], "300": [2, 3], "200": [2, 3], "100": [2, 3], "50": [2, 3], "geopotenti": 2, "height": 2, "z": [2, 3, 6], "gpm": 2, "humid": [2, 6], "q": [2, 6], "kg": 2, "temperatur": [2, 6], "k": 2, "u": 2, "compon": 2, "wind": 2, "ms": [2, 4], "v": 2, "vertic": 2, "veloc": 2, "w": [2, 3], "pa": 2, "after": 3, "your": [3, 5, 6, 7], "can": [3, 5, 6, 7], "simpli": 3, "perform": [3, 6], "run": [3, 5, 7], "python": [3, 7], "eval_it": 3, "py": [3, 5, 6, 7], "your_model": [3, 6, 7], "eval_year": 3, "version_num": 3, "eval_direct": 3, "task_num": 3, "where": [3, 6], "pytorch_lightn": 3, "gener": [3, 6, 7], "dure": 3, "For": [3, 4, 5, 6], "exampl": [3, 7], "our": [3, 4, 6], "complet": 3, "csv": [3, 4], "format": 3, "climatolog": [3, 5], "persist": 3, "snippet": 3, "log": [3, 7], "eval": [3, 4], "rmse_climatolog": 3, "row": 3, "repres": 3, "rmse": [3, 4], "futur": [3, 6], "timestep": 3, "539": 3, "7944": 3, "285": 3, "9499": 3, "215": 3, "14742": 3, "186": 3, "43161": 3, "166": 3, "28784": 3, "07912156": 3, "538": 3, "9591": 3, "43832": 3, "214": 3, "82317": 3, "23743": 3, "16902": 3, "07907272": 3, "1366": 3, "284": 3, "96063": 3, "51791": 3, "04941": 3, "04732": 3, "07903882": 3, "divid": 4, "class": [4, 7], "which": 4, "cover": 4, "convent": 4, "comput": 4, "vision": 4, "aim": [4, 6], "construct": 4, "faith": 4, "explain": 4, "bia": 4, "anomali": 4, "correl": 4, "coeffici": 4, "acc": 4, "multiscal": 4, "structur": 4, "similar": 4, "index": 4, "ssim": 4, "spectral": 4, "diverg": 4, "specdiv": 4, "residu": 4, "specr": 4, "etc": 4, "folder": 4, "individu": [4, 5], "file": [4, 6, 7], "within": 4, "score": 4, "question": 4, "entir": [4, 6], "60": [4, 6], "n": 4, "across": 4, "time": [4, 5, 6], "clone": 5, "repositori": 5, "creat": 5, "local": 5, "directori": 5, "store": 5, "cd": 5, "mkdir": 5, "navig": 5, "config": [5, 6, 7], "chang": [5, 6, 7], "data_dir": 5, "your_working_dir": 5, "provid": [5, 6, 7], "absolut": 5, "path": [5, 7], "initi": [5, 7], "space": 5, "wget": 5, "blob": 5, "main": 5, "process": 5, "sh": 5, "chmod": 5, "x": 5, "download": 5, "also": [5, 6, 7], "line": 5, "one": [5, 7], "retriev": 5, "requir": 5, "option": 5, "simul": 5, "present": [6, 7], "still": 6, "take": 6, "full": 6, "benchmark": 6, "subset": 6, "dynam": 6, "It": 6, "simultan": 6, "harder": 6, "build": 6, "emul": 6, "condit": 6, "spars": 6, "surfac": 6, "befor": [6, 7], "own": [6, 7], "instruct": 6, "specifi": 6, "By": 6, "go": 6, "To": 6, "modifi": 6, "headline_var": 6, "addit": 6, "flag": 6, "either": 6, "directli": 6, "current": 6, "next": 6, "defin": [6, 7], "set": 6, "ensur": [6, 7], "case": 6, "workflow": 7, "02a_s2s_model": 7, "03a_s2s_train": 7, "outlin": 7, "how": 7, "implement": 7, "sever": 7, "ed": 7, "been": 7, "At": 7, "support": 7, "built": 7, "pytorch": 7, "under": 7, "__init__": 7, "method": 7, "s2sbenchmarkmodel": 7, "write": 7, "configur": 7, "recommend": 7, "read": 7, "detail": 7, "definit": 7, "differ": 7, "html": 7, "correct": 7, "config_filepath": 7, "done": 7, "rememb": 7, "replac": 7, "checkpoint": 7, "would": 7, "automat": 7}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"chaosbench": 0, "A": 0, "multi": 0, "channel": 0, "physic": 0, "base": 0, "benchmark": 0, "subseason": 0, "season": 0, "climat": 0, "predict": 0, "featur": 0, "get": 0, "start": 0, "build": 0, "your": 0, "own": 0, "model": [0, 1], "baselin": [1, 3], "definit": 1, "checkpoint": 1, "dataset": 2, "inform": 2, "evalu": 3, "access": 3, "score": 3, "leaderboard": 4, "quickstart": 5, "task": 6, "overview": 6, "train": 7}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 6, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinx": 56}})
\ No newline at end of file
+Search.setIndex({"docnames": ["README", "baseline", "dataset", "evaluation", "leaderboard", "quickstart", "task", "training"], "filenames": ["README.md", "baseline.md", "dataset.md", "evaluation.md", "leaderboard.md", "quickstart.md", "task.md", "training.md"], "titles": ["ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction", "Baseline Models", "Dataset Information", "Evaluation", "Leaderboard", "Quickstart", "Task Overview", "Training"], "terms": {"project": 0, "improv": 0, "long": [0, 6], "term": [0, 6], "forecast": [0, 1, 4, 6], "chaotic": 0, "system": 0, "particular": [0, 1], "s2": 0, "us": [0, 1, 4, 6], "ml": [0, 4], "approach": [0, 1], "homepag": 0, "http": [0, 5], "leap": 0, "stc": 0, "github": [0, 5], "io": 0, "paper": [0, 1], "arxiv": 0, "org": 0, "dataset": [0, 5], "huggingfac": [0, 5], "co": [0, 5], "1": [0, 1, 2, 3, 4, 5, 6], "extend": 0, "observ": [0, 2], "span": 0, "over": 0, "45": 0, "year": [0, 1], "1979": [0, 2], "2023": [0, 2, 3], "era5": [0, 2, 5], "reanalysi": [0, 2], "2": [0, 1, 4, 5, 6], "divers": 0, "baselin": [0, 6], "wide": 0, "select": 0, "from": [0, 1, 3, 5], "lead": [0, 4, 6], "nation": [0, 1], "agenc": 0, "europ": 0, "uk": [0, 1], "america": 0, "asia": 0, "3": [0, 1, 5], "differenti": [0, 1], "metric": [0, 3, 4], "introduc": 0, "two": [0, 6], "minim": 0, "decai": 0, "power": 0, "spectra": 0, "horizon": 0, "blurri": 0, "4": [0, 1, 3, 5], "larg": 0, "scale": 0, "systemat": 0, "evalu": [0, 4], "state": 0, "art": 0, "weather": [0, 1, 6], "like": 0, "panguweath": [0, 1], "fourcastnetv2": [0, 1], "vit": [0, 1], "climax": [0, 1], "graphcast": [0, 1], "quickstart": 0, "overview": 0, "task": [0, 1, 4, 7], "train": [0, 1, 3], "leaderboard": 0, "collaps": 0, "climatolog": [0, 3, 5], "perform": [0, 3, 6], "compar": 0, "wors": 0, "than": 0, "render": 0, "operation": 0, "unus": 0, "blur": 0, "artifact": 0, "averag": 0, "out": 0, "littl": 0, "when": 0, "one": [0, 5, 7], "attempt": 0, "extrem": 0, "event": 0, "requir": [0, 5], "high": 0, "fidel": 0, "e": [0, 1, 4, 5, 7], "g": [0, 1, 4, 5, 7], "drought": 0, "hurrican": 0, "we": [1, 3, 4, 6, 7], "between": 1, "physic": [1, 2, 3, 4], "base": [1, 2, 3, 4], "data": [1, 3, 4, 5, 7], "driven": [1, 3, 4, 7], "The": [1, 2, 6], "former": 1, "succintli": 1, "illustr": 1, "figur": 1, "below": [1, 3], "ukmo": [1, 5], "meteorolog": 1, "offic": 1, "ncep": [1, 5], "center": 1, "environment": 1, "predict": [1, 6], "cma": [1, 5], "china": 1, "administr": 1, "ecmwf": [1, 5], "european": 1, "centr": 1, "medium": 1, "rang": 1, "lag": 1, "autoencod": 1, "fourier": 1, "neural": 1, "oper": 1, "fno": [1, 7], "resnet": [1, 7], "unet": [1, 7], "ar": [1, 2, 4, 6], "access": 1, "here": [1, 3, 6], "indic": [1, 2], "_s2": [1, 3, 6, 7], "suffix": 1, "unet_s2": [1, 3], "hyperparamet": [1, 7], "specif": [1, 2, 6], "locat": 1, "version_xx": 1, "lightning_log": 1, "hparam": 1, "yaml": [1, 6, 7], "encod": 1, "follow": [1, 2], "lead_tim": [1, 6, 7], "default": [1, 6], "arbitrari": [1, 4], "delta_t": 1, "finetun": 1, "direct": [1, 3, 6, 7], "n_step": [1, 6, 7], "number": [1, 6], "autoregress": [1, 3, 6, 7], "step": [1, 5, 6], "s": [1, 3, 6], "only_headlin": [1, 6, 7], "fals": [1, 6, 7], "optim": [1, 6], "true": [1, 6, 7], "batch_siz": 1, "batch": 1, "size": 1, "train_year": 1, "list": 1, "val_year": 1, "valid": 1, "epoch": 1, "input_s": 1, "input": [1, 2, 5, 6], "channel": [1, 2, 4, 6], "learning_r": 1, "updat": 1, "each": [1, 3, 4, 5], "iter": [1, 6], "model_nam": [1, 3, 7], "name": [1, 4, 7], "consist": 1, "num_work": 1, "worker": 1, "dataload": 1, "output_s": 1, "output": [1, 6], "t_max": 1, "cosin": 1, "learn": 1, "rate": 1, "schedul": 1, "cycl": 1, "note": [1, 2, 5, 6, 7], "you": [1, 3, 5, 6], "notic": 1, "version": [1, 3], "0": [1, 3], "up": 1, "dai": [1, 4], "ahead": 1, "5": [1, 3, 5, 6], "onli": 1, "do": 1, "have": [1, 7], "mani": 1, "more": [1, 4], "thi": [1, 4, 6], "check": 1, "effect": 1, "vs": 1, "describ": 1, "In": [1, 6], "6": [1, 3], "7": [1, 3], "8": [1, 3], "9": [1, 3], "10": [1, 2, 3, 6], "11": [1, 3], "12": [1, 3], "13": 1, "14": 1, "15": [1, 6], "16": 1, "17": 1, "18": 1, "19": 1, "20": [1, 6], "21": 1, "element": 1, "arrai": 1, "correspond": [1, 3], "delta": 1, "t": [1, 2, 6], "25": [1, 6], "30": [1, 6], "35": [1, 6], "40": [1, 6], "44": [1, 4, 6], "hand": [2, 7], "explor": 2, "chaosbench": [2, 5, 6, 7], "notebook": [2, 7], "01a_s2s_data_explor": 2, "ipynb": [2, 7], "target": [2, 6], "tabl": 2, "48": [2, 4], "variabl": [2, 6], "avail": 2, "model": [2, 3, 4, 6, 7], "contain": [2, 4], "all": [2, 4, 6], "field": [2, 5, 6], "includ": [2, 7], "uncheck": 2, "box": 2, "paramet": 2, "level": 2, "hpa": 2, "1000": [2, 3, 6], "925": 2, "850": [2, 6], "700": [2, 6], "500": [2, 6], "300": [2, 3], "200": [2, 3], "100": [2, 3], "50": [2, 3], "geopotenti": 2, "height": 2, "z": [2, 3, 6], "gpm": 2, "humid": [2, 6], "q": [2, 6], "kg": 2, "temperatur": [2, 6], "k": 2, "u": 2, "compon": 2, "wind": 2, "ms": [2, 4], "v": 2, "vertic": 2, "veloc": 2, "w": [2, 3], "pa": 2, "after": 3, "your": [3, 5, 6, 7], "can": [3, 5, 6, 7], "simpli": 3, "run": [3, 5, 7], "python": [3, 7], "eval_it": 3, "py": [3, 5, 6, 7], "your_model": [3, 6, 7], "eval_year": 3, "version_num": 3, "eval_direct": 3, "task_num": 3, "where": [3, 6], "pytorch_lightn": 3, "gener": [3, 6, 7], "dure": 3, "For": [3, 4, 5, 6], "exampl": [3, 7], "our": [3, 4, 6], "complet": 3, "csv": [3, 4], "format": 3, "persist": 3, "snippet": 3, "log": [3, 7], "eval": [3, 4], "rmse_climatolog": 3, "row": 3, "repres": 3, "rmse": [3, 4], "futur": [3, 6], "timestep": 3, "539": 3, "7944": 3, "285": 3, "9499": 3, "215": 3, "14742": 3, "186": 3, "43161": 3, "166": 3, "28784": 3, "07912156": 3, "538": 3, "9591": 3, "43832": 3, "214": 3, "82317": 3, "23743": 3, "16902": 3, "07907272": 3, "1366": 3, "284": 3, "96063": 3, "51791": 3, "04941": 3, "04732": 3, "07903882": 3, "divid": 4, "class": [4, 7], "which": 4, "cover": 4, "convent": 4, "comput": 4, "vision": 4, "aim": [4, 6], "construct": 4, "faith": 4, "explain": 4, "bia": 4, "anomali": 4, "correl": 4, "coeffici": 4, "acc": 4, "multiscal": 4, "structur": 4, "similar": 4, "index": 4, "ssim": 4, "spectral": 4, "diverg": 4, "specdiv": 4, "residu": 4, "specr": 4, "etc": 4, "folder": 4, "individu": [4, 5], "file": [4, 6, 7], "within": 4, "score": 4, "question": 4, "entir": [4, 6], "60": [4, 6], "n": 4, "across": 4, "time": [4, 5, 6], "clone": 5, "repositori": 5, "creat": 5, "local": 5, "directori": 5, "store": 5, "cd": 5, "mkdir": 5, "navig": 5, "config": [5, 6, 7], "chang": [5, 6, 7], "data_dir": 5, "initi": [5, 7], "space": 5, "wget": 5, "juannat7": 5, "blob": 5, "main": 5, "process": 5, "sh": 5, "chmod": 5, "x": 5, "download": 5, "also": [5, 6, 7], "line": 5, "retriev": 5, "option": 5, "simul": 5, "present": 6, "still": 6, "take": 6, "full": 6, "benchmark": 6, "subset": 6, "dynam": 6, "It": 6, "simultan": 6, "harder": 6, "build": 6, "emul": 6, "condit": 6, "spars": 6, "surfac": 6, "befor": 6, "own": [6, 7], "instruct": 6, "specifi": 6, "By": 6, "go": 6, "To": 6, "modifi": 6, "headline_var": 6, "addit": 6, "provid": [6, 7], "flag": 6, "either": 6, "directli": 6, "current": 6, "next": 6, "defin": [6, 7], "set": 6, "ensur": [6, 7], "case": 6, "workflow": 7, "02a_s2s_model": 7, "03a_s2s_train": 7, "outlin": 7, "how": 7, "implement": 7, "sever": 7, "ed": 7, "been": 7, "under": 7, "s2sbenchmarkmodel": 7, "__init__": 7, "write": 7, "configur": 7, "detail": 7, "definit": 7, "differ": 7, "correct": 7, "path": 7, "config_filepath": 7, "rememb": 7, "replac": 7, "checkpoint": 7, "would": 7, "automat": 7}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"chaosbench": 0, "A": 0, "multi": 0, "channel": 0, "physic": 0, "base": 0, "benchmark": 0, "subseason": 0, "season": 0, "climat": 0, "predict": 0, "featur": 0, "get": 0, "start": 0, "build": 0, "your": 0, "own": 0, "model": [0, 1], "motiv": 0, "baselin": [1, 3], "definit": 1, "checkpoint": 1, "dataset": 2, "inform": 2, "evalu": 3, "access": 3, "score": 3, "leaderboard": 4, "quickstart": 5, "task": 6, "overview": 6, "train": [6, 7], "strategi": 6, "step": 7, "1": 7, "2": 7, "3": 7, "4": 7}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 6, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinx": 56}})
\ No newline at end of file
diff --git a/task.html b/task.html
index bcbd08a..be76db3 100644
--- a/task.html
+++ b/task.html
@@ -170,6 +170,7 @@
 
    • Quickstart
      • 
 
    • Dataset Information
      • 
 
    • Task Overview
      • 
+
 
    • Training
      • 
 
    • Evaluation
      • 
 
    • Baseline Models
      • 
@@ -346,7 +347,9 @@
   
 `);
 
-
+
      @@ -362,6 +365,16 @@

      Task Overview

      +
      +

      Contents

      +
      +
      @@ -385,6 +398,9 @@

      Task Overviewonly_headline: False

    • Task 2️⃣: only_headline: True. By default, it is going to optimize on {t-850, z-500, q-700}. To change this, modify the HEADLINE_VARS field in chaosbench/config.py

      • + +
      +

      Training Strategies#

      In addition, we also provide flags to train the model either autoregressively or directly.

      z-10