deploy: c5c20b9

README.html

@@ -169,6 +169,7 @@
 <li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1"><a class="reference internal" href="evaluation.html">Evaluation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="baseline.html">Baseline Models</a></li>
@@ -372,6 +373,7 @@ <h2> Contents </h2>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#getting-started">Getting Started</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#build-your-own-model">Build Your Own Model</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#benchmarking">Benchmarking</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#motivation">Motivation</a></li>
 </ul>
             </nav>
         </div>
@@ -388,10 +390,10 @@ <h1>ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seas
 <p>ChaosBench is a benchmark project to improve long-term forecasting of chaotic systems, in particular subseasonal-to-seasonal (S2S) climate, using ML approaches.</p>
 <p>Homepage 🔗: <a class="reference external" href="https://leap-stc.github.io/ChaosBench">https://leap-stc.github.io/ChaosBench</a></p>
 <p>Paper 📚: <a class="reference external" href="https://arxiv.org/">https://arxiv.org/</a></p>
-<p>Dataset 🤗: <a class="reference external" href="https://huggingface.co/datasets/juannat7/ChaosBench">https://huggingface.co/datasets/juannat7/ChaosBench</a></p>
+<p>Dataset 🤗: <a class="reference external" href="https://huggingface.co/datasets/LEAP/ChaosBench">https://huggingface.co/datasets/LEAP/ChaosBench</a></p>
 <section id="features">
 <h2>Features<a class="headerlink" href="#features" title="Permalink to this heading">#</a></h2>
-<p><img alt="Overview of ChaosBench" src="docs/scheme/chaosbench_scheme.jpg" /></p>
+<p><img alt="Overview of ChaosBench" src="_images/chaosbench_scheme.jpg" /></p>
 <p>1️⃣ <strong>Extended Observations</strong>. Spanning over 45 years (1979 - 2023) of ERA5 reanalysis</p>
 <p>2️⃣ <strong>Diverse Baselines</strong>. Wide selection of physics-based forecasts from leading national agencies in Europe, the UK, America, and Asia</p>
 <p>3️⃣ <strong>Differentiable Physics Metrics</strong>. Introduces two differentiable physics-based metrics to minimize the decay of power spectra at long forecasting horizon (blurriness)</p>
@@ -419,6 +421,13 @@ <h2>Benchmarking<a class="headerlink" href="#benchmarking" title="Permalink to t
 <li><p><a class="reference external" href="https://leap-stc.github.io/ChaosBench/leaderboard.html">Leaderboard</a></p></li>
 </ul>
 </section>
+<section id="motivation">
+<h2>Motivation<a class="headerlink" href="#motivation" title="Permalink to this heading">#</a></h2>
+<p>1️⃣ <strong>Collapse to Climatology</strong>. Performing comparable or worse than climatology renders these state-of-the-art-models operationally unusable
+<img alt="Collapse" src="_images/all_rmse_sota.png" /></p>
+<p>2️⃣ <strong>Blurring Artifact</strong>. Averaged-out forecasts is of little use when one attempts to predict extreme events requiring high-fidelity on the S2S scale (e.g., droughts, hurricanes)
+<img alt="Blurring" src="_images/preds_climax_q700_direct_Task1.png" /></p>
+</section>
 <div class="toctree-wrapper compound">
 </div>
 </section>
@@ -482,6 +491,7 @@ <h2>Benchmarking<a class="headerlink" href="#benchmarking" title="Permalink to t
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#getting-started">Getting Started</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#build-your-own-model">Build Your Own Model</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#benchmarking">Benchmarking</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#motivation">Motivation</a></li>
 </ul>
   </nav></div>
 

_sources/README.md

@@ -7,8 +7,7 @@ Homepage 🔗: https://leap-stc.github.io/ChaosBench
 
 Paper 📚: https://arxiv.org/
 
-Dataset 🤗: https://huggingface.co/datasets/juannat7/ChaosBench 
-
+Dataset 🤗: https://huggingface.co/datasets/LEAP/ChaosBench
 
 ## Features
 
@@ -22,7 +21,6 @@ Dataset 🤗: https://huggingface.co/datasets/juannat7/ChaosBench
 
 4️⃣ __Large-Scale Benchmarking__. Systematic evaluation for state-of-the-art ML-based weather models like PanguWeather, FourcastNetV2, ViT/ClimaX, and Graphcast
 
-
 ## Getting Started
 - [Quickstart](https://leap-stc.github.io/ChaosBench/quickstart.html)
 - [Dataset Overview](https://leap-stc.github.io/ChaosBench/dataset.html)
@@ -35,4 +33,12 @@ Dataset 🤗: https://huggingface.co/datasets/juannat7/ChaosBench
 
 ## Benchmarking
 - [Baseline Models](https://leap-stc.github.io/ChaosBench/baseline.html)
-- [Leaderboard](https://leap-stc.github.io/ChaosBench/leaderboard.html)
+- [Leaderboard](https://leap-stc.github.io/ChaosBench/leaderboard.html)
+
+
+## Motivation
+1️⃣ __Collapse to Climatology__. Performing comparable or worse than climatology renders these state-of-the-art-models operationally unusable
+![Collapse](docs/all_rmse_sota.png)
+
+2️⃣ __Blurring Artifact__. Averaged-out forecasts is of little use when one attempts to predict extreme events requiring high-fidelity on the S2S scale (e.g., droughts, hurricanes)
+![Blurring](docs/preds_climax_q700_direct_Task1.png)

_sources/baseline.md

@@ -23,7 +23,7 @@ We differentiate between physics-based and data-driven models. The former is suc
     - [x] GraphCast
 
 ## Model Checkpoints
-Checkpoints for data-driven models are accessible from [here](https://huggingface.co/datasets/juannat7/ChaosBench/tree/main/logs)
+Checkpoints for data-driven models are accessible from [here](https://huggingface.co/datasets/LEAP/ChaosBench/tree/main/logs)
 
 - Data-driven models are indicated by the `_s2s` suffix (e.g., `unet_s2s`). 
 

_sources/evaluation.md

@@ -22,7 +22,7 @@ __For example__, in our `unet_s2s` baseline model, we can run:
 
 
 ## Accessing Baseline Scores
-You can access the complete scores (in `.csv` format) for data-driven, physics-based models, climatology, and persistence [here](https://huggingface.co/datasets/juannat7/ChaosBench/tree/main/logs). Below is a snippet from `logs/climatology/eval/rmse_climatology.csv`, where each row represents `<METRIC>`, such as `RMSE`, at each future timestep.
+You can access the complete scores (in `.csv` format) for data-driven, physics-based models, climatology, and persistence [here](https://huggingface.co/datasets/LEAP/ChaosBench/tree/main/logs). Below is a snippet from `logs/climatology/eval/rmse_climatology.csv`, where each row represents `<METRIC>`, such as `RMSE`, at each future timestep.
 
 | z-10     | z-50     | z-100    | z-200    | z-300    | ... | w-1000   |
 |----------|----------|----------|----------|----------|-----|----------|

_sources/quickstart.md

@@ -8,11 +8,11 @@ cd ChaosBench
 mkdir data
 ```
 
-**Step 3**: Navigate to `chaosbench/config.py` and change the field `DATA_DIR = /<YOUR_WORKING_DIR>/ChaosBench/data` (_Provide absolute path_)
+**Step 3**: Navigate to `chaosbench/config.py` and change the field `DATA_DIR = ChaosBench/data`
 
 **Step 4**: Initialize the space by running
 ```
-cd /<YOUR_WORKING_DIR>/ChaosBench/data/
+cd ChaosBench/data/
 wget https://huggingface.co/datasets/juannat7/ChaosBench/blob/main/process.sh
 chmod +x process.sh
 ```

_sources/task.md

@@ -14,6 +14,7 @@ __NOTE__: Before training your own model [instructions here](https://leap-stc.gi
 
 - Task 2️⃣: `only_headline: True`. By default, it is going to optimize on {t-850, z-500, q-700}. To change this, modify the `HEADLINE_VARS` field in `chaosbench/config.py` 
 
+# Training Strategies
 In addition, we also provide flags to train the model either __autoregressively__ or __directly__. 
 
 - Autoregressive: Using current output as the next model input. The number of iterative steps is defined in the `n_step: <N_STEP>` field. For our baselines, we set `N_STEP = 5`.

_sources/training.md

@@ -4,11 +4,14 @@
 
 We will outline how one can implement their own data-driven models. Several examples, including ED, FNO, ResNet, and UNet have been provided. 
 
-**Step 1**: Define your model class in `chaosbench/models/<YOUR_MODEL>.py`. At present, we only support models built with `PyTorch`
+### Step 1
+Define your model class in `chaosbench/models/<YOUR_MODEL>.py`
 
-**Step 2**: Initialize your model in `chaosbench/models/model.py` under `__init__` method in `S2SBenchmarkModel` class
+### Step 2
+Initialize your model in `chaosbench/models/model.py` under `S2SBenchmarkModel.__init__`
 
-**Step 3**: Write a configuration file in `chaosbench/configs/<YOUR_MODEL>_s2s.yaml`. We recommend reading the details on the definition of [hyperparameters](https://leap-stc.github.io/ChaosBench/baseline.html) and the different [task]((https://leap-stc.github.io/ChaosBench/task.html) before training. Also change the `model_name: <YOUR_MODEL>_s2s` to ensure  correct pathing
+### Step 3
+Write a configuration file in `chaosbench/configs/<YOUR_MODEL>_s2s.yaml`. Details on the definition of [hyperparameters](https://leap-stc.github.io/ChaosBench/baseline.html) and the different [task](https://leap-stc.github.io/ChaosBench/task.html). Also change the `model_name: <YOUR_MODEL>_s2s` to ensure  correct pathing
 
 - Task 1️⃣ (autoregressive): `only_headline: False ; n_step: <N_STEP>`
 - Task 1️⃣ (direct): `only_headline: False ; n_step: 1 ; lead_time: <LEAD_TIME>`
@@ -17,8 +20,8 @@ We will outline how one can implement their own data-driven models. Several exam
 - Task 2️⃣ (direct): `only_headline: True ; n_step: 1 ; lead_time: <LEAD_TIME>`
 
 
-**Step 4**: Train by running `python train.py --config_filepath chaosbench/configs/<YOUR_MODEL>_s2s.yaml`  
+### Step 4
+Train by running `python train.py --config_filepath chaosbench/configs/<YOUR_MODEL>_s2s.yaml`  
 
-**Step 5**: Done! 
 
-__NOTE__: Remember to replace `<YOUR_MODEL>` with your own model name, e.g., `unet`. Checkpoints and logs would be automatically generated in `logs/<YOUR_MODEL>_s2s/`.
+Remember to replace `<YOUR_MODEL>` with your own model name, e.g., `unet`. Checkpoints and logs would be automatically generated in `logs/<YOUR_MODEL>_s2s/`.

baseline.html

@@ -170,6 +170,7 @@
 <li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1"><a class="reference internal" href="evaluation.html">Evaluation</a></li>
 <li class="toctree-l1 current active"><a class="current reference internal" href="#">Baseline Models</a></li>
@@ -415,7 +416,7 @@ <h2>Model Definition<a class="headerlink" href="#model-definition" title="Permal
 </section>
 <section id="model-checkpoints">
 <h2>Model Checkpoints<a class="headerlink" href="#model-checkpoints" title="Permalink to this heading">#</a></h2>
-<p>Checkpoints for data-driven models are accessible from <a class="reference external" href="https://huggingface.co/datasets/juannat7/ChaosBench/tree/main/logs">here</a></p>
+<p>Checkpoints for data-driven models are accessible from <a class="reference external" href="https://huggingface.co/datasets/LEAP/ChaosBench/tree/main/logs">here</a></p>
 <ul class="simple">
 <li><p>Data-driven models are indicated by the <code class="docutils literal notranslate"><span class="pre">_s2s</span></code> suffix (e.g., <code class="docutils literal notranslate"><span class="pre">unet_s2s</span></code>).</p></li>
 <li><p>The hyperparameter specifications are located in <code class="docutils literal notranslate"><span class="pre">version_xx/lightning_logs/hparams.yaml</span></code>. The hyperparameters encode the following:</p>

dataset.html

@@ -170,6 +170,7 @@
 <li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
 <li class="toctree-l1 current active"><a class="current reference internal" href="#">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1"><a class="reference internal" href="evaluation.html">Evaluation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="baseline.html">Baseline Models</a></li>

evaluation.html

@@ -168,6 +168,7 @@
 <li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1 current active"><a class="current reference internal" href="#">Evaluation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="baseline.html">Baseline Models</a></li>
@@ -402,7 +403,7 @@ <h1>Evaluation<a class="headerlink" href="#evaluation" title="Permalink to this
 </ul>
 <section id="accessing-baseline-scores">
 <h2>Accessing Baseline Scores<a class="headerlink" href="#accessing-baseline-scores" title="Permalink to this heading">#</a></h2>
-<p>You can access the complete scores (in <code class="docutils literal notranslate"><span class="pre">.csv</span></code> format) for data-driven, physics-based models, climatology, and persistence <a class="reference external" href="https://huggingface.co/datasets/juannat7/ChaosBench/tree/main/logs">here</a>. Below is a snippet from <code class="docutils literal notranslate"><span class="pre">logs/climatology/eval/rmse_climatology.csv</span></code>, where each row represents <code class="docutils literal notranslate"><span class="pre">&lt;METRIC&gt;</span></code>, such as <code class="docutils literal notranslate"><span class="pre">RMSE</span></code>, at each future timestep.</p>
+<p>You can access the complete scores (in <code class="docutils literal notranslate"><span class="pre">.csv</span></code> format) for data-driven, physics-based models, climatology, and persistence <a class="reference external" href="https://huggingface.co/datasets/LEAP/ChaosBench/tree/main/logs">here</a>. Below is a snippet from <code class="docutils literal notranslate"><span class="pre">logs/climatology/eval/rmse_climatology.csv</span></code>, where each row represents <code class="docutils literal notranslate"><span class="pre">&lt;METRIC&gt;</span></code>, such as <code class="docutils literal notranslate"><span class="pre">RMSE</span></code>, at each future timestep.</p>
 <table class="table">
 <thead>
 <tr class="row-odd"><th class="head"><p>z-10</p></th>

genindex.html

@@ -167,6 +167,7 @@
 <li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1"><a class="reference internal" href="evaluation.html">Evaluation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="baseline.html">Baseline Models</a></li>

leaderboard.html

@@ -167,6 +167,7 @@
 <li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1"><a class="reference internal" href="evaluation.html">Evaluation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="baseline.html">Baseline Models</a></li>

quickstart.html

@@ -168,6 +168,7 @@
 <li class="toctree-l1 current active"><a class="current reference internal" href="#">Quickstart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1"><a class="reference internal" href="evaluation.html">Evaluation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="baseline.html">Baseline Models</a></li>
@@ -377,9 +378,9 @@ <h1>Quickstart<a class="headerlink" href="#quickstart" title="Permalink to this
 <span class="n">mkdir</span> <span class="n">data</span>
 </pre></div>
 </div>
-<p><strong>Step 3</strong>: Navigate to <code class="docutils literal notranslate"><span class="pre">chaosbench/config.py</span></code> and change the field <code class="docutils literal notranslate"><span class="pre">DATA_DIR</span> <span class="pre">=</span> <span class="pre">/&lt;YOUR_WORKING_DIR&gt;/ChaosBench/data</span></code> (<em>Provide absolute path</em>)</p>
+<p><strong>Step 3</strong>: Navigate to <code class="docutils literal notranslate"><span class="pre">chaosbench/config.py</span></code> and change the field <code class="docutils literal notranslate"><span class="pre">DATA_DIR</span> <span class="pre">=</span> <span class="pre">ChaosBench/data</span></code></p>
 <p><strong>Step 4</strong>: Initialize the space by running</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cd</span> <span class="o">/&lt;</span><span class="n">YOUR_WORKING_DIR</span><span class="o">&gt;/</span><span class="n">ChaosBench</span><span class="o">/</span><span class="n">data</span><span class="o">/</span>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cd</span> <span class="n">ChaosBench</span><span class="o">/</span><span class="n">data</span><span class="o">/</span>
 <span class="n">wget</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">huggingface</span><span class="o">.</span><span class="n">co</span><span class="o">/</span><span class="n">datasets</span><span class="o">/</span><span class="n">juannat7</span><span class="o">/</span><span class="n">ChaosBench</span><span class="o">/</span><span class="n">blob</span><span class="o">/</span><span class="n">main</span><span class="o">/</span><span class="n">process</span><span class="o">.</span><span class="n">sh</span>
 <span class="n">chmod</span> <span class="o">+</span><span class="n">x</span> <span class="n">process</span><span class="o">.</span><span class="n">sh</span>
 </pre></div>

search.html

@@ -169,6 +169,7 @@
 <li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset Information</a></li>
 <li class="toctree-l1"><a class="reference internal" href="task.html">Task Overview</a></li>
+
 <li class="toctree-l1"><a class="reference internal" href="training.html">Training</a></li>
 <li class="toctree-l1"><a class="reference internal" href="evaluation.html">Evaluation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="baseline.html">Baseline Models</a></li>