diff --git a/docs/afterword/resources.md b/docs/afterword/resources.md index 7ff3dee..07f208a 100644 --- a/docs/afterword/resources.md +++ b/docs/afterword/resources.md @@ -1,5 +1,5 @@ -![Pangeo logo](.././figures/pangeo_name_logo.png) +![Pangeo logo](../images/pangeo_name_logo.png) **A community platform for Big Data geoscience** @@ -47,4 +47,4 @@ _List of some active initiatives. Find more in https://github.com/pangeo-data_. - [verde](https://github.com/fatiando/verde), _package_, aims at processing spatial data (bathymetry, geophysics surveys, etc) and interpolating it on regular grids (i.e., gridding). - [xarray-sentinel](https://github.com/bopen/xarray-sentinel): _package_, facilitates access and exploration of the SAR data products of the Copernicus Sentinel-1 satellite mission. - [xESMF](https://github.com/pangeo-data/xESMF): _package_, a regridding tool suited for non-orthogonal grids. xESMF tries to be simple and intuitive. -- [xMIP](https://github.com/jbusecke/xMIP): _package_, facilitates the cleaning, organization and interactive analysis of Model Intercomparison Projects (MIPs) within the Pangeo software stack. \ No newline at end of file +- [xMIP](https://github.com/jbusecke/xMIP): _package_, facilitates the cleaning, organization and interactive analysis of Model Intercomparison Projects (MIPs) within the Pangeo software stack. diff --git a/docs/pangeo/chunking_introduction.ipynb b/docs/pangeo/chunking_introduction.ipynb index 40ddf71..635cd3b 100755 --- a/docs/pangeo/chunking_introduction.ipynb +++ b/docs/pangeo/chunking_introduction.ipynb @@ -962,7 +962,7 @@ "\n", "If you look carefully to `LTS`, each Data Variable is a `dask.array` with a chunk size of `(15680, 40320)`. So basically accessing one data variable would load arrays of dimensions `(15680, 40320)` into the computer's RAM. You can see this information and more details by clicking the icon as indicated in the image below.\n", "\n", - "![Dask.array](images/datasize.png)\n", + "![Dask.array](../images/datasize.png)\n", "\n", "When you open one or several netCDF files with `open_mdfataset`, by default, the chunks correspond to the entire size of the variable data array read from each file. When you need to analyze large files, a computer's memory may not be sufficient anymore (see in this example, 2.36GiB for one chunk!).\n", "\n", @@ -977,10 +977,10 @@ "__Chunking__ is splitting a dataset into small pieces. \n", "\n", "Original dataset is in one piece, \n", - "![Dask.array](images/notchunked.png)\n", + "![Dask.array](../images/notchunked.png)\n", "\n", "and we split it into several smaller pieces. \n", - "![Dask.array](images/chunked.png)\n", + "![Dask.array](../images/chunked.png)\n", "\n", "We split it into pieces so that we can process our data block by block or __chunk__ by __chunk__.\n", "\n", diff --git a/docs/pangeo/dask_introduction.ipynb b/docs/pangeo/dask_introduction.ipynb index c9dcc80..78da98f 100755 --- a/docs/pangeo/dask_introduction.ipynb +++ b/docs/pangeo/dask_introduction.ipynb @@ -159,7 +159,7 @@ "\n", "When we use chunks with `Xarray`, the real computation is only done when needed or asked for, usually when invoking `compute()` function. Dask generates a **task graph** describing the computations to be done. When using [Dask Distributed](https://distributed.dask.org/en/stable/) a **Scheduler** distributes these tasks across several **Workers**.\n", "\n", - "![Xarray with dask](images/dask-xarray-explained.png)" + "![Xarray with dask](../images/dask-xarray-explained.png)" ] }, { @@ -579,8 +579,8 @@ "you will just need too look at the html link you have for your jupyterlab, and Dask dashboard port number, as highlighted in the figure below.\n", "\n", "\n", - "\n", - "\n", + "\n", + "\n", "\n", "Then click the orange icon indicated in the above figure, and type 'your' dashboard link (normally, you just need to replace 'todaka' to 'your username'). \n", "\n", @@ -592,7 +592,7 @@ "You can click several buttons indicated with blue arrows in above figures, then drag and drop to place them as your convenience. \n", "\n", "\n", - "\n", + "\n", "\n", "\n", "It's really helpfull to understand your computation and how it is distributed." diff --git a/docs/pangeo/visualization.ipynb b/docs/pangeo/visualization.ipynb old mode 100755 new mode 100644 index f495d9a..d49a2da --- a/docs/pangeo/visualization.ipynb +++ b/docs/pangeo/visualization.ipynb @@ -17,12 +17,12 @@ "\n", "### Authors\n", "\n", - "- Pier Lorenzo Marasco, Ispra (Italy), [@pl-marasco](https://github.com/pl-marasco)\n", - "- Anne Fouilloux, Simula Research Laboratory (Norway), [@annefou](https://github.com/annefou)\n", + "- Pier Lorenzo Marasco, Provare LTD (UK), [@pl-marasco](https://github.com/pl-marasco)\n", "\n", "### Contributors\n", "\n", - "- Alejandro Coca-Castro, The Alan Turing Institute (United Kingdom), [@acocac](https://github.com/acocac)" + "- Alejandro Coca-Castro, The Alan Turing Institute (United Kingdom), [@acocac](https://github.com/acocac)\n", + "- Anne Fouilloux, Simula Research Laboratory (Norway), [@annefou](https://github.com/annefou)" ] }, { @@ -59,27 +59,27 @@ "## Context\n", "\n", "\n", - "We will be using [HoloViews](https://holoviews.org/), a tool part of the [HoloViz](https://holoviz.org/) ecosystem, with Xarray to visualize the near-surface temperature from CMIP6 CESM2 (monthly dataset).\n", + "We will be using [HoloViews](https://holoviews.org/), a tool part of the [HoloViz](https://holoviz.org/) ecosystem, with Xarray to visualize the Vegetation Condition Index (VCI) {cite:ps}`b-kogan1995-VCI`, a well-established indicator to estimate droughts from remote sensing data.\n", "\n", "### Data\n", "\n", - "We will visualize the same data that have been used in the previous episode.\n", + "We will use data that have been generated in the previous episode.\n", "\n", "If the dataset is not present in the same folder as this Jupyter notebook, it will be downloaded from zenodo using `pooch`, a very handy python-based library to download and cache your data files locally (see further info [here](https://www.fatiando.org/pooch/latest/index.html))." ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "id": "e26e7ca3", "metadata": {}, "outputs": [], "source": [ "import pooch\n", "\n", - "tas_file = pooch.retrieve(\n", - " url=\"https://zenodo.org/record/7181714/files/CMIP6_NCAR_CESM2_historical_amon_gn.nc\",\n", - " known_hash=\"md5:5f86251e5bc5ef9b86a3a86cd06a536b\",\n", + "cgls_file = pooch.retrieve(\n", + " url=\"https://zenodo.org/record/6969999/files/C_GLS_NDVI_20220101_20220701_Lombardia_S3_2_masked.nc\",\n", + " known_hash=\"md5:be3f16913ebbdb4e7af227f971007b22\",\n", " path=f\".\",\n", ")" ] @@ -110,7 +110,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "id": "589df42a", "metadata": {}, "outputs": [], @@ -128,12 +128,12 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "id": "3a4e25c7", "metadata": {}, "outputs": [], "source": [ - "tas_ds = xr.open_dataset(tas_file)" + "cgls_ds = xr.open_dataset(cgls_file)" ] }, { @@ -142,500 +142,18 @@ "metadata": {}, "source": [ ":::{tip}\n", - "If you get an error with the previous command, check the previous episode where the input file some_hash-CMIP6_NCAR_CESM2_historical_amon_gn.nc is downloaded locally and it is in the same directory as your Jupyter Notebook.\n", + "If you get an error with the previous command, check the previous episode where the input file some_hash-C_GLS_NDVI_20220101_20220701_Lombardia_S3_2_masked.netcdf is downloaded locally and it is in the same directory as your Jupyter Notebook.\n", ":::" ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "id": "c471061b", "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
<xarray.Dataset>\n", - "Dimensions: (lat: 192, nbnd: 2, lon: 288, member_id: 1, time: 1980)\n", - "Coordinates:\n", - " * lat (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0\n", - " lat_bnds (lat, nbnd) float32 -90.0 -89.53 -89.53 ... 89.53 89.53 90.0\n", - " * lon (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8\n", - " lon_bnds (lon, nbnd) float32 -0.625 0.625 0.625 ... 358.1 358.1 359.4\n", - " * time (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00\n", - " time_bnds (time, nbnd) object 1850-01-01 00:00:00 ... 2015-01-01 00:00:00\n", - " * member_id (member_id) object 'r1i1p1f1'\n", - "Dimensions without coordinates: nbnd\n", - "Data variables:\n", - " tas (member_id, time, lat, lon) float32 ...\n", - "Attributes: (12/50)\n", - " Conventions: CF-1.7 CMIP-6.2\n", - " activity_id: CMIP\n", - " branch_method: standard\n", - " branch_time_in_child: 674885.0\n", - " branch_time_in_parent: 219000.0\n", - " case_id: 15\n", - " ... ...\n", - " variant_label: r1i1p1f1\n", - " status: 2019-10-25;created;by nhn2@columbia.edu\n", - " netcdf_tracking_ids: hdl:21.14100/d9a7225a-49c3-4470-b7ab-a8180926f839\n", - " version_id: v20190308\n", - " intake_esm_varname: tas\n", - " intake_esm_dataset_key: CMIP.NCAR.CESM2.historical.Amon.gn
\n", - " | asap1_id | \n", - "name1 | \n", - "name1_shr | \n", - "name0 | \n", - "asap0_id | \n", - "name0_shr | \n", - "km2_tot | \n", - "km2_crop | \n", - "km2_range | \n", - "an_crop | \n", - "an_range | \n", - "water_lim | \n", - "geometry | \n", - "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", - "1238 | \n", - "Hordaland | \n", - "Hordaland | \n", - "Norway | \n", - "173 | \n", - "Norway | \n", - "14021 | \n", - "631 | \n", - "81 | \n", - "1 | \n", - "0 | \n", - "0 | \n", - "MULTIPOLYGON (((5.67021 60.69386, 5.67357 60.6... | \n", - "
1 | \n", - "1446 | \n", - "Nord-troendelag | \n", - "Nord-troendelag | \n", - "Norway | \n", - "173 | \n", - "Norway | \n", - "22315 | \n", - "1344 | \n", - "9 | \n", - "1 | \n", - "0 | \n", - "0 | \n", - "MULTIPOLYGON (((11.93731 65.10981, 11.88333 65... | \n", - "
2 | \n", - "1472 | \n", - "Soer-Troendelag | \n", - "Soer-Troendelag | \n", - "Norway | \n", - "173 | \n", - "Norway | \n", - "18441 | \n", - "1313 | \n", - "4 | \n", - "1 | \n", - "0 | \n", - "0 | \n", - "MULTIPOLYGON (((9.74026 63.62652, 9.72871 63.6... | \n", - "
3 | \n", - "1426 | \n", - "Oestfold | \n", - "Oestfold | \n", - "Norway | \n", - "173 | \n", - "Norway | \n", - "4128 | \n", - "1030 | \n", - "0 | \n", - "1 | \n", - "0 | \n", - "0 | \n", - "POLYGON ((11.94171 59.69688, 11.89933 59.69182... | \n", - "
4 | \n", - "1496 | \n", - "Vestfold | \n", - "Vestfold | \n", - "Norway | \n", - "173 | \n", - "Norway | \n", - "2232 | \n", - "623 | \n", - "2 | \n", - "1 | \n", - "0 | \n", - "0 | \n", - "POLYGON ((10.39417 59.66410, 10.42124 59.62203... | \n", - "
<xarray.Dataset>\n", - "Dimensions: (lat: 192, nbnd: 2, lon: 288, member_id: 1, time: 1980)\n", - "Coordinates:\n", - " * lat (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0\n", - " lat_bnds (lat, nbnd) float32 -90.0 -89.53 -89.53 ... 89.53 89.53 90.0\n", - " * lon (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8\n", - " lon_bnds (lon, nbnd) float32 -0.625 0.625 0.625 ... 358.1 358.1 359.4\n", - " * time (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00\n", - " time_bnds (time, nbnd) object 1850-01-01 00:00:00 ... 2015-01-01 00:00:00\n", - " * member_id (member_id) object 'r1i1p1f1'\n", - " spatial_ref int64 0\n", - "Dimensions without coordinates: nbnd\n", - "Data variables:\n", - " tas (member_id, time, lat, lon) float32 ...\n", - "Attributes: (12/50)\n", - " Conventions: CF-1.7 CMIP-6.2\n", - " activity_id: CMIP\n", - " branch_method: standard\n", - " branch_time_in_child: 674885.0\n", - " branch_time_in_parent: 219000.0\n", - " case_id: 15\n", - " ... ...\n", - " variant_label: r1i1p1f1\n", - " status: 2019-10-25;created;by nhn2@columbia.edu\n", - " netcdf_tracking_ids: hdl:21.14100/d9a7225a-49c3-4470-b7ab-a8180926f839\n", - " version_id: v20190308\n", - " intake_esm_varname: tas\n", - " intake_esm_dataset_key: CMIP.NCAR.CESM2.historical.Amon.gn
<xarray.DataArray 'tas' (member_id: 1, time: 1980, lat: 13, lon: 21)>\n", - "array([[[[ nan, 268.18625, 263.21762, ..., nan, nan,\n", - " nan],\n", - " [ nan, 261.4894 , 259.58054, ..., nan, nan,\n", - " nan],\n", - " [270.17227, 260.38303, 257.93915, ..., nan, nan,\n", - " nan],\n", - " ...,\n", - " [ nan, nan, nan, ..., nan, nan,\n", - " nan],\n", - " [ nan, nan, nan, ..., nan, nan,\n", - " nan],\n", - " [ nan, nan, nan, ..., 262.4566 , 262.70773,\n", - " 262.97894]],\n", - "\n", - " [[ nan, 274.94598, 272.1838 , ..., nan, nan,\n", - " nan],\n", - " [ nan, 272.17087, 270.1366 , ..., nan, nan,\n", - " nan],\n", - " [276.09476, 271.16046, 269.08408, ..., nan, nan,\n", - " nan],\n", - "...\n", - " [ nan, nan, nan, ..., nan, nan,\n", - " nan],\n", - " [ nan, nan, nan, ..., nan, nan,\n", - " nan],\n", - " [ nan, nan, nan, ..., 268.94943, 268.97186,\n", - " 269.03262]],\n", - "\n", - " [[ nan, 280.14944, 277.7591 , ..., nan, nan,\n", - " nan],\n", - " [ nan, 276.78476, 274.67453, ..., nan, nan,\n", - " nan],\n", - " [280.1642 , 275.19962, 273.14044, ..., nan, nan,\n", - " nan],\n", - " ...,\n", - " [ nan, nan, nan, ..., nan, nan,\n", - " nan],\n", - " [ nan, nan, nan, ..., nan, nan,\n", - " nan],\n", - " [ nan, nan, nan, ..., 268.04092, 268.38055,\n", - " 268.77368]]]], dtype=float32)\n", - "Coordinates:\n", - " * lat (lat) float64 58.9 59.84 60.79 61.73 ... 68.32 69.27 70.21\n", - " * lon (lon) float64 5.0 6.25 7.5 8.75 10.0 ... 26.25 27.5 28.75 30.0\n", - " * time (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00\n", - " * member_id (member_id) object 'r1i1p1f1'\n", - " spatial_ref int64 0\n", - "Attributes:\n", - " cell_measures: area: areacella\n", - " cell_methods: area: time: mean\n", - " comment: near-surface (usually, 2 meter) air temperature\n", - " description: near-surface (usually, 2 meter) air temperature\n", - " frequency: mon\n", - " id: tas\n", - " long_name: Near-Surface Air Temperature\n", - " mipTable: Amon\n", - " out_name: tas\n", - " prov: Amon ((isd.003))\n", - " realm: atmos\n", - " standard_name: air_temperature\n", - " time: time\n", - " time_label: time-mean\n", - " time_title: Temporal mean\n", - " title: Near-Surface Air Temperature\n", - " type: real\n", - " units: K\n", - " variable_id: tas