Skip to content

Commit

Permalink
change data for interactive visualisation
Browse files Browse the repository at this point in the history
  • Loading branch information
annefou committed Jun 17, 2024
1 parent f10e289 commit 22683c3
Show file tree
Hide file tree
Showing 5 changed files with 76 additions and 4,089 deletions.
4 changes: 2 additions & 2 deletions docs/afterword/resources.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

![Pangeo logo](.././figures/pangeo_name_logo.png)
![Pangeo logo](../images/pangeo_name_logo.png)

**A community platform for Big Data geoscience**

Expand Down Expand Up @@ -47,4 +47,4 @@ _List of some active initiatives. Find more in https://github.com/pangeo-data_.
- [verde](https://github.com/fatiando/verde), _package_, aims at processing spatial data (bathymetry, geophysics surveys, etc) and interpolating it on regular grids (i.e., gridding).
- [xarray-sentinel](https://github.com/bopen/xarray-sentinel): _package_, facilitates access and exploration of the SAR data products of the Copernicus Sentinel-1 satellite mission.
- [xESMF](https://github.com/pangeo-data/xESMF): _package_, a regridding tool suited for non-orthogonal grids. xESMF tries to be simple and intuitive.
- [xMIP](https://github.com/jbusecke/xMIP): _package_, facilitates the cleaning, organization and interactive analysis of Model Intercomparison Projects (MIPs) within the Pangeo software stack.
- [xMIP](https://github.com/jbusecke/xMIP): _package_, facilitates the cleaning, organization and interactive analysis of Model Intercomparison Projects (MIPs) within the Pangeo software stack.
6 changes: 3 additions & 3 deletions docs/pangeo/chunking_introduction.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -962,7 +962,7 @@
"\n",
"If you look carefully to `LTS`, each Data Variable is a `dask.array` with a chunk size of `(15680, 40320)`. So basically accessing one data variable would load arrays of dimensions `(15680, 40320)` into the computer's RAM. You can see this information and more details by clicking the icon as indicated in the image below.\n",
"\n",
"![Dask.array](images/datasize.png)\n",
"![Dask.array](../images/datasize.png)\n",
"\n",
"When you open one or several netCDF files with `open_mdfataset`, by default, the chunks correspond to the entire size of the variable data array read from each file. When you need to analyze large files, a computer's memory may not be sufficient anymore (see in this example, 2.36GiB for one chunk!).\n",
"\n",
Expand All @@ -977,10 +977,10 @@
"__Chunking__ is splitting a dataset into small pieces. \n",
"\n",
"Original dataset is in one piece, \n",
"![Dask.array](images/notchunked.png)\n",
"![Dask.array](../images/notchunked.png)\n",
"\n",
"and we split it into several smaller pieces. \n",
"![Dask.array](images/chunked.png)\n",
"![Dask.array](../images/chunked.png)\n",
"\n",
"We split it into pieces so that we can process our data block by block or __chunk__ by __chunk__.\n",
"\n",
Expand Down
8 changes: 4 additions & 4 deletions docs/pangeo/dask_introduction.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@
"\n",
"When we use chunks with `Xarray`, the real computation is only done when needed or asked for, usually when invoking `compute()` function. Dask generates a **task graph** describing the computations to be done. When using [Dask Distributed](https://distributed.dask.org/en/stable/) a **Scheduler** distributes these tasks across several **Workers**.\n",
"\n",
"![Xarray with dask](images/dask-xarray-explained.png)"
"![Xarray with dask](../images/dask-xarray-explained.png)"
]
},
{
Expand Down Expand Up @@ -579,8 +579,8 @@
"you will just need too look at the html link you have for your jupyterlab, and Dask dashboard port number, as highlighted in the figure below.\n",
"\n",
"\n",
"<img src=\"images/dashboardlink.png\" width=\"50%\">\n",
"<img src=\"images/dasklab.png\" width=\"30%\">\n",
"<img src=\"../images/dashboardlink.png\" width=\"50%\">\n",
"<img src=\"../images/dasklab.png\" width=\"30%\">\n",
"\n",
"Then click the orange icon indicated in the above figure, and type 'your' dashboard link (normally, you just need to replace 'todaka' to 'your username'). \n",
"\n",
Expand All @@ -592,7 +592,7 @@
"You can click several buttons indicated with blue arrows in above figures, then drag and drop to place them as your convenience. \n",
"\n",
"\n",
"<img src=\"images/exampledasklab.png\" width=\"50%\">\n",
"<img src=\"../images/exampledasklab.png\" width=\"50%\">\n",
"\n",
"\n",
"It's really helpfull to understand your computation and how it is distributed."
Expand Down
Loading

0 comments on commit 22683c3

Please sign in to comment.