diff --git a/README.md b/README.md index 5269eff0..8dbd3c3a 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,15 @@ Learn more about this project [https://www.earthdata.nasa.gov/esds/veda](https://www.earthdata.nasa.gov/esds/veda). + +## Contributing + +This site is rendered using [Quarto](https://quarto.org/), which comes with various developer/contributor resources. + +The gist: download the package from their [Get Started](https://quarto.org/docs/get-started/) page and `quarto build` locally to preview your changes +and open a Pull Request on this repository. + + ## License This project is licensed under **Apache 2**, see the [LICENSE](LICENSE) file for more details. diff --git a/_freeze/site_libs/clipboard/clipboard.min.js b/_freeze/site_libs/clipboard/clipboard.min.js index 41c6a0f7..1103f811 100644 --- a/_freeze/site_libs/clipboard/clipboard.min.js +++ b/_freeze/site_libs/clipboard/clipboard.min.js @@ -1,7 +1,7 @@ /*! - * clipboard.js v2.0.10 + * clipboard.js v2.0.11 * https://clipboardjs.com/ * * Licensed MIT © Zeno Rocha */ -!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1/`​ + +5. Create dataset definitions + +6. The next step is to divide all the data into logical collections. A collection is basically what it sounds like, a collection of data files that share the same properties like the data it's measuring, the periodicity, the spatial region, etc. Examples no2-mean and no2-diff should be two different collections because one measures the mean and the other the diff. no2-monthly and no2-yearly should be different because the periodicity is different. + +7. Once you've logically grouped the datasets into collections, create dataset definitions for each of these collections. The data definition is a JSON file that contains some metadata of the dataset and information on how to discover these datasets in the s3 bucket. An example is shown below: + + lis-global-da-evap.json + + ```json + { + "collection": "lis-global-da-evap", + "title": "Evapotranspiration - LIS 10km Global DA", + "description": "Gridded total evapotranspiration (in kg m-2 s-1) from 10km global LIS with assimilation", + "license": "CC0-1.0", + "is_periodic": true, + "time_density": "day", + "spatial_extent": { + "xmin": -179.95, + "ymin": -59.45, + "xmax": 179.95, + "ymax": 83.55 + }, + "temporal_extent": { + "startdate": "2002-08-02T00:00:00Z", + "enddate": "2021-12-01T00:00:00Z" + }, + "sample_files": [ + "s3://veda-data-store-staging/EIS/COG/LIS_GLOBAL_DA/Evap/LIS_Evap_200208020000.d01.cog.tif" + ], + "discovery_items": [ + { + "discovery": "s3", + "cogify": false, + "upload": false, + "dry_run": false, + "prefix": "EIS/COG/LIS_GLOBAL_DA/Evap/", + "bucket": "veda-data-store-staging", + "filename_regex": "(.*)LIS_Evap_(.*).tif$" + } + ] + } + ``` + +For a detailed description of what each of these fields means, visit [here](https://github.com/NASA-IMPACT/veda-stac-ingestor/blob/documentation/ingestion-process/API_usage.md#field-description). + +3\. Once the dataset definitions are created, send them to the VEDA team for publication. + + + +## 3/3 Dataset Configuration {#sec-dataset-configuration} + +Once you have ingested a dataset into the VEDA backend, you will need to configure the Dashboard. + +### Key Steps + +1. With the new dataset ingested, configure the dashboard to show this dataset using the [VEDA Configuration UI](https://visex.netlify.app/admin/#/collections/dataset). + +2. For more detailed instructions on this tool, visit [documentation within Github](https://github.com/NASA-IMPACT/veda-config/blob/develop/docs/NETLIFY_CMS.md). + +3. After successfully configuring and previewing your dataset through the Configuration UI, your dataset configuration will still needs to be reviewed and confirmed by somebody on the VEDA team. Send an email to [veda@uah.edu](mailto:veda@uah.edu) with information about your dataset and a request for review. + + + +## 3/3 Discovery Configuration {#sec-discovery-configuration} + +By this point, you should have a few things: + +1. A draft of a Discovery that you want to show in VEDA + +2. Necessary datasets identified, and ideally already ingested in to VEDA + +3. Images, csv files, or any other supporting assets prepared + + +🧑‍🏫 We recommend you follow the [video walkthrough](#) on how to setup a virtual environment to facilitate discovery creation. + +### Key Steps + +1. Go to the [veda-config](https://github.com/NASA-IMPACT/veda-config) repo in Github + +2. If using a local environment: + + 1. Familiarize yourself with the [Setup](https://github.com/NASA-IMPACT/veda-config/blob/develop/docs/SETUP.md) and [Configuration](https://github.com/NASA-IMPACT/veda-config/blob/develop/docs/CONFIGURATION.md) sections of the documentation + + 2. Using your local environment, create a branch for your Discovery + + 3. Following the guidelines outlined in the [Content](https://github.com/NASA-IMPACT/veda-config/blob/develop/docs/CONTENT.md) section of the Github documentation, create your discovery mdx file + + 4. Add relevant files and assets as needed + + 5. Push your branch and create a pull request in Github + +3. If configuring through Github + + 1. Create a new branch for the discovery + + ![Branching on GitHub](https://res.cloudinary.com/almanac/image/upload/v1678300216/workspace_portal_uploads/hw7elccu5sil1sy2qu4c.png) + + Following the guidelines outlined in the [Content](https://github.com/NASA-IMPACT/veda-config/blob/develop/docs/CONTENT.md) section of the Github documentation, create your discovery mdx file + + 2. Add relevant files and assets as needed + + 3. Commit your changes and open a Pull Request + + +4. Once the pull request is created, you will be able to see a preview of the Discovery in a Netlify box under the Conversation tab of the pull request + + + ![Netlify preview in GitHub Pull Request](https://res.cloudinary.com/almanac/image/upload/v1677706325/workspace_portal_uploads/saevbqahnjlybjiftqfr.png) + + 🍀 You don't have to fully finish your Discovery all in one go. Every time you make a commit the preview will be regenerated with your changes (takes about 3 minutes). + +5. Once you feel good about the Discovery, add the VEDA team as reviewers to your pull request + + 1. If you know who you want to review, add them + + 2. Otherwise, here as some good GitHub handles to start with: @hanbyul-here, @danielfdsilva, @aboydnw + +6. Paste a comment in the pull request with any additional information, such as any goal dates for publishing this discovery or any outstanding questions you have + +7. Once the pull request is merged, the files will still need to be pushed to production. Coordinate with Anthony Boyd on this production push + + + +## Video Walkthrough {#sec-video-walkthrough} + +### Setting up github codespaces + +Codespaces will allow you to have a development environment in the cloud without the need to setup anything on your local machine. [VIDEO](https://drive.google.com/file/d/1u2hkokW3ZDmrjYNkg10OgWU0-nNtHpJ6/view) + +### Creating a discovery + +Walkthrough of how to use github codespaces to create a discovery. From creating the needed files to the Pull Request that will eventually get the content published. [VIDEO](https://drive.google.com/file/d/1Jkbt2csXntPPe8G5TBGic9UYZsj2rgW3/view) + + + +## Appendix - Useful Links + +* [Dataset ingest in almanac](https://almanac.io/handbook/veda-product-development-documentation-I3UwKo/data-ingestion-guide-JuxFFfPT0WlGJJ1MWjE0tXrIFq2bppkv) +* [Data processing from EIS](https://github.com/Earth-Information-System/veda-data-processing) +* [VEDA data pipelines in github](https://github.com/NASA-IMPACT/veda-data-pipelines) +* [VEDA config in github](https://github.com/NASA-IMPACT/veda-config) +* [Alexey’s notes on helpful tips](https://docs.google.com/document/d/13go47lheeIU2kQqoZo4DVwLWJBEQp25S61vQz-a4T9A/edit) \ No newline at end of file diff --git a/contributing/docs-and-notebooks.qmd b/contributing/docs-and-notebooks.qmd new file mode 100644 index 00000000..b50c7916 --- /dev/null +++ b/contributing/docs-and-notebooks.qmd @@ -0,0 +1,68 @@ +--- +title: Docs and example notebooks +--- + +Contribution to VEDA's documentation is always welcome - just open a [Pull Request on the veda-docs repository](https://github.com/NASA-IMPACT/veda-docs). + +Please note that this documentation site is rendered using [Quarto](https://quarto.org/), which adds a small set of configuration options on top of vanilla Markdown and Jupyter Notebooks. + + +## Notebook Author Guidelines + +There are two template notebooks in this directory titled: `template-using-the-raster-api.ipynb` and `template-accessing-the-data-directly.ipynb` that you can use as a starting place. Alternatively you can pull specific cells from that notebook into your own. + + +### Style + +- Each code cell should come after a markdown cell with some explanatory text. This is preferred over comments in the code cells. +- The max header should be `##`. +- Only include imports that are needed for the notebook to run. +- We don't enforce any formatting, but periodically run black on all the notebooks. If you would like to run black yourself do `pip install black[jupyter]` and then `black`. + + +### Rendering information + +The first cell in every notebook is a raw cell that contains the following metadata for rendering. + +``` +--- +title: Short title +description: One sentence description +author: Author Name +date: May 2, 2023 +execute: + freeze: true +--- +``` + +We store evaluted notebooks in this repository. So before you commit your notebook, you should restart your kernel and run all cells in order. + + +### Standard sections + +To give the notebooks a standard look and feel we typically include the following sections: + +- **Run this Notebook**: The section explains how to run the notebook locally, on VEDA JupyterHub or on [mybinder](https://mybinder.org/). There are several examples of what this section can look like in the template notebooks. +- **Approach**: List a few steps that outline the approach you be taking in this notebook. +- **About the data**: Optional description of the datatset +- **Declare your collection of interest**: This section reiterates how you can discover which collections are available. You can copy the example of this section from one of the template notebooks. + +From then on the standard sections diverge depending on whether the notebook access the data directly or uses the raster API. Check the template notebooks for some ideas of common patterns. + + +### Using complex geometries + +If you are defining the AOI using a bounding box, you can include it in the text of the notebook, but for more complex geometries we prefer that the notebook access the geometry directly from a canonical source. You can check the template notebooks for exmples of this. If the complex geometry is not available online the VEDA team can help get it up in a public s3 bucket. + + +### Accessing the data + +To present consistent best practices, we always access data via the STAC API. Often we use `stackstac` for this. + + +### Generate "Launch in VEDA JupyterHub" link + +We use [`nbgitpuller`](https://hub.jupyter.org/nbgitpuller/) links to open the VEDA JupyterHub with a particular notebook pulled in. These links have the form: `https://nasa-veda.2i2c.cloud/hub/user-redirect/git-pull?repo=https://github.com/NASA-IMPACT/veda-docs&urlpath=lab/tree/veda-docs/example-notebooks/open-and-plot.ipynb&branch=main` + +If you are writing a notebook and want to share it with others you can generate your own `nbgitpuller` link using this +[link generator](https://hub.jupyter.org/nbgitpuller/link?hub=https://nasa-veda.2i2c.cloud&repo=https://github.com/NASA-impact/veda-docs&branch=main&app=jupyterlab). \ No newline at end of file diff --git a/example-notebooks/index.qmd b/example-notebooks/index.qmd index 84ebe933..3b4144a1 100644 --- a/example-notebooks/index.qmd +++ b/example-notebooks/index.qmd @@ -37,62 +37,6 @@ jupyter lab If the notebook needs access to protected data on S3, you will need to specifically get access. Please request access by emailing aimee@developmentseed.org or alexandra@developmentseed.org and providing your affiliation, interest in or expected use of the dataset and an AWS IAM role or user Amazon Resource Name (ARN). -## Contributing to VEDA - -Contribution to VEDA's documentation is always welcome. You can add new notebooks by making a [Pull Request on the veda-docs repository](https://github.com/NASA-IMPACT/veda-docs/compare) - -There are two template notebooks in this directory titled: `template-using-the-raster-api.ipynb` and `template-accessing-the-data-directly.ipynb` that you can use as a starting place. Alternatively you can pull specific cells from that notebook into your own. - -### Style - -- Each code cell should come after a markdown cell with some explanatory text. This is preferred over comments in the code cells. -- The max header should be `##`. -- Only include imports that are needed for the notebook to run. -- We don't enforce any formatting, but periodically run black on all the notebooks. If you would like to run black yourself do `pip install black[jupyter]` and then `black`. - -### Rendering information - -The first cell in every notebook is a raw cell that contains the following metadata for rendering. - -``` ---- -title: Short title -description: One sentence description -author: Author Name -date: May 2, 2023 -execute: - freeze: true ---- -``` - -We store evaluted notebooks in this repository. So before you commit your notebook, you should restart your kernel and run all cells in order. - -### Standard sections - -To give the notebooks a standard look and feel we typically include the following sections: - -- **Run this Notebook**: The section explains how to run the notebook locally, on VEDA JupyterHub or on [mybinder](https://mybinder.org/). There are several examples of what this section can look like in the template notebooks. -- **Approach**: List a few steps that outline the approach you be taking in this notebook. -- **About the data**: Optional description of the datatset -- **Declare your collection of interest**: This section reiterates how you can discover which collections are available. You can copy the example of this section from one of the template notebooks. - -From then on the standard sections diverge depending on whether the notebook access the data directly or uses the raster API. Check the template notebooks for some ideas of common patterns. - - -### Using complex geometries - -If you are defining the AOI using a bounding box, you can include it in the text of the notebook, but for more complex geometries we prefer that the notebook access the geometry directly from a canonical source. You can check the template notebooks for exmples of this. If the complex geometry is not available online the VEDA team can help get it up in a public s3 bucket. - - -### Accessing the data - -To present consistent best practices, we always access data via the STAC API. Often we use `stackstac` for this. - -### Generate "Launch in VEDA JupyterHub" link - -We use [`nbgitpuller`](https://hub.jupyter.org/nbgitpuller/) links to open the VEDA JupyterHub with a particular notebook pulled in. These links have the form: `https://nasa-veda.2i2c.cloud/hub/user-redirect/git-pull?repo=https://github.com/NASA-IMPACT/veda-docs&urlpath=lab/tree/veda-docs/example-notebooks/open-and-plot.ipynb&branch=main` - -If you are writing a notebook and want to share it with others you can generate your own `nbgitpuller` link using this -[link generator](https://hub.jupyter.org/nbgitpuller/link?hub=https://nasa-veda.2i2c.cloud&repo=https://github.com/NASA-impact/veda-docs&branch=main&app=jupyterlab). - +## How to contribute +Please refer to the [notebook style guide in these docs](../contributing/docs-and-notebooks.qmd). \ No newline at end of file diff --git a/external-resources.qmd b/external-resources.qmd index 9ec68e1a..8e62bbd3 100644 --- a/external-resources.qmd +++ b/external-resources.qmd @@ -1,5 +1,5 @@ --- -title: "Geospatial Cloud Computing and Analysis" +title: "External resources" subtitle: "Learn more about tools and practice open source geospatial data science - a list of external resources to get you started" --- diff --git a/index.qmd b/index.qmd index 6948b9d6..d7b44e5c 100644 --- a/index.qmd +++ b/index.qmd @@ -14,7 +14,7 @@ For data visualization and exploration in a graphical user interface, visit the ## Interactive and Collaborative Data Science Environments -VEDA promotes the use of JupyterHub environments for interactive data science. Selected user groups can get access to a JupyterHub service provided by VEDA. Please find instructions for requesting access on the [VEDA JupyterHub Access page](./veda-jh-access.qmd). +VEDA promotes the use of JupyterHub environments for interactive data science. Users affiliated with VEDA can get access to a dedicated JupyterHub service. Please find instructions for requesting access on the [VEDA JupyterHub Access page](./veda-jh-access.qmd). If you are a scientist affiliated with NASA projects such as [EIS](https://eis.smce.nasa.gov/about.html) and [MAAP](https://ops.maap-project.org), you can also keep using the resources provided by these projects. diff --git a/veda-apis.qmd b/veda-apis.qmd index 018503de..04bfb22e 100644 --- a/veda-apis.qmd +++ b/veda-apis.qmd @@ -1,4 +1,6 @@ -# VEDA APIs {.unnumbered} +--- +title: VEDA APIs +--- The VEDA API source code is currently stored in a private repository: [veda-backend](https://github.com/NASA-IMPACT/veda-backend), but is based off [eoAPI](https://github.com/developmentseed/eoAPI). diff --git a/veda-jh-access.qmd b/veda-jh-access.qmd index b2941fa6..9a1f577e 100644 --- a/veda-jh-access.qmd +++ b/veda-jh-access.qmd @@ -1,15 +1,18 @@ -# VEDA Analytics Platform and JupyterHub {#veda-jh-access} +--- +title: JupyterHub access +--- -Explore VEDA datasets and stories using a JupyterHub notebook environment on [https://nasa-veda.2i2c.cloud/](https://nasa-veda.2i2c.cloud/) +JupyterHub enables you to analyze massive archives of Earth science data in the cloud in an interactive environment that alleviates the complexities of managing compute resources (virtual machines, roles and permissions, etc). +Users affiliated with the VEDA project can get access to a dedicated JupyterHub environment, provided in collaboration with [2i2c](https://2i2c.org/): [nasa-veda.2i2c.cloud](https://nasa-veda.2i2c.cloud/) -## Getting JupyterHub Access -Access to the VEDA notebook environment is currently on an as-need basis. To gain access follow these steps: +## Getting access to VEDA's JupyterHub environment -* Make sure you have a [Github Account](https://docs.github.com/en/get-started/signing-up-for-github/signing-up-for-a-new-github-account). Take note of your [Github username](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/remembering-your-github-username-or-email) +Access to the VEDA notebook environment is currently on an as-need basis. If you are a user afficiliated with VEDA, you can gain access by following these steps: -* Send an email to the veda team (`veda@uah.edu`) asking for access to the VEDA notebook environment. Please include your Github username. They will invite you through Github to join the [VEDA Analytics Github Team](https://github.com/veda-analytics-access/). Please watch your email for the invite. +* Make sure you have a [Github Account](https://docs.github.com/en/get-started/signing-up-for-github/signing-up-for-a-new-github-account). Take note of your [Github username](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/remembering-your-github-username-or-email) -* Once you accept the invitation you should be able to go to [https://nasa-veda.2i2c.cloud/](https://nasa-veda.2i2c.cloud/) and login via your Github credentials. +* Send an email to the VEDA team ([veda@uah.edu](mailto:veda@uah.edu)) asking for access to the VEDA notebook environment. Please include your Github username. They will invite you through Github to join the [VEDA Analytics Github Team](https://github.com/veda-analytics-access/). Please watch your email for the invite. +* Once you accepted the invitation, you should be able to go to [https://nasa-veda.2i2c.cloud/](https://nasa-veda.2i2c.cloud/) and login via your Github credentials. \ No newline at end of file