From f1a395a1ea2b35168806961a3d385ce3edea0fe9 Mon Sep 17 00:00:00 2001 From: Marjorie Lucas <47004511+marjo-luc@users.noreply.github.com> Date: Wed, 17 Jan 2024 07:49:41 -0800 Subject: [PATCH] Develop (#82) * Feature/update maap help tour (#61) updated maap help jupyter extension to version 1.0.0 * Bugs/presigned url (#62) Main fix for the presigned url bug was changing 'maapsec/environment' to 'jupyter-server-extension/getConfig' There was another bug caused by a race condition that was causing users to not be able to get the presigned url about every 10th time. What fixed this bug is calling updateKeycloakToken upon the launch of the extension and also trying to call updateKeycloakToken again if getting the keycloak user information fails to get I confirmed this bug was fixed by trying to get the presigned url 25 times and it worked Successful build in vanilla here: https://repo.dit.maap-project.org/root/maap-workspaces/-/pipelines/3389 * Added memory extension (#59) added changes from cluttered commit branch * hysds base-image [skip ci] (#65) * Feature/libmamba solver (#64) * upgraded maap help jupyter extension version to 1.0.0 * reverted back changes * changed r docker to use libmamba solver * changed mamba install to conda * converted all images use libmamba solver like R image * added lang tz and debian frontend commands to vanilla docker image * copied pangeo dockerfile into vanilla * first line of packages back to vanilla versions * updated package versions for vanilla to be same as pangeo * updated isce2 package versions to match vanilla * updating continuumio/miniconda3 for rgedi and making other 3 top lines consistent * removed changes from rgedi * Update stac_ipyleaflet to 0.3.5 for Pangeo (#66) * Updated pinned version of stac_ipyleaflet to 0.3.5 for pangeo jupyter workspace. * Added environment variables to Pangeo environment yml. * Updated entrypoint bash script so the the base environment is activated before launching jupyter so that environment variables are accessible within jupyter. --------- Co-authored-by: Alex * Remove hysds base_image [skip-ci] (#68) * Bugs/quick fix libmamba (#69) * removed libmamba sovler from conda env update * added solver to environment.yml * added conda libmamba solver version * added back in solvers and pinned version for conda-libmamba-solver * added back in space * Bugs/presigned url (#70) * changed dit url to test * changed dit back and changed ops to be maap-ops-workspace instead of dataset * Bugs/git tour link (#71) * manual build for change to maap help * updated npm version of maap help * updated custom ext versions (#72) * Feature/update maap py (#73) upgraded all maap py versions to 3.1.2 * bumped jupyter server extension (#74) * update server ext * bumped server ext * Bump maap-py to v3.1.3 (#75) * Update Dockerfile * Bugs/libmamba default (#77) * added conda config setting libmamba solver to entrypoint.sh * updated maap help to version 1.0.1 * converted back to 1.0.0 * trying to create .condarc file * removed touch because might work without it * upgraded conda libmamba solver * undid change because forgot I had PR open for this branch * added --system tag and removed conda config from entrypoint * added the system tag to all base image dockerfiles * Update entrypoint.sh (#78) * Bugs/double submit null username (#79) * updated maap help to version 1.0.1 * converted back to 1.0.0 * upgraded dps version to add feature disabling submit button/ saying last submitted info and fixing the null username bug * Features/update workspace dependencies (#80) * Updated how conda packages installs are handled at the base image layer for Pangeo to use an environment.yml file like we do for its respective jupyterlab image. * Updated to miniconda 23.5.2-0 for the pangeo base image. * Updated list of packages for the base and jupyter images for the Pangeo Workspace. * updated maap help to version 1.0.1 * converted back to 1.0.0 * Clean up of pangeo base environment.yml * Reverted the miniconda version of base pangeo image to 22.11.1. * Updated jupyterlab3 pangeo docker conda solver parameter * Modified base pangeo image conda env update solver parameter spec * explicility installing libarchive fron conda-forge to test resolution of issue with libmamba. * Testing base pangeo conda install parameter updates. * Test builds using new version of miniconda. * Test another build configuration on gitlab using new pinned version of conda-libmamba-solver * Bumped miniconda version to 23.10.0-1 and removed conda-libmamba-solver since libmamba is the default solver in this new version of miniconda. * Updated Pangeo base image and environment yml so that packages are installed in a new environment named with the workspace type to resolve issues with libarchive/libmamba and the installation of packages. * Updated Pangeo jupyterlab dockerfile and environment yml to use new custom environment instead of base environment. * Moved a couple jupyter python packages from dockerfile to the environment yml. * Updated entrypoint to test new environment variable that stores the type of workspace being run to ensure jupyter can be launched appropriately. * Patch issue with ARG values not carrying over to downstream build stages in docker file. Also replaced use of BUILD_IMAGE_TYPE with ENV var, "WORKSPACE_TYPE". * Pinned jupyterlab package to 3.6.1 to fix build issue with jupyterlab_plotly. * Pinned additional packages to assist with resolving jupyterlab build issues. * Commented line "conda init" in entrypoint.sh. * Reverted commenting out "conda init" in entrypoint that was previously done for testing. * Testing changes "~/.bash_profile" to properly set the default conda environment for terminals. * Updated path variable to point to our custom conda environment bin instead of the conda base env bin path. * Updated how the default environment is activated for terminals. * removed jupyter-server from envirionment yml as the jupyterlab installation takes care of installing the jupyter_server package already. * Added Makefile which contains commonly run recipes when modifying workspaces locally and needing to test building them. * Updated notebook lib path and jupyter server lib path to point to the appropriate custom conda environment. * current attempt at fixing r workspace issues, building for base image but not for jupyterlab3 * r image building successfully locally * added conda lists * Added perl package to r base environment.yml to resolve issue with r-tinytex not installing. * Added string replacement commands to remove conda env related commands from .bash_profile that are no longer needed. * Updated jupyterlab3's dockerfile so that it modifies the "jupyterlab3/shared/environment.yml" by doing a string replacement on the workspace environment name for workspaces that don't have a custom environment.yml. * r image building locally * pinned versions * updated pinned versions * changed version for r-dplyr * new botocore version * added aiobotocore dependency * removed aiobotocore from install * removed botocore install * added half of the packages to see if I can get rid of the botocore warning * removed some more environment.yml packages because still getting botocore error * removed even more packages because still getting botocore warnining * removed more packages because still getting botocore warning * Added conda list to dockerfiles for testing on ADE. * added maap py into pip install from source * updated git install from source * added v to version tag for maappy * added a bunch of conda lists trying to debug * installing maap py to root * corrected installing into root bug * added run pwd line to try to debug * tried setting the working directory to root before git clone * tried adding another pwd command * removed . from git clone * set working directory to maap py * added back in . for git clone * Updated packages for base and jupyterlab reported by data team that are needed in Python based workspaces. * Updated the folder parameter for where maap-py is installed to hopefully resolve maap-py installation issues. * Moved maap py conf environment variable to base environment yml. * commented out some of the code trying to test maap py again * commented back in code before adding in Anils code that worked * cleaning up jupyterlab3 dockerfile so is consistent * removed extra ; * removed pip install of maap py * updated botocore version in base environment.yml * added binary library files to r install * Removed commented ENV command in pangeo dockerfile. * Updated miniconda version for base vanilla image. * Updated base vanilla image to use an environment yml to store it's configuration environment. Vanilla base image now relies on a custom vanilla conda environment. * Updated jupyterlab shared environment yml so that 1) missing packages needed by data team are included, 2) each workspace uses a custom conda environment, and 3) updated entrypoint so that .bash_profile is cleaned up. * added r-car * pinned r-car version and removed botocore3 * added maap py to environment.yml to avoid botocore dependency errors * Added help documentation to Makefile along with other minor improvements. * commented out git clone for r image * Pinned pangeo packages and removed duplicates. * Pinned packages for vanilla and shared workspace environment ymls. Added missing jupyter package to shared environment yml. * Bumped stac_ipyleaflet to version 0.3.6 for pangeo workspace. * created isce3 environment.yml and pinned r binary library versions * fixed name of fsspec * added isce3 to the jupyterlab3 dockerfile * removed dependencies from isce3 environment.yml * added back in dependencies and removed environment.yml for isce3 in jupyter image * removed packages specific to isce3 * added in 3 packages but isce3 to see if that is the problem * added back in isce3 * removed all vanilla packages and just trying to install base isce3 packages * pinned python version * attempting to install r packages via script * pinned versions for fsspec, hvplot, isce3, xarray * added and modified r scripts and removed awscli trying to fix maappy dependency error * trying to add awscli after maappy install * capitalized Fgmutils * added packages to environment.yml * changed maappy branch to bugs/dependency issues * upgraded awscli version * unpinned versions for vanilla environment.yml * removed all r packages except python and r * unpinned python version * changed back to v3.1.3 tag for maap-py * downgraded awscli to 2.14.1 * upgraded awscali to 2.14.2 * removed pinnection vertsions for cryptography and prompt-toolkit * r image should be building now, doesnt include Fgmutils * unpinned versions to fix maap-py bug in vanilla workspace * pinned vanilla versions and changed to dependecies branch for pangeo and isce3 * changed branch for maappy to version with ipython pinned * pinned botocore version and added conda list * moved conda list command and just pushed maappy pinning boto3=1.33.13 * removed botocore from r/environment.yml * changed to correctly use repos and removed print statements * fixed but for Fgmutils install * pinned package versions that were dependency of Fgmutils * added r channel to r environment.yml * downgraded version of r-devemf * downgraded r and r-base to 4.2.3 to be compatible with r-devemf * changed to r version 4.2 * downgraded r essentials to 4.2 * Added geocube to vanilla base image which was added to pangeo but accidentally omitted from vanilla. * Added additional packages requested by user group for python based workspaces. * Removed pins for vanilla and pangeo base image to test new image builds on GitLab with additional packages. * Updated pinned versions for vanilla base image as well as shared environment yml for jupyterlab. Also bumped jupyterlab to 3.6.3 and added missing package, rio-tiler. * Added pinned version information for pangeo base and jupyterlab images. Also modified version information for jupyterlab to 3.6.3 to resolve version descrepancy with pangeo-notebook package. * Removed pin to test installation of the morecantile package. * Fixed typo of morecantile package in pangeo base image. * Repinned morecantile package for pangeo and vanilla base images. * removed print statements and conda lists * added conda config lines just for testing * added descriptions necessary to use install2.r with GPL license * removed lines making conda forge default because that should be a different PR * updated all maap-py branches to be new 3.1.4 release --------- Co-authored-by: Anil Natha * Feature/conda forge default (#81) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * fixed typo --------- Co-authored-by: Grace Llewellyn <114033465+grallewellyn@users.noreply.github.com> Co-authored-by: bsatoriu Co-authored-by: Anil Natha Co-authored-by: Alex Co-authored-by: Sujen Shah Co-authored-by: Anil Natha --- Makefile | 56 ++++++ base_images/isce2/docker/Dockerfile | 2 +- base_images/isce3/docker/Dockerfile | 35 ++-- base_images/isce3/environment.yml | 51 +++++ base_images/pangeo/docker/Dockerfile | 28 +-- base_images/pangeo/environment.yml | 96 +++++++++ base_images/r/docker/Dockerfile | 68 +++++-- base_images/r/environment.yml | 108 ++++++++++ base_images/r/scripts/install2.r | 187 ++++++++++++++++++ .../r/scripts/install_cran_packages_r.sh | 22 +++ base_images/rgedi/docker/Dockerfile | 2 +- base_images/vanilla/docker/Dockerfile | 28 +-- base_images/vanilla/environment.yml | 51 +++++ jupyterlab3/docker/Dockerfile | 49 +++-- jupyterlab3/entrypoint.sh | 24 ++- jupyterlab3/isce3/environment.yml | 16 -- jupyterlab3/pangeo/environment.yml | 35 ++-- jupyterlab3/shared/environment.yml | 21 +- 18 files changed, 756 insertions(+), 123 deletions(-) create mode 100644 Makefile create mode 100644 base_images/isce3/environment.yml create mode 100644 base_images/pangeo/environment.yml create mode 100644 base_images/r/environment.yml create mode 100644 base_images/r/scripts/install2.r create mode 100755 base_images/r/scripts/install_cran_packages_r.sh create mode 100644 base_images/vanilla/environment.yml delete mode 100644 jupyterlab3/isce3/environment.yml diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..917e8f49 --- /dev/null +++ b/Makefile @@ -0,0 +1,56 @@ +export IMAGE_NAME ?= vanilla +export PROJECT_PREFIX = maap_ +export BASE_IMAGE_NAME = "${PROJECT_PREFIX}base_${IMAGE_NAME}" +export JUPYTER_IMAGE_NAME = "${PROJECT_PREFIX}jupyter_${IMAGE_NAME}" +export RUN_OPTIONS = + +build-images: build-base-image build-jupyter-image + +build-images-no-cache: build-base-image-no-cache build-jupyter-image-no-cache + +build-base-image: ## Build the base image for the 'vanilla' workspace (default), or the workspace specified by setting the argument, IMAGE_NAME= + @cd base_images/${IMAGE_NAME}; \ + pwd; \ + docker buildx build --platform linux/amd64 --progress=plain $(RUN_OPTIONS) -t ${BASE_IMAGE_NAME} -f docker/Dockerfile . + +build-base-image-no-cache: RUN_OPTIONS = "--no-cache" +build-base-image-no-cache: build-base-image + +build-jupyter-image: ## Build the jupyter image for the 'vanilla' workspace (default), or the workspace specified by setting the argument, IMAGE_NAME= + @cd jupyterlab3; \ + pwd; \ + docker buildx build --platform linux/amd64 --progress=plain $(RUN_OPTIONS) -t ${JUPYTER_IMAGE_NAME} --build-arg BASE_IMAGE_TYPE=${IMAGE_NAME} --build-arg BASE_IMAGE=${BASE_IMAGE_NAME} -f docker/Dockerfile . + +build-jupyter-image-no-cache: RUN_OPTIONS = "--no-cache" +build-jupyter-image-no-cache: build-jupyter-image + +delete-image: delete-base-image delete-jupyter-image ## Delete the base and jupyter image for the 'vanilla' workspace (default), or the workspace specified by setting the argument, IMAGE_NAME= + +delete-base-image: ## Delete the base image for the 'vanilla' workspace (default), or the workspace specified by setting the argument, IMAGE_NAME= + docker image rm ${BASE_IMAGE_NAME} + +delete-jupyter-image: ## Delete the jupyter image for the 'vanilla' workspace (default), or the workspace specified by setting the argument, IMAGE_NAME= + docker image rm ${JUPYTER_IMAGE_NAME} + +open-jupyter: ## Open a browser window to access an already running instance of a 'vanilla' jupyter image, or the workspace specified by setting the argument, IMAGE_NAME= + open http://localhost:3100/ + +start-jupyter: ## Start the jupyter image for the 'vanilla' workspace (default), or the workspace specified by setting the argument, IMAGE_NAME= + docker run -p 3100:3100 ${JUPYTER_IMAGE_NAME} + +# ---------------------------------------------------------------------------- +# Self-Documented Makefile +# ref: http://marmelab.com/blog/2016/02/29/auto-documented-makefile.html +# ---------------------------------------------------------------------------- +help: ## (DEFAULT) This help information + @echo ==================================================================== + @grep -E '^## .*$$' \ + $(MAKEFILE_LIST) \ + | awk 'BEGIN { FS="## " }; {printf "\033[33m%-20s\033[0m \n", $$2}' + @echo + @grep -E '^[0-9a-zA-Z_-]+:.*?## .*$$' \ + $(MAKEFILE_LIST) \ + | awk 'BEGIN { FS=":.*?## " }; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' \ +# | sort +.PHONY: help +.DEFAULT_GOAL := help \ No newline at end of file diff --git a/base_images/isce2/docker/Dockerfile b/base_images/isce2/docker/Dockerfile index 7f345dbe..5d12e233 100644 --- a/base_images/isce2/docker/Dockerfile +++ b/base_images/isce2/docker/Dockerfile @@ -32,7 +32,7 @@ RUN sed -i -e 's/\/root/\/projects/g' /etc/passwd RUN conda install -y -c conda-forge pyOpenSSL=23.2.0 && \ conda install -y -n base conda-libmamba-solver=23.7.0 && \ - conda config --set solver libmamba + conda config --system --set solver libmamba # need to uninstall jupyter_server_terminals as it conflicts with Jupyterlab 3.4.x. Doesn't seem to break anything # but can get rid of if/when we upgrade jlab diff --git a/base_images/isce3/docker/Dockerfile b/base_images/isce3/docker/Dockerfile index 0d6e7f49..07f08148 100644 --- a/base_images/isce3/docker/Dockerfile +++ b/base_images/isce3/docker/Dockerfile @@ -1,32 +1,31 @@ -FROM continuumio/miniconda3:22.11.1 +FROM continuumio/miniconda3:23.10.0-1 ENV LANG en_US.UTF-8 ENV TZ US/Pacific ARG DEBIAN_FRONTEND=noninteractive -# install maap-py library -ENV MAAP_CONF='/maap-py/' -RUN git clone --single-branch --branch v3.1.3 https://github.com/MAAP-Project/maap-py.git \ - && cd maap-py \ - && pip install -e . - -RUN set -ex \ - && apt-get update +#RUN set -ex \ +# && apt-get update RUN mkdir /projects WORKDIR /projects RUN sed -i -e 's/\/root/\/projects/g' /etc/passwd -RUN conda install -y -c conda-forge pyOpenSSL=23.2.0 && \ - conda install -y -n base conda-libmamba-solver=23.7.0 && \ - conda config --set solver libmamba +COPY ./environment.yml /tmp +RUN conda env create -y -f "/tmp/environment.yml" \ + && find /opt/conda/ -follow -type f -name '*.a' -delete \ + && find /opt/conda/ -follow -type f -name '*.js.map' -delete \ + && /opt/conda/bin/conda clean -afy + +# Set the conda environment for all subsequent docker RUN commands +# When maap-py is updated and installed via our environment.yml, +# we won't need this line unless other envrionment specific commands are added +SHELL ["conda", "run", "-n", "isce3", "/bin/bash", "-c"] -RUN conda install -y --solver=libmamba -c conda-forge isce3=0.12.0 xarray=2023.4.2 \ - hvplot=0.8.3 fsspec=2023.5.0 scikit-learn=1.2.2 && \ - find /opt/conda/ -follow -type f -name '*.a' -delete && \ - find /opt/conda/ -follow -type f -name '*.js.map' -delete && \ - /opt/conda/bin/conda clean -afy +RUN mkdir /maap-py \ + && git clone --single-branch --branch v3.1.4 https://github.com/MAAP-Project/maap-py.git /maap-py/ \ + && pip install -e /maap-py/ -RUN pip install git+https://github.com/opera-adt/RTC.git@v0.3 +RUN conda list ARG IMAGE_REF ENV DOCKERIMAGE_PATH=${IMAGE_REF} diff --git a/base_images/isce3/environment.yml b/base_images/isce3/environment.yml new file mode 100644 index 00000000..f4f13165 --- /dev/null +++ b/base_images/isce3/environment.yml @@ -0,0 +1,51 @@ +name: isce3 +channels: + - conda-forge + - nodefaults +dependencies: + - awscli=2.14.1 + - backoff=2.2.1 + - boto3=1.34.3 + - cython=3.0.7 + - earthengine-api=0.1.384 + - fsspec=2023.12.2 + - gdal=3.7.0 + - geopandas=0.14.1 + - h5py=3.9.0 + - hdf5=1.14.1 + - httpx=0.25.2 + - hvplot=0.9.0 + - isce3=0.14.0 + - mapclassify=2.6.1 + - matplotlib=3.8.2 + - mpl-scatter-density=0.7 + - numba=0.58.1 + - numpy=1.26.2 + - pandas=2.1.4 + - pycurl=7.45.1 + - pygeos=0.14 + - pyogrio=0.6.0 + - pyproj=3.6.1 + - pystac-client=0.7.5 + - python=3.10.13 + - rasterio=1.3.8 + - rasterstats=0.19.0 + - requests=2.31.0 + - rio-cogeo=5.1.0 + - rtree=1.1.0 + - s3fs=0.4.2 + - scikit-learn=1.3.2 + - scipy=1.11.4 + - seaborn=0.13.0 + - shapely=2.0.1 + - sliderule=4.1.0 + - statsmodels=0.14.1 + - tqdm=4.66.1 + - unidecode=1.3.7 + - xmltodict=0.13.0 + - xarray=2023.12.0 + - pip=23.3.2 + - pip: + - git+https://github.com/opera-adt/RTC.git@v0.3 +variables: + MAAP_CONF: '/maap-py/' \ No newline at end of file diff --git a/base_images/pangeo/docker/Dockerfile b/base_images/pangeo/docker/Dockerfile index 2e49605d..653d6ded 100644 --- a/base_images/pangeo/docker/Dockerfile +++ b/base_images/pangeo/docker/Dockerfile @@ -1,27 +1,29 @@ -FROM continuumio/miniconda3:22.11.1 +FROM continuumio/miniconda3:23.10.0-1 ENV LANG en_US.UTF-8 ENV TZ US/Pacific ARG DEBIAN_FRONTEND=noninteractive -# install maap-py library -ENV MAAP_CONF='/maap-py/' -RUN git clone --single-branch --branch v3.1.3 https://github.com/MAAP-Project/maap-py.git \ - && cd maap-py \ - && pip install -e . - RUN mkdir /projects WORKDIR /projects RUN sed -i -e 's/\/root/\/projects/g' /etc/passwd -RUN conda install -y -c conda-forge pyOpenSSL=23.2.0 && \ - conda install -y -n base conda-libmamba-solver=23.7.0 && \ - conda config --set solver libmamba - -RUN conda install -y --solver=libmamba -c conda-forge gdal=3.6.2 matplotlib=3.6.2 Cython=0.29.33 h5py=3.7.0 numba=0.56.4 \ - pygeos=0.14 pyproj=3.4.1 rasterio=1.3.4 scipy=1.10.0 \ +COPY ./environment.yml /tmp +RUN conda env create -y -f "/tmp/environment.yml" \ && find /opt/conda/ -follow -type f -name '*.a' -delete \ && find /opt/conda/ -follow -type f -name '*.js.map' -delete \ && /opt/conda/bin/conda clean -afy +# Set the conda environment for all subsequent docker RUN commands +# When maap-py is updated and installed via our environment.yml, +# we won't need this line unless other envrionment specific commands are added +SHELL ["conda", "run", "-n", "pangeo", "/bin/bash", "-c"] + +# Install maap-py library +RUN mkdir /maap-py \ + && git clone --single-branch --branch v3.1.4 https://github.com/MAAP-Project/maap-py.git /maap-py/ \ + && pip install -e /maap-py/ + +RUN conda list + ARG IMAGE_REF ENV DOCKERIMAGE_PATH=${IMAGE_REF} \ No newline at end of file diff --git a/base_images/pangeo/environment.yml b/base_images/pangeo/environment.yml new file mode 100644 index 00000000..eeecb06c --- /dev/null +++ b/base_images/pangeo/environment.yml @@ -0,0 +1,96 @@ +name: pangeo +channels: + - conda-forge + - nodefaults +dependencies: + - awscli=2.14.1 + - backoff=2.2.1 + - basemap=1.3.7 + - bokeh=3.3.3 + - boto3=1.33.13 + - bottleneck=1.3.7 + - cartopy=0.22.0 + - cfgrib=0.9.10.4 + - cython=3.0.7 + - dask-ml=2023.3.24 + - datashader=0.16.0 + - descartes=1.1.0 + - earthaccess=0.8.2 + - earthengine-api=0.1.384 + - eofs=1.4.0 + - erddapy=2.2.0 + - esmpy=8.4.2 + - flox=0.8.6 + - fsspec=2023.12.2 + - gcm_filters=0.3.0 + - gdal=3.7.0 + - geocube=0.4.2 + - geopandas=0.14.2 + - geoviews-core=1.11.0 + - h5netcdf=1.3.0 + - h5py=3.9.0 + - hdf5=1.14.0 + - holoviews=1.18.1 + - httpx=0.26.0 + - hvplot=0.9.1 + - intake=0.7.0 + - intake-geopandas=0.4.0 + - intake-stac=0.4.0 + - intake-xarray=0.7.0 + - kerchunk=0.2.2 + - lxml=4.9.3 + - lz4=4.3.2 + - mapclassify=2.6.1 + - matplotlib=3.7.3 + - mizani=0.10.0 + - mpl-scatter-density=0.7 + - nc-time-axis=1.4.1 + - netcdf4=1.6.4 + - numba=0.58.1 + - numcodecs=0.11.0 + - numpy=1.26.3 + - odc-stac=0.3.8 + - pandas=2.1.4 + - pandarallel=1.6.5 + - pyarrow=14.0.1 + - pycurl=7.45.1 + - pydap=3.4.0 + - pygeos=0.14 + - pyogrio=0.6.0 + - pyproj=3.5.0 + - pystac=1.9.0 + - pystac-client=0.6.1 + - python=3.10.13 + - python-blosc=1.10.6 + - rasterio=1.3.7 + - rasterstats=0.19.0 + - rechunker=0.5.2 + - requests=2.31.0 + - rio-cogeo=5.1.1 + - rioxarray=0.15.0 + - rtree=1.1.0 + - s3fs=2023.12.2 + - scikit-image=0.22.0 + - scikit-learn=1.3.2 + - scipy=1.11.4 + - seaborn=0.13.1 + - shapely=2.0.1 + - sliderule=4.1.0 + - sparse=0.15.0 + - stackstac=0.5.0 + - statsmodels=0.14.1 + - tqdm=4.66.1 + - unidecode=1.3.7 + - xarray=2023.12.0 + - xarray-datatree=0.0.13 + - xarray-spatial=0.3.5 + - xarrayutils=2.0.0 + - xesmf=0.8.2 + - xhistogram=0.3.2 + - xmltodict=0.13.0 + - zarr=2.16.1 + - pip=23.3.2 + - pip: + - morecantile==5.1.0 +variables: + MAAP_CONF: '/maap-py/' \ No newline at end of file diff --git a/base_images/r/docker/Dockerfile b/base_images/r/docker/Dockerfile index cd9f645b..8f98a98e 100644 --- a/base_images/r/docker/Dockerfile +++ b/base_images/r/docker/Dockerfile @@ -1,31 +1,63 @@ -FROM continuumio/miniconda3:22.11.1 +FROM continuumio/miniconda3:23.10.0-1 ENV LANG en_US.UTF-8 ENV TZ US/Pacific ARG DEBIAN_FRONTEND=noninteractive -# install maap-py library -ENV MAAP_CONF='/maap-py/' -RUN git clone --single-branch --branch v3.1.3 https://github.com/MAAP-Project/maap-py.git \ - && cd maap-py \ - && pip install -e . - -RUN set -ex \ - && apt-get update +#RUN set -ex \ +# && apt-get update RUN mkdir /projects WORKDIR /projects RUN sed -i -e 's/\/root/\/projects/g' /etc/passwd -RUN conda install -y -c conda-forge pyOpenSSL=23.2.0 && \ - conda install -y -n base conda-libmamba-solver=23.7.0 && \ - conda config --set solver libmamba +RUN apt-get clean && apt-get update && \ + apt-get install -y --no-install-recommends gdal-bin=3.2.2+dfsg-2+deb11u2 \ + lbzip2=2.5-2.1 \ + libfftw3-dev=3.3.8-2 \ + libgdal-dev=3.2.2+dfsg-2+deb11u2 \ + libgeos-dev=3.9.0-1 \ + libgl1-mesa-dev=20.3.5-1 \ + libglu1-mesa-dev=9.0.1-1 \ + libhdf4-alt-dev=4.2.15-3 \ + libhdf5-dev=1.10.6+repack-4+deb11u1 \ + libjq-dev=1.6-2.1 \ + libpq-dev=13.13-0+deb11u1 \ + libproj-dev=7.2.1-1 \ + libprotobuf-dev=3.12.4-1+deb11u1 \ + libnetcdf-dev=1:4.7.4-1 \ + libsqlite3-dev=3.34.1-3 \ + libssl-dev=1.1.1w-0+deb11u1 \ + libudunits2-dev=2.2.28-3 \ + netcdf-bin=1:4.7.4-1 \ + postgis=3.1.1+dfsg-1+deb11u2 \ + protobuf-compiler=3.12.4-1+deb11u1 \ + sqlite3=3.34.1-3 \ + tk-dev=8.6.11+1 \ + unixodbc-dev=2.3.6-0.1+b1 && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +COPY ./environment.yml /tmp +RUN conda env create -y -f "/tmp/environment.yml" \ + && find /opt/conda/ -follow -type f -name '*.a' -delete \ + && find /opt/conda/ -follow -type f -name '*.js.map' -delete \ + && /opt/conda/bin/conda clean -afy + +# Set the conda environment for all subsequent docker RUN commands +# When maap-py is updated and installed via our environment.yml, +# we won't need this line unless other envrionment specific commands are added +SHELL ["conda", "run", "-n", "r", "/bin/bash", "-c"] + +ADD . / +RUN ["chmod", "+x", "/scripts/install_cran_packages_r.sh"] +RUN /scripts/install_cran_packages_r.sh + +# Install maap-py library +RUN mkdir /maap-py \ + && git clone --single-branch --branch v3.1.4 https://github.com/MAAP-Project/maap-py.git /maap-py/ \ + && pip install -e /maap-py/ -RUN conda install -y --solver=libmamba -c conda-forge r==4.2 r-rgdal==1.5_32 r-sf==1.0_7 r-irkernel==1.3.2 r-gridExtra==2.3 \ - r-tidyverse==2.0.0 r-randomForest==4.7_1.1 r-raster==3.6_20 r-data.table==1.14.8 r-rlist==0.4.6.2 \ - r-gdalutils==2.0.3.2 r-stringr==1.5.0 && \ - find /opt/conda/ -follow -type f -name '*.a' -delete && \ - find /opt/conda/ -follow -type f -name '*.js.map' -delete && \ - /opt/conda/bin/conda clean -afy +RUN conda list ARG IMAGE_REF ENV DOCKERIMAGE_PATH=${IMAGE_REF} \ No newline at end of file diff --git a/base_images/r/environment.yml b/base_images/r/environment.yml new file mode 100644 index 00000000..7a1e13e1 --- /dev/null +++ b/base_images/r/environment.yml @@ -0,0 +1,108 @@ +name: r +channels: + - conda-forge + - r + - nodefaults +dependencies: + - awscli=2.14.1 + - earthaccess=0.8.2 + - parallel=20231122 + - perl=5.32.1 + - postgis=3.4.0 + - python=3.10.13 + - r=4.2 + - r-aws.s3=0.3.22 + - r-base=4.2.3 + - r-broom=1.0.5 + - r-car=3.1_2 + - r-chron=2.3_61 + - r-classint=0.4_10 + - r-covr=3.6.4 + - r-data.table=1.14.10 + - r-dbi=1.1.3 + - r-deldir=2.0_2 + - r-devemf=4.0_2 + - r-dplyr=1.1.4 + - r-egg=0.4.5 + - r-essentials=4.2 + - r-fs=1.6.3 + - r-geojsonio=0.11.3 + - r-geojsonsf=2.0.3 + - r-geor=1.9_2 + - r-geosphere=1.5_18 + - r-ggnewscale=0.4.9 + - r-ggplot2=3.4.4 + - r-ggspatial=1.1.9 + - r-ggtext=0.1.2 + - r-gridExtra=2.3 + - r-gstat=2.1_1 + - r-gsubfn=0.7 + - r-hdf5r=1.3.8 + - r-irkernel=1.3.2 + - r-jpeg=0.1_10 + - r-knitr=1.45 + - r-leafem=0.2.3 + - r-leaflet=2.2.1 + - r-lidR=4.0.4 + - r-lwgeom=0.2_13 + - r-mapdata=2.3.1 + - r-mapproj=1.2.11 + - r-maps=3.4.1.1 + - r-maptools=1.1_8 + - r-mapview=2.11.2 + - r-microbenchmark=1.4.10 + - r-ncdf4=1.22 + - r-nlme=3.1_164 + - r-patchwork=1.1.3 + - r-plogr=0.2.0 + - r-plyr=1.8.9 + - r-png=0.1_8 + - r-pool=1.0.1 + - r-proj4=1.0_13 + - r-proto=1.0.0 + - r-protolite=2.3.0 + - r-randomFields=3.3.14 + - r-randomForest=4.7_1.1 + - r-raster=3.6_26 + - r-rcolorbrewer=1.1_3 + - r-readxl=1.4.3 + - r-reticulate=1.34.0 + - r-rgee=1.1.7 + - r-rgeos=0.6_4 + - r-rlas=1.6.2 + - r-rlist=0.4.6.2 + - r-rmarkdown=2.25 + - r-rnaturalearth=0.3.4 + - r-rnaturalearthdata=0.1.0 + - r-RNetCDF=2.6_2 + - r-rockchalk=1.8.157 + - r-rodbc=1.3_23 + - r-RPostgres=1.4.5 + - r-RPostgreSQL=0.7_5 + - r-RSQLite=2.3.4 + - r-rstac=0.9.2_5 + - r-scales=1.3.0 + - r-sf=1.0_14 + - r-snow=0.4_4 + - r-spacetime=1.3_1 + - r-spatialreg=1.3_1 + - r-spatstat=3.0_7 + - r-spdep=1.3_1 + - r-sqldf=0.4_11 + - r-stars=0.6_4 + - r-stringr=1.5.1 + - r-terra=1.7_55 + - r-testthat=3.2.1 + - r-tibble=3.2.1 + - r-tidync=0.3.0 + - r-tidyr=1.3.0 + - r-tidyverse=2.0.0 + - r-tmap=3.3_4 + - r-units=0.8_5 + - r-viridis=0.6.4 + - pip=23.3.2 + - pip: + - ff==0.0.10 + - grid==0.7.1 +variables: + MAAP_CONF: '/maap-py/' \ No newline at end of file diff --git a/base_images/r/scripts/install2.r b/base_images/r/scripts/install2.r new file mode 100644 index 00000000..29ca3181 --- /dev/null +++ b/base_images/r/scripts/install2.r @@ -0,0 +1,187 @@ +#!/usr/bin/env r +# +# A second example to install one or more packages, now with option parsing +# +# Copyright (C) 2011 - 2014 Dirk Eddelbuettel +# Copyright (C) 2014 - 2017 Carl Boettiger and Dirk Eddelbuettel +# Copyright (C) 2018 - 2023 Carl Boettiger, Dirk Eddelbuettel, Brandon Bertelsen, and SHIMA Tatsuya +# +# Released under GPL (>= 2) +# This file was extracted from this repository https://github.com/rocker-org/rocker-versioned2 +# The license of this repository (rocker-versioned2) can be view here: +# https://github.com/rocker-org/rocker-versioned2/blob/master/LICENSE + +# Warranty description we are required to convey: +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Any questions, please contact: grace.llewellyn@jpl.nasa.gov + + +## load docopt package from CRAN +# Grace Llewellyn added the repos parameter to fix a bug +install.packages('docopt', repos = "http://cran.us.r-project.org") +library(docopt) + +## default to first library location in .libPaths() +libloc <- .libPaths()[1] + +## configuration for docopt +doc <- paste0("Usage: install2.r [-l LIBLOC] [-h] [-x] [-s] [-d DEPS] [-n NCPUS] [-r REPOS...] [-m METHOD] [-t TYPE] [--error] [--skipmissing] [--] [PACKAGES ...] + +-l --libloc LIBLOC location in which to install [default: ", libloc, "] +-d --deps DEPS install suggested dependencies as well [default: NA] +-n --ncpus NCPUS number of processes to use for parallel install, -1 selects all cores [default: getOption] +-r --repos REPOS repositor(y|ies) to use, or NULL for file [default: getOption] +-e --error throw error and halt instead of a warning [default: FALSE] +--skipmissing use with the --error option, skip the packages missing error [default: FALSE] +-s --skipinstalled skip installing already installed packages [default: FALSE] +-m --method METHOD method to be used for downloading files [default: auto] +-t --type TYPE installation type as used by `install.packages` [default: getOption] +-h --help show this help text +-x --usage show help and short example usage") +opt <- docopt(doc) # docopt parsing +if (opt$usage) { + cat(doc, "\n\n") + cat("where PACKAGES... can be one or more CRAN package names, or local (binary or source) +package files (where extensions .tar.gz, .tgz and .zip are recognised). Optional +arguments understood by R CMD INSTALL can be passed interspersed in the PACKAGES, though +this requires use of '--'. + +Examples: + install2.r -l /tmp/lib Rcpp BH # install into given library + install2.r -- --with-keep.source drat # keep the source + install2.r -- --data-compress=bzip2 stringdist # prefer bz2 compression + install2.r \".\" # install package in current directory + install2.r -n 6 ggplot2 # parallel install: (6 processes) + +install2.r is part of littler which brings 'r' to the command-line. +See https://dirk.eddelbuettel.com/code/littler.html for more information.\n") + q("no") +} +if (opt$deps == "TRUE" || opt$deps == "FALSE") { + opt$deps <- as.logical(opt$deps) +} else if (opt$deps == "NA") { + opt$deps <- NA +} +## docopt results are characters, so if we meant NULL we have to set NULL +if (length(opt$repos) == 1 && "NULL" %in% opt$repos) { + opt$repos <- NULL +} + +if ("getOption" %in% opt$repos) { + ## as littler can now read ~/.littler.r and/or /etc/littler.r we can preset elsewhere + opt$repos <- c(opt$repos[which(opt$repos != "getOption")], getOption("repos")) +} + +if (opt$ncpus == "getOption") { + opt$ncpus <- getOption("Ncpus", 1L) +} else if (opt$ncpus == "-1") { + ## parallel comes with R 2.14+ + opt$ncpus <- max(1L, parallel::detectCores()) +} + +## type should reflects bspm where available +if (opt$type == "getOption") { + opt$type <- getOption("pkgType") + #if (requireNamespace("bspm", quietly=TRUE) && Sys.info()[["sysname"]] == "Linux") opt$type <- "binary-source" + +} + +## ensure installation is stripped +Sys.setenv("_R_SHLIB_STRIP_"="true") +install_packages2 <- function(pkgs, ..., error = FALSE, skipmissing = FALSE, skipinstalled = FALSE) { + e <- NULL + capture <- function(e) { + if (error) { + catch <- + grepl("(download|installation) of package .* failed", e$message) || + (grepl("(dependenc|package).*(is|are) not available", e$message) && !skipmissing) || + grepl("installation of package.*had non-zero exit status", e$message) || + grepl("installation of .+ package(|s) failed", e$message) + if (catch) { + e <<- e + } + } + } + if (skipinstalled) { + pkgs <- setdiff(pkgs, installed.packages()[,1]) + } + if (length(pkgs) > 0) { + withCallingHandlers(install.packages(pkgs, ...), warning = capture) + if (!is.null(e)) { + stop(e$message, call. = FALSE) + } + } +} + +## helper function to for existing files with matching extension +isMatchingFile <- function(f) (file.exists(f) && grepl("(\\.tar\\.gz|\\.tgz|\\.zip)$", f)) || (f == ".") + +## helper function which switches to local (ie NULL) repo if matching file is presented +installArg <- function(f, lib, rep, dep, iopts, error, skipmissing, skipinstalled, ncpus, method, type) { + install_packages2(pkgs=f, + lib=lib, + repos=if (isMatchingFile(f)) NULL else rep, + dependencies=dep, + INSTALL_opts=iopts, + Ncpus = ncpus, + method = method, + type = type, + error = error, + skipmissing = skipmissing, + skipinstalled = skipinstalled) +} + +## strip out arguments to be passed to R CMD INSTALL +isArg <- grepl('^--',opt$PACKAGES) +installOpts <- opt$PACKAGES[isArg] +opt$PACKAGES <- opt$PACKAGES[!isArg] + +if (length(opt$PACKAGES)==0 && file.exists("DESCRIPTION") && file.exists("NAMESPACE")) { + ## we are in a source directory, so build it + message("* installing *source* package found in current working directory ...") + opt$PACKAGES <- "." +} + +## helper function to for existing files with matching extension +isMatchingFile <- + function(f) (file.exists(f) && + grepl("(\\.tar\\.gz|\\.tgz|\\.zip)$", f)) || (f == ".") + +## check arguments for local files -- if none, then we can pass vector on +isLocal <- sapply(opt$PACKAGES, isMatchingFile) + +## for any local sources loop explicitly as before, otherwise for remote +## packages pass vector to install_packages2 which does the rest (and +## possibly in parallel using up to ncpus) +if (any(isLocal)) { + sapply(opt$PACKAGES, installArg, opt$libloc, opt$repos, opt$deps, + installOpts, opt$error, opt$skipmissing, opt$skipinstalled, opt$ncpus, opt$method, opt$type) +} else { + install_packages2(pkgs = opt$PACKAGES, + lib = opt$libloc, + repos = opt$repos, + dependencies = opt$deps, + INSTALL_opts = installOpts, + Ncpus = opt$ncpus, + method = opt$method, + type = opt$type, + error = opt$error, + skipmissing = opt$skipmissing, + skipinstalled = opt$skipinstalled) +} + +## clean up any temp file containing CRAN directory information +sapply(list.files(path=tempdir(), pattern="^(repos|libloc).*\\.rds$", full.names=TRUE), unlink) \ No newline at end of file diff --git a/base_images/r/scripts/install_cran_packages_r.sh b/base_images/r/scripts/install_cran_packages_r.sh new file mode 100755 index 00000000..b3b1e62e --- /dev/null +++ b/base_images/r/scripts/install_cran_packages_r.sh @@ -0,0 +1,22 @@ +#!/bin/bash +set -e + +# always set this for scripts but don't declare as ENV.. +export DEBIAN_FRONTEND=noninteractive + +## build ARGs +NCPUS=${NCPUS:--1} + +Rscript /scripts/install2.r --error --skipmissing --skipinstalled -r "http://cran.us.r-project.org" -n "$NCPUS" nlraa sfarrow Fgmutils + +# Clean up +rm -rf /var/lib/apt/lists/* +if [ -d '/tmp/downloaded_packages' ]; then + rm -r /tmp/downloaded_packages +fi +#rm -r /tmp/downloaded_packages + +if [ -f '/usr/local/lib/R/site-library/*/libs/*.so' ]; then + strip /usr/local/lib/R/site-library/*/libs/*.so +fi +#strip /usr/local/lib/R/site-library/*/libs/*.so \ No newline at end of file diff --git a/base_images/rgedi/docker/Dockerfile b/base_images/rgedi/docker/Dockerfile index 6725ec7a..14d27346 100644 --- a/base_images/rgedi/docker/Dockerfile +++ b/base_images/rgedi/docker/Dockerfile @@ -13,7 +13,7 @@ RUN apt-get update && apt-get install -y libxt-dev && apt-get clean RUN conda install -y -c conda-forge pyOpenSSL=23.2.0 && \ conda install -y -n base conda-libmamba-solver=23.7.0 && \ - conda config --set solver libmamba + conda config --system --set solver libmamba RUN conda install -y --solver=libmamba -c conda-forge nb_conda_kernels diff --git a/base_images/vanilla/docker/Dockerfile b/base_images/vanilla/docker/Dockerfile index 47a8f194..8b49f89e 100644 --- a/base_images/vanilla/docker/Dockerfile +++ b/base_images/vanilla/docker/Dockerfile @@ -1,27 +1,29 @@ -FROM continuumio/miniconda3:22.11.1 +FROM continuumio/miniconda3:23.10.0-1 ENV LANG en_US.UTF-8 ENV TZ US/Pacific ARG DEBIAN_FRONTEND=noninteractive -# install maap-py library -ENV MAAP_CONF='/maap-py/' -RUN git clone --single-branch --branch v3.1.3 https://github.com/MAAP-Project/maap-py.git \ - && cd maap-py \ - && pip install -e . - RUN mkdir /projects WORKDIR /projects RUN sed -i -e 's/\/root/\/projects/g' /etc/passwd -RUN conda install -y -c conda-forge pyOpenSSL=23.2.0 && \ - conda install -y -n base conda-libmamba-solver=23.7.0 && \ - conda config --set solver libmamba - -RUN conda install -y --solver=libmamba -c conda-forge gdal=3.6.2 matplotlib=3.6.2 Cython=0.29.33 h5py=3.7.0 numba=0.56.4 \ - pygeos=0.14 pyproj=3.4.1 rasterio=1.3.4 scipy=1.10.0 \ +COPY ./environment.yml /tmp +RUN conda env create -y -f "/tmp/environment.yml" \ && find /opt/conda/ -follow -type f -name '*.a' -delete \ && find /opt/conda/ -follow -type f -name '*.js.map' -delete \ && /opt/conda/bin/conda clean -afy +# Set the conda environment for all subsequent docker RUN commands +# When maap-py is updated and installed via our environment.yml, +# we won't need this line unless other envrionment specific commands are added +SHELL ["conda", "run", "-n", "vanilla", "/bin/bash", "-c"] + +# Install maap-py library +RUN mkdir /maap-py \ + && git clone --single-branch --branch v3.1.4 https://github.com/MAAP-Project/maap-py.git /maap-py/ \ + && pip install -e /maap-py/ + +RUN conda list + ARG IMAGE_REF ENV DOCKERIMAGE_PATH=${IMAGE_REF} diff --git a/base_images/vanilla/environment.yml b/base_images/vanilla/environment.yml new file mode 100644 index 00000000..d89bea89 --- /dev/null +++ b/base_images/vanilla/environment.yml @@ -0,0 +1,51 @@ +name: vanilla +channels: + - conda-forge + - nodefaults +dependencies: + - awscli=2.14.1 + - backoff=2.2.1 + - basemap=1.3.7 + - boto3=1.34.15 + - cython=3.0.7 + - earthengine-api=0.1.384 + - gdal=3.7.0 + - geocube=0.4.2 + - geopandas=0.14.2 + - h5py=3.9.0 + - hdf5=1.14.0 + - httpx=0.26.0 + - mapclassify=2.6.1 + - matplotlib=3.7.3 + - mizani=0.10.0 + - mpl-scatter-density=0.7 + - numba=0.58.1 + - numpy=1.26.3 + - pandas=2.1.4 + - pandarallel=1.6.5 + - pycurl=7.45.1 + - pygeos=0.14 + - pyogrio=0.6.0 + - pyproj=3.5.0 + - pystac-client=0.7.5 + - python=3.10.13 + - rasterio=1.3.7 + - rasterstats=0.19.0 + - requests=2.31.0 + - rio-cogeo=5.1.1 + - rtree=1.1.0 + - s3fs=0.4.2 + - scikit-learn=1.3.2 + - scipy=1.11.4 + - seaborn=0.13.1 + - shapely=2.0.1 + - sliderule=4.1.0 + - statsmodels=0.14.1 + - tqdm=4.66.1 + - unidecode=1.3.7 + - xmltodict=0.13.0 + - pip=23.3.2 + - pip: + - morecantile==5.1.0 +variables: + MAAP_CONF: '/maap-py/' \ No newline at end of file diff --git a/jupyterlab3/docker/Dockerfile b/jupyterlab3/docker/Dockerfile index 72b6782a..00055a2f 100644 --- a/jupyterlab3/docker/Dockerfile +++ b/jupyterlab3/docker/Dockerfile @@ -1,7 +1,27 @@ ARG BASE_IMAGE -FROM ${BASE_IMAGE} as jupyterlab_base +ARG BASE_IMAGE_TYPE +FROM ${BASE_IMAGE} as jupyterlab_base ARG BASE_IMAGE_TYPE +# Set WORKSPACE_TYPE, used by entrypoint.sh +ENV WORKSPACE_TYPE=${BASE_IMAGE_TYPE} + +# Create base images for each workspace needed for build stage later +FROM jupyterlab_base as workspace_base_pangeo +SHELL ["conda", "run", "-n", "pangeo", "/bin/bash", "-c"] + +FROM jupyterlab_base as workspace_base_vanilla +SHELL ["conda", "run", "-n", "vanilla", "/bin/bash", "-c"] + +FROM jupyterlab_base as workspace_base_r +SHELL ["conda", "run", "-n", "r", "/bin/bash", "-c"] + +FROM jupyterlab_base as workspace_base_isce3 +SHELL ["conda", "run", "-n", "isce3", "/bin/bash", "-c"] + +FROM workspace_base_${BASE_IMAGE_TYPE} AS workspace_base + +FROM workspace_base as workspace_build_stage ADD . / @@ -11,6 +31,9 @@ RUN sed -i -e 's/\/root/\/projects/g' /etc/passwd USER root +RUN conda config --system --add channels conda-forge +RUN conda config --system --remove channels defaults + # Install additional libraries required by Python packages which are in # the minimal base image. Also install 'rsync' so the 'oc rsync' command # can be used to copy files into the running container. @@ -20,18 +43,17 @@ RUN apt-get clean && apt-get update && \ rm -rf /var/lib/apt/lists/* # Install workspace specific packages for conda environment -RUN echo "Checking if environment.yml exists for ${BASE_IMAGE_TYPE}" \ - ; if [ -f "/${BASE_IMAGE_TYPE}/environment.yml" ]; then \ - echo "Installing packages from '/${BASE_IMAGE_TYPE}/environment.yml'" \ - ; conda env update --name base --file "/${BASE_IMAGE_TYPE}/environment.yml" --solver libmamba \ +RUN echo "Checking if environment.yml exists for ${WORKSPACE_TYPE}" \ + ; if [ -f "/${WORKSPACE_TYPE}/environment.yml" ]; then \ + echo "Installing packages from '/${WORKSPACE_TYPE}/environment.yml'" \ + ; conda env update --file "/${WORKSPACE_TYPE}/environment.yml" \ ; else \ - echo "Installing packages from '/shared/environment.yml'" \ - ; conda env update --name base --file "/shared/environment.yml" --solver libmamba \ + sed -i "s/WORKSPACE_ENVIRONMENT_NAME/${WORKSPACE_TYPE}/g" /shared/environment.yml \ + ; echo "Installing packages from '/shared/environment.yml'" \ + ; conda env update --file "/shared/environment.yml" \ ; fi RUN npm install typescript -g -RUN pip install jupyter-server==2.5.0 -RUN pip install jupyter-resource-usage==0.7.2 # Adjust permissions on home directory so writable by group root. RUN chgrp -Rf root /home/$NB_USER && chmod -Rf g+w /home/$NB_USER @@ -51,7 +73,7 @@ RUN jupyter labextension disable @jupyterlab/apputils-extension:announcements ############################### # Custom Jupyter Extensions ############################### -RUN jupyter labextension install @maap-jupyterlab/dps-jupyter-extension@0.5.1 --no-build +RUN jupyter labextension install @maap-jupyterlab/dps-jupyter-extension@0.5.2 --no-build # PyPi package prepended with 'maap' so it more easily discoverable RUN pip install maap-jupyter-server-extension==1.2.2 @@ -85,4 +107,9 @@ COPY ./environments.json $ENVIRONMENTS_FILE_PATH RUN echo 'printf "\e[101mNOTE: This terminal uses the default environment which might not be the same as your notebook.\e[49m\n\n"' >> /etc/profile -ENTRYPOINT ["/bin/bash", "/entrypoint.sh"] +RUN conda list + +# Reset Shell +SHELL ["/bin/sh", "-c"] + +ENTRYPOINT ["/bin/bash", "/entrypoint.sh"] \ No newline at end of file diff --git a/jupyterlab3/entrypoint.sh b/jupyterlab3/entrypoint.sh index fa32971c..2254a59e 100644 --- a/jupyterlab3/entrypoint.sh +++ b/jupyterlab3/entrypoint.sh @@ -90,8 +90,11 @@ done # allows `base_url` to be set so that HTTP *responses* include the `base_url` so that browsers make the correct # call. However, it strips out `base_url` when listening for *requests* so that handles the ingress/nginx proxy # requests correctly. -export NOTEBOOKLIBPATH=$(find /opt/conda/lib/ -maxdepth 3 -type d -name "notebook") -export JUPYTERSERVERLIBPATH=$(find /opt/conda/lib -maxdepth 3 -type d -name "jupyter_server") +export NOTEBOOKLIBPATH=$(find /opt/conda/envs/$WORKSPACE_TYPE/lib -maxdepth 3 -type d -name "notebook") +export JUPYTERSERVERLIBPATH=$(find /opt/conda/envs/$WORKSPACE_TYPE/lib -maxdepth 3 -type d -name "jupyter_server") + +# Fix "fatal: not a git repository" error on startup +export GIT_DISCOVERY_ACROSS_FILESYSTEM=1 read -r -d '' JUPYTER_PATCH << EOM # Fix for Tornado's inability to handle proxy requests @@ -120,10 +123,15 @@ fi # Dump all env variables into file so they exist still though SSH env | grep _ >> /etc/environment -# Add conda bin to path -export PATH=$PATH:/opt/conda/bin -cp /root/.bashrc ~/.bash_profile -conda init +# Add workspace env conda bin to path +export PATH=$PATH:/opt/conda/envs/$WORKSPACE_TYPE/bin + +# Set up default conda environment for terminals +echo ". /opt/conda/etc/profile.d/conda.sh ; conda activate $WORKSPACE_TYPE" > /etc/profile.d/init_conda.sh + +# Cleanup ~/.bash_profile +sed -i "s/\. \/opt\/conda\/etc\/profile.d\/conda\.sh//g" ~/.bash_profile +sed -i "s/conda activate base//g" ~/.bash_profile # Need to fix directory permissions for publickey authentication chmod 700 /projects @@ -155,7 +163,7 @@ MEMORY=$(get_max_memory "$NAMESPACE") # TBD maap-py install -source /opt/conda/bin/activate base +source /opt/conda/bin/activate $WORKSPACE_TYPE export SHELL=/bin/bash VERSION=$(jupyter lab --version) if [[ $VERSION > '2' ]] && [[ $VERSION < '3' ]]; then @@ -164,4 +172,4 @@ elif [[ $VERSION > '3' ]] && [[ $VERSION < '4' ]]; then jupyter lab --ip=0.0.0.0 --port=3100 --allow-root --ContentsManager.allow_hidden=True --ServerApp.token='' --ServerApp.base_url=$PREVIEW_URL --no-browser --debug --ServerApp.disable_check_xsrf=True --ResourceUseDisplay.mem_limit=$MEMORY --ResourceUseDisplay.mem_warning_threshold=0.2 else echo "Error! Jupyterlab version not supported." -fi \ No newline at end of file +fi diff --git a/jupyterlab3/isce3/environment.yml b/jupyterlab3/isce3/environment.yml deleted file mode 100644 index ba3be7e5..00000000 --- a/jupyterlab3/isce3/environment.yml +++ /dev/null @@ -1,16 +0,0 @@ -name: base -channels: - - conda-forge - - nodefaults -dependencies: - - gitpython=3.1.26 - - ipyleaflet=0.17.2 - - jupyterlab=3.6.3 - - jupyterlab-git=0.34.2 - - jupyter-packaging=0.12.3 - - jupyterlab_widgets=1.0.2 - - nodejs=16.19.0 - - plotly=5.5.0 - - xmltodict=0.13.0 - - pip: - - jupyter-ydoc==0.2.4 diff --git a/jupyterlab3/pangeo/environment.yml b/jupyterlab3/pangeo/environment.yml index 46e3c5be..54d1cd03 100644 --- a/jupyterlab3/pangeo/environment.yml +++ b/jupyterlab3/pangeo/environment.yml @@ -1,28 +1,31 @@ -name: base +name: pangeo channels: - conda-forge - nodefaults dependencies: - - pangeo-notebook=2023.04.15 - - gitpython=3.1.30 - - ipyevents=2.0.1 - - ipyleaflet=0.17.2 + - folium=0.15.1 + - gitpython=3.1.40 + - gh=2.40.1 + - git-lfs=3.4.1 + - ipyevents=2.0.2 + - ipyleaflet=0.18.1 + - ipywidgets=8.0.6 + - jupyter-packaging=0.12.3 - jupyterlab=3.6.3 - jupyterlab-git=0.34.2 - - jupyter-packaging=0.12.3 - jupyterlab_widgets=3.0.7 + - nbstripout=0.6.1 - nodejs=18.15.0 - - plotly=5.14.1 - - pydantic=1.10.8 - - rioxarray=0.14.1 - - shapely=2.0.1 - - xmltodict=0.13.0 - - xarray=2023.5.0 - - pip + - pangeo-notebook=2023.04.15 + - plotly=5.18.0 + - plotnine=0.12.2 + - pydantic=2.5.2 + - xarray_leaflet=0.2.3 + - pip=23.3.2 - pip: - - rio_tiler==4.1.11 - - pystac_client==0.6.1 - - git+https://github.com/MAAP-Project/stac_ipyleaflet.git@0.3.5#egg-info=stac_ipyleaflet + - jupyter-resource-usage==0.7.2 + - rio-tiler==6.2.8 + - git+https://github.com/MAAP-Project/stac_ipyleaflet.git@0.3.6#egg-info=stac_ipyleaflet variables: TITILER_STAC_ENDPOINT: 'https://titiler-stac.maap-project.org/' TITILER_ENDPOINT: 'https://titiler.maap-project.org/' diff --git a/jupyterlab3/shared/environment.yml b/jupyterlab3/shared/environment.yml index a9d5dbaf..863796e9 100644 --- a/jupyterlab3/shared/environment.yml +++ b/jupyterlab3/shared/environment.yml @@ -1,14 +1,19 @@ -name: base +name: WORKSPACE_ENVIRONMENT_NAME channels: - conda-forge - nodefaults dependencies: - - gitpython=3.1.26 - - ipyleaflet=0.17.2 - - jupyterlab=3.6.1 + - folium=0.15.1 + - gitpython=3.1.40 + - ipyleaflet=0.18.1 + - jupyterlab=3.6.3 - jupyterlab-git=0.34.2 - jupyter-packaging=0.12.3 - - jupyterlab_widgets=1.0.2 - - nodejs=16.14.2 - - plotly=5.5.0 - - xmltodict=0.13.0 \ No newline at end of file + - jupyterlab_widgets=3.0.7 + - nodejs=18.15.0 + - plotly=5.18.0 + - plotnine=0.12.2 + - pip=23.3.2 + - pip: + - jupyter-resource-usage==0.7.2 + - rio-tiler==6.2.8