Make it possible to run titiler-cmr locally (#28)

* add config options for running locally via external access

* update dependencies

* fix deprecation warnings

.github/workflows/ci.yml

@@ -31,7 +31,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.8', '3.9', '3.10', '3.11']
+        python-version: ['3.10', '3.11', '3.12']
 
     steps:
       - uses: actions/checkout@v3

.gitignore

@@ -107,3 +107,5 @@ node_modules
 cdk.context.json
 
 notebooks/
+
+uv.lock

Dockerfile

@@ -0,0 +1,32 @@
+ARG PYTHON_VERSION=3.11
+
+
+FROM python:${PYTHON_VERSION}-slim
+
+WORKDIR /tmp
+
+COPY titiler/ titiler/
+COPY pyproject.toml pyproject.toml
+COPY README.md README.md
+COPY LICENSE LICENSE
+
+RUN pip install --no-cache-dir --upgrade . uvicorn
+RUN rm -rf titiler/ pyproject.toml README.md LICENSE
+
+ARG EARTHDATA_USERNAME
+ARG EARTHDATA_PASSWORD
+
+# Check if EARTHDATA_USERNAME and EARTHDATA_PASSWORD are provided
+RUN if [ -z "$EARTHDATA_USERNAME" ] || [ -z "$EARTHDATA_PASSWORD" ]; then \
+  echo "Error: EARTHDATA_USERNAME and EARTHDATA_PASSWORD build args must be provided"; \
+  exit 1; \
+  fi && \
+  echo "machine urs.earthdata.nasa.gov\nlogin ${EARTHDATA_USERNAME}\npassword ${EARTHDATA_PASSWORD}" > ~/.netrc && \
+  unset EARTHDATA_USERNAME && \
+  unset EARTHDATA_PASSWORD
+
+# http://www.uvicorn.org/settings/
+ENV HOST 0.0.0.0
+ENV PORT 80
+CMD uvicorn titiler.cmr.main:app --host ${HOST} --port ${PORT} --log-level debug
+

README.md

@@ -1,3 +1,115 @@
+<p align="center">
+  <img width="500" src="./titiler-cmr.png"/>
+  <p align="center">A modern dynamic tile server with a NASA CMR backend built on top of FastAPI and Rasterio/GDAL.</p>
+</p>
+
+<p align="center">
+  <a href="https://github.com/developmentseed/titiler/actions?query=workflow%3ACI" target="_blank">
+      <img src="https://github.com/developmentseed/titiler/workflows/CI/badge.svg" alt="Test">
+  </a>
+  <a href="https://github.com/developmentseed/titiler-cmr/blob/main/LICENSE" target="_blank">
+      <img src="https://img.shields.io/github/license/developmentseed/titiler-cmr.svg" alt="Downloads">
+  </a>
+</p>
+
 # titiler-cmr
 
 An API for creating image tiles from CMR queries.
+
+## Features
+
+- Render tiles from assets discovered via queries to [NASA's CMR](https://cmr.earthdata.nasa.gov/search)
+- Uses the [`earthaccess` python package](https://github.com/nsidc/earthaccess) to query the CMR
+- Built on top of [titiler](https://github.com/developmentseed/titiler)
+- Multiple projections support (see [TileMatrixSets](https://www.ogc.org/standards/tms)) via [`morecantile`](https://github.com/developmentseed/morecantile).
+- JPEG / JP2 / PNG / WEBP / GTIFF / NumpyTile output format support
+- Automatic OpenAPI documentation (FastAPI builtin)
+- Example of AWS Lambda / ECS deployment (via CDK)
+
+## Installation
+
+To install from sources and run for development:
+
+```bash
+git clone https://github.com/developmentseed/titiler-cmr.git
+cd titiler-cmr
+
+python -m pip install -U pip
+python -m pip install uvicorn -e .[dev,test]
+```
+
+## Authentication for data read access
+
+`titiler-cmr` can read data either over `HTTP` (external) or directly from `AWS S3` (direct) depending on the app configuration.
+The behavior of the application is controlled by the S3 authentication settings in [`settings.py`](./titiler/cmr/settings.py), which you can set either with environment variables (`TITILER_CMR_S3_AUTH_ACCESS`, `TITILER_CMR_S3_AUTH_STRATEGY`) or in an environment file (`.env`).
+
+### Direct from S3
+
+When running in an AWS context (e.g. Lambda), you should configure the application to access the data directly from `S3`.
+You can do this in two ways:
+
+- Configure an AWS IAM role for your runtime environment that has read access to the NASA buckets so that `rasterio/GDAL` can find the AWS credentials when reading data
+- Set the `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD` environment variables so that the `earthaccess` package can issue temporary AWS credentials
+
+> [!NOTE]
+> Direct S3 access configuration will only work if the application is running in the same AWS region as the data are stored!
+
+### External access
+
+When running outside of the AWS context (e.g. locally) you will need to configure the application to access data over `HTTP`.
+You can do this by creating an Earthdata account, configuring your `.netrc` file with your Earthdata login credentials (which GDAL will find when trying to access data over the network), and setting a few environment variables:
+
+```bash
+# environment variables for GDAL to read data from NASA over HTTP
+export GDAL_DISABLE_READDIR_ON_OPEN=YES
+export CPL_VSIL_CURL_USE_HEAD=FALSE
+export GDAL_HTTP_COOKIEFILE=/tmp/cookies.txt
+export GDAL_HTTP_COOKIEJAR=/tmp/cookies.txt
+export EARTHDATA_USERNAME={your earthdata username}
+export EARTHDATA_PASSWORD={your earthdata password}
+
+# write your .netrc file to the home directory
+echo "machine urs.earthdata.nasa.gov login ${EARTHDATA_USERNAME} password ${EARTHDATA_PASSWORD}" > ~/.netrc
+```
+
+> [!NOTE]
+> See [NASA's docs](https://uat.urs.earthdata.nasa.gov/documentation/for_users/data_access/curl_and_wget) for details
+
+## Docker deployment
+
+You can run the application in a docker container using the [docker-compose.yml](./docker-compose.yml).
+The docker container is configured to read the `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD` environment variables so make sure set those before starting the docker network.
+
+```bash
+docker compose up --build 
+```
+
+The application will be available at this address: [http://localhost:8081/api.html](http://localhost:8081/api.html)
+
+## Local deployment
+
+To run the application directly in your local environment, configure the application to access data over `HTTP` then run it using `uvicorn`:
+
+```bash
+TITILER_CMR_S3_AUTH_ACCESS=external uvicorn titiler.cmr.main:app --reload
+```
+
+The application will be available at this address: [http://localhost:8000/api.html](http://localhost:8000/api.html)
+
+## Contribution & Development
+
+See [CONTRIBUTING.md](https://github.com/developmentseed/titiler-cmr/blob/develop/CONTRIBUTING.md)
+
+## License
+
+See [LICENSE](https://github.com/developmentseed/titiler-cmr/blob/develop/LICENSE)
+
+## Authors
+
+Created by [Development Seed](<http://developmentseed.org>)
+
+See [contributors](https://github.com/developmentseed/titiler-cmr/graphs/contributors) for a listing of individual contributors.
+
+## Changes
+
+See [CHANGES.md](https://github.com/developmentseed/titiler-cmr/blob/main/CHANGES.md).

docker-compose.yml

@@ -0,0 +1,56 @@
+services:
+  tiler:
+    container_name: tiler-cmr
+    platform: linux/amd64
+    build:
+      context: .
+      dockerfile: Dockerfile
+      args:
+        EARTHDATA_USERNAME: ${EARTHDATA_USERNAME}
+        EARTHDATA_PASSWORD: ${EARTHDATA_PASSWORD}
+    ports:
+      - "8081:8081"
+    environment:
+      # Application
+      - HOST=0.0.0.0
+      - PORT=8081
+      - WEB_CONCURRENCY=1
+      - TITILER_CMR_API_DEBUG=TRUE
+      - TITILER_CMR_S3_AUTH_STRATEGY=iam
+      - TITILER_CMR_S3_AUTH_ACCESS=external
+      # earthdata authentication
+      - EARTHDATA_USERNAME=${EARTHDATA_USERNAME}
+      - EARTHDATA_PASSWORD=${EARTHDATA_PASSWORD}
+      # GDAL Config
+      # This option controls the default GDAL raster block cache size.
+      # If its value is small (less than 100000), it is assumed to be measured in megabytes, otherwise in bytes.
+      - GDAL_CACHEMAX=200
+      - GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
+      - GDAL_INGESTED_BYTES_AT_OPEN=32768
+      - GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES
+      - GDAL_HTTP_COOKIEJAR=/tmp/cookies.txt
+      - GDAL_HTTP_MULTIPLEX=YES
+      - GDAL_HTTP_VERSION=2
+      # The file can be cached in RAM by setting the configuration option VSI_CACHE to TRUE.
+      # The cache size defaults to 25 MB, but can be modified by setting the configuration option VSI_CACHE_SIZE (in bytes).
+      # Content in that cache is discarded when the file handle is closed.
+      - VSI_CACHE=TRUE
+      - VSI_CACHE_SIZE=536870912
+      # In addition, a global least-recently-used cache of 16 MB shared among all downloaded content is enabled by default,
+      # and content in it may be reused after a file handle has been closed and reopen,
+      # during the life-time of the process or until VSICurlClearCache() is called.
+      # Starting with GDAL 2.3, the size of this global LRU cache can be modified by
+      # setting the configuration option CPL_VSIL_CURL_CACHE_SIZE (in bytes).
+      - CPL_VSIL_CURL_CACHE_SIZE=200000000
+      # TiTiler Config
+      - MOSAIC_CONCURRENCY=5
+      # AWS S3 endpoint config
+      # - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+      # - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+    develop:
+      watch:
+        - action: sync+restart
+          path: ./titiler/cmr/
+          target: /usr/local/lib/python3.11/site-packages/titiler/cmr/
+        - action: rebuild
+          path: ./pyproject.toml