feat: support terminology codes (#106)

* feat: code matching client * feat(remote): register fuzon endpoint in client * refactor(cli): prompt logic to dedicated prompt module * refactor(cli): prompt logic to dedicated prompt module * chore(deps): pyfuzon as extra dep * chore: update deps * feat(cli): support code completion in modos add * perf(codes): limit suggestions to 50 codes * fix(cli): use labels in recommendations * refactor(codes): custom Code struct * chore(deps): bump modos-schema version * feat(cli): prompt autocompletes text, persists uris * fix(cli): disable unnecessary autocomplete on modos create * test(data): use uris when required * fix(codes): fuzon-http api parameters * chore(make): document deploy recipe * feat(compose): add fuzon service * feat(nginx): register fuzon in reverse proxy * feat(fuzon): dockerized fuzon-http setup * fix(compose): add envvar for fuzon service * fix(compose): fuzon envvars * fix(nginx): typo * chore(deps): add prompt-toolkit * fix(deploy): pin fuzon to tag 0.2.3 * fix(compose): env var typo Co-authored-by: supermaxiste <[email protected]> * docs(deploy): describe config variables * feat(codes): parametrize n top code matches * fix(codes): update protocol signatures * fix(codes): default top value in code matcher protocol * docs(prompt): clearer comments * fix(deploy): typo in env var * feat(cli): list remote modos * fix(server): nest /list values in response * chore: bump pyfuzon to 0.2.5 for blank node fix and caching support * feat(cli): add search-codes command * chore(deploy): bump fuzon server to 0.2.5 * refactor(prompt): allow prompt override in SlotPrompter * refactor(codes): define protocol attributes in CodeMatcher * feat(cli): non-interactive code search --------- Co-authored-by: supermaxiste <[email protected]>
sdsc-ordes · Oct 23, 2024 · eef83b0 · eef83b0
1 parent f7542c2
commit eef83b0
Show file tree

Hide file tree

Showing 17 changed files with 1,941 additions and 1,383 deletions.
diff --git a/Makefile b/Makefile
@@ -6,7 +6,7 @@ VERSION :=$(shell grep -E '^version += +' pyproject.toml | sed -E 's/.*= +//')
 .PHONY: install
 install: ## Install with the poetry and add pre-commit hooks
 	@echo "🚀 Installing packages with poetry"
-	@poetry install
+	@poetry install --all-extras --no-cache
 	@poetry run pre-commit install
 
 .PHONY: check
@@ -20,7 +20,7 @@ check: ## Run code quality tools.
 doc: ## Build sphinx documentation website locally
 	@echo "📖 Building documentation"
 	@cd docs
-	@poetry install --with docs
+	@poetry install --with docs --all-extras
 	@poetry run sphinx-build docs/ docs/_build
 
 .PHONY: docker-build
@@ -36,10 +36,10 @@ test: ## Test the code with pytest
 	@poetry run pytest
 
 .PHONY: deploy
-deploy:
+deploy: ## Deploy services using docker compose
 	@echo "$(LOCAL_IP)";exit 0
 	@echo "🐋 Deploying server with docker compose"
-	cd deploy; S3_PUBLC_URL="http://$(LOCAL_IP):9000" docker compose up --build --force-recreate
+	cd deploy; S3_PUBLIC_URL="http://$(LOCAL_IP):9000" docker compose up --build --force-recreate
 
 
 .PHONY: help

diff --git a/data/ex/data.zarr/.zmetadata b/data/ex/data.zarr/.zmetadata
@@ -61,10 +61,10 @@
             "data_path": "demo1.cram",
             "description": "Dummy data for tests",
             "has_reference": [
-                "reference1"
+                "reference/reference1"
             ],
             "has_sample": [
-                "sample1"
+                "sample/sample1"
             ],
             "name": "Demo 1"
         },
@@ -88,11 +88,11 @@
         },
         "sample/sample1/.zattrs": {
             "@type": "Sample",
-            "cell_type": "astrocyte",
+            "cell_type": "http://purl.obolibrary.org/obo/CL_0002627",
             "description": "Dummy sample for tests",
             "name": "Sample 1",
             "sex": "Male",
-            "source_material": "brain tissue"
+            "source_material": "http://purl.obolibrary.org/obo/UBERON_0002316"
         },
         "sample/sample1/.zgroup": {
             "zarr_format": 2

diff --git a/data/ex/data.zarr/sample/sample1/.zattrs b/data/ex/data.zarr/sample/sample1/.zattrs
@@ -1,8 +1,8 @@
 {
     "@type": "Sample",
-    "cell_type": "astrocyte",
+    "cell_type": "http://purl.obolibrary.org/obo/CL_0002627",
     "description": "Dummy sample for tests",
     "name": "Sample 1",
     "sex": "Male",
-    "source_material": "brain tissue"
+    "source_material": "http://purl.obolibrary.org/obo/UBERON_0002316"
 }
diff --git a/deploy/.example.env b/deploy/.example.env
@@ -14,4 +14,5 @@ HTSGET_DATA_DIR="/data"
 # PUBLIC
 # URL_PREFIX="https://modos.example.org"
 # S3_PUBLIC_URL="${URL_PREFIX}/s3"
+# FUZON_PUBLIC_URL="${URL_PREFIX}/fuzon"
 # HTSGET_PUBLIC_URL="${URL_PREFIX}/htsget"
diff --git a/deploy/README.md b/deploy/README.md
@@ -45,15 +45,14 @@ end
 ## Setup
 
 > [!IMPORTANT]
-> The instructions below are meant for local testing.
+> The instructions below are meant for development.
 > Production use would require changing the default
 > credentials and use authentication.
 
 1. Start the server
 
 ```sh
-cd deploy
-docker compose up --build
+make deploy
 ```
 
 2. Upload MODO(s) to the default bucket from the minio console (default is http://localhost:9001)
@@ -80,14 +79,17 @@ docker compose up --build
 # docker compose automatically reads the .env file
 ```
 
+In the `.env` file, each service has a `<service>_PUBLIC_URL` and a `<service>_LOCAL_URL` variable. The public URL is the address used by external clients, whereas LOCAL_URL is the address used by other services. For services deployed as part of the compose setup, the local address is `http://<service-name>:<service-port>` and the public address is the endpoint exposed by the nginx reverse proxy, typically `http://<server-host>/<service-name>`. If a service is deployed outside the compose, e.g. an external s3 bucket, the public and local address will both be pointing to the external address.
+
 ### Streaming with minio
 
 There are two options to use htsget streaming with the minio embedded in the compose setup:
 
-1. Either manually create a host mapping from the minio service to localhost:
+1. Set `S3_PUBLIC_URL=http://<LOCAL-IP>:9000` where `<LOCAL-IP>` is your local IP address (find it using hostname -I). **This is done automatically when starting the server with `make deploy`**.
+
+2. Manually create a host mapping from the minio service to localhost on the machine:
 > `echo "127.0.0.1 minio" >> /etc/hosts`
 
-2. Or set `S3_PUBLIC_URL=http://<LOCAL-IP>:9000` where `<LOCAL-IP>` is your local IP address (find it using hostname -I).
 
 These steps are not needed when using an external S3 server, in which case `S3_PUBLIC_URL` can just be set to the external S3 endpoint.
 

diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml
@@ -15,11 +15,13 @@ services:
     volumes:
       - ./nginx/default.conf.template:/etc/nginx/templates/default.conf.template:ro
     environment:
-      - S3_PUBLIC_URL=${S3_PUBLIC_URL:-http://localhost/s3/}
-      - S3_LOCAL_URL=${S3_lOCAL_URL:-http://minio:9000}
+      - FUZON_PUBLIC_URL=${FUZON_PUBLIC_URL:-http://localhost/fuzon}
       - HTSGET_PUBLIC_URL=${HTSGET_PUBLIC_URL:-http://localhost/htsget}
+      - S3_PUBLIC_URL=${S3_PUBLIC_URL:-http://localhost/s3/}
+      - FUZON_LOCAL_URL=${FUZON_LOCAL_URL:-http://fuzon:8080}
       - HTSGET_LOCAL_URL=${HTSGET_LOCAL_URL:-http://htsget:8080}
       - MODOS_LOCAL_URL=${MODOS_LOCAL_URL:-http://modos-server:8000}
+      - S3_LOCAL_URL=${S3_LOCAL_URL:-http://minio:9000}
     ports:
       - "80:80"
     networks:
@@ -64,6 +66,16 @@ services:
       - RUST_LOG=debug
 
 
+  fuzon:
+    build: ./fuzon
+    volumes:
+      - ./fuzon/config.json.template:/fuzon/config.json.template:ro
+    expose:
+      - "8080"
+    networks:
+      - modos-network
+
+
   modos-server:
     build: ./modos-server
     expose:

diff --git a/deploy/fuzon/Dockerfile b/deploy/fuzon/Dockerfile
@@ -0,0 +1,26 @@
+FROM rust:1.75.0 AS builder
+
+WORKDIR /build
+
+RUN cargo install cargo-strip
+
+RUN apt-get update && apt-get install -y git
+
+RUN git clone https://github.com/sdsc-ordes/fuzon.git --branch v0.2.5 .
+
+RUN cd fuzon-http && cargo build --release && cargo strip
+
+FROM debian:stable-slim
+
+# gettext-base is required for envsubst (config templating)
+RUN apt update && apt install -y gettext-base libc6-dev && rm -rf /var/lib/apt/lists/*
+
+COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs/
+COPY --from=builder /build/target/release/fuzon-http /usr/local/bin/fuzon-http
+COPY docker-entrypoint.sh /
+
+EXPOSE 8080
+
+ENTRYPOINT ["/docker-entrypoint.sh"]
+
+CMD ["fuzon-http", "--config", "/fuzon/config.json"]
diff --git a/deploy/fuzon/config.json.template b/deploy/fuzon/config.json.template
@@ -0,0 +1,9 @@
+{
+  "host": "::",
+  "port": 8080,
+  "collections": {
+    "cell_type": ["https://purl.obolibrary.org/obo/cl.owl"],
+    "source_material": ["https://purl.obolibrary.org/obo/uberon.owl"],
+    "taxon_id": ["https://purl.obolibrary.org/obo/ncbitaxon/subsets/taxslim.owl"]
+  }
+}
diff --git a/deploy/fuzon/docker-entrypoint.sh b/deploy/fuzon/docker-entrypoint.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+set -e
+
+# only use entrypoint if running fuzon-http
+if [ "$1" = "fuzon-http" ] ; then
+
+  # Templating
+  if [ -e "/fuzon/config.json" ]; then
+    echo "Using existing config.json"
+
+  elif [ -e "/fuzon/config.json.template" ]; then
+    echo "Generating config.json from config.json.template"
+    envsubst < /fuzon/config.json.template > /fuzon/config.json
+    cat /fuzon/config.json
+    echo "$@"
+
+  else
+    echo "No config.json or config.json.template found. Exiting."
+    exit 1
+
+  fi
+fi
+
+exec "$@"
diff --git a/deploy/modos-server/server.py b/deploy/modos-server/server.py
@@ -25,11 +25,11 @@
 
 
 @app.get("/list")
-def list_modos() -> list[str]:
+def list_modos() -> dict[str, list[str]]:
     """List MODO entries in bucket."""
     modos = minio.ls(BUCKET, refresh=True)
     # NOTE: modo contains bucket name
-    return [modo for modo in modos]
+    return {"modos": [f"s3://{modo}" for modo in modos]}
 
 
 @app.get("/meta")

diff --git a/deploy/nginx/default.conf.template b/deploy/nginx/default.conf.template
@@ -14,7 +14,8 @@ server {
       {
         "status": "success",
         "s3": "${S3_PUBLIC_URL}",
-        "htsget": "${HTSGET_PUBLIC_URL}"
+        "htsget": "${HTSGET_PUBLIC_URL}",
+        "fuzon": "${FUZON_PUBLIC_URL}"
       }' ;
   }
   # S3 bucket service
@@ -41,4 +42,9 @@ server {
   location / {
     proxy_pass ${MODOS_LOCAL_URL} ;
   }
+  # terminology code matching server
+  location /fuzon {
+    rewrite ^/fuzon/(.*) /$1 break ;
+    proxy_pass ${FUZON_LOCAL_URL} ;
+  }
 }