Merge stable to master: v2.5.0

.gitignore

@@ -18,3 +18,18 @@ ubuntu-xenial-16.04-cloudimg-console.log
 !.env.template
 *.pem
 /src/log
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+pytestdebug.log

ENV_VARS.md

@@ -59,4 +59,5 @@ copy .env.template to .env and fill the required values from below table. Requir
 | ENABLE_V1_ENDPOINTS                     | no       | True
 | ENABLE_V2_ENDPOINTS                     | no       | True
 | VALIDATE_TOKEN_URL                      | no       | https://127.0.0.1/secure/validate_token                                               | URL where bearer tokens get validated
-| WKT_FILENAME                            | no       | src/metax_api/tasks/refdata/refdata_indexer/resources/uri_to_wkt.json                 |
+| WKT_FILENAME                            | no       | src/metax_api/tasks/refdata/refdata_indexer/resources/uri_to_wkt.json                 |
+| ENABLE_DJANGO_WATCHMAN                  | no       | False                                                                                 | Should watchman monitoring be enabled                                                                      |

README.md

@@ -9,7 +9,8 @@ Licensed under [GNU GPLv2 License](LICENSE)
 
 ## Setting up local development environment
 
-You can also set up the development environment with [Docker-swarm setup](/docs/docker-stack.md) or with [standalone Docker-containers setup](/docs/single-docker-images.md).
+The recommended way to run the development setup is to use [Docker-swarm setup](/docs/docker-stack.md). You can also set up the development environment with
+[standalone Docker-containers setup](/docs/single-docker-images.md) or local install documented below.
 
 ### Python dependencies
 
@@ -28,7 +29,6 @@ Dependencies can be updated using `poetry update`. Please notice that this will
 
 Dependencies can be removed with `poetry remove (-D) <package>`
 
-
 ### Required environmental variables
 
 copy `src/metax_api/settings/.env.template` as `src/metax_api/settings/.env` and fill required variables, you can find examples in ENV_VARS.md
@@ -41,31 +41,27 @@ copy `src/metax_api/settings/.env.template` as `src/metax_api/settings/.env` and
 
 Activate your python 3.8 virtualenv, install requirements with `pip install -r requirements.txt`
 
-`cd` into `src` folder and run following commands:
+`cd` into `src` folder and run following command:
 
-`python manage.py migrate`
+`python manage.py first_time_setup`
 
 start the development server with:
 
 `python manage.py runserver 8008`
 
-Open another terminal and `cd` into `src`, and load the initial data with following commands:
+Metax api is available from your browser at http://localhost:8008. To use https refer the [ssl-setup](/docs/local-ssl-setup.md).
+## Running tests
 
-`python manage.py index_refdata`
+run the tests with command `DJANGO_ENV=unittests python manage.py test --parallel --failfast --keepdb -v 0`
 
-`python manage.py reload_refdata_cache`
+### Running coverage (Docker)
 
-`python manage.py loadinitialdata`
+`docker exec -it -e DJANGO_ENV=unittests $(docker ps -q -f name="metax-web*") coverage run manage.py test --parallel`
 
-`python manage.py loaddata metax_api/tests/testdata/test_data.json`
+### Generating coverage report
 
-Metax api is available from your browser at http://localhost:8008
+cli-report:`docker exec -it $(docker ps -q -f name="metax-web*") coverage report`
 
 ## Running tests
 
-run the tests with command `DJANGO_ENV=test python manage.py test --parallel --failfast --keepdb -v 0`
-
-
-
-
-
+Run all tests with command `DJANGO_ENV=unittests python manage.py test --parallel --failfast`

docker-compose.yml

@@ -54,6 +54,13 @@ services:
     volumes:
       - metax-rabbitmq:/var/lib/rabbitmq
 
+  metax-auth:
+    image: fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-httpd
+    configs:
+      - source: metax-httpd-config
+        target: /usr/local/apache2/conf/extra/httpd-metax.conf
+    hostname: metax-auth
+
   fairdata-nginx:
     image: nginx:latest
     configs:
@@ -81,6 +88,8 @@ volumes:
 configs:
   metax-web-config:
     external: True
+  metax-httpd-config:
+    external: True
 
   fairdata-nginx-config:
     external: True

docs/docker-prerequisites.md

@@ -3,7 +3,7 @@
 
 ## Docker-Engine
 
-Install Docker-Engine either following instructions below or looking up your platform specific instructions [from docs.docker.com][1] 
+Install Docker-Engine either following instructions below or looking up your platform specific instructions [from docs.docker.com][https://docs.docker.com/engine/install/] 
 
 ### Linux
 
@@ -27,4 +27,10 @@ You can use portainer container management tool for monitoring various developme
 
 Finish the Portainer setup by logging in at http://localhost:9000, create a local endpoint from the Portainer interface. 
 
-[1]: https://docs.docker.com/engine/install/
+## Dozzle (Optional)
+
+[Dozzle](https://github.com/amir20/dozzle) can be used for reading the container logs:
+
+`docker run --name dozzle -d --volume=/var/run/docker.sock:/var/run/docker.sock -p 8888:8080 amir20/dozzle:latest`
+
+To show stopped/crashed container logs: `settings -> Show stopped containers`

docs/docker-stack.md

@@ -1,46 +1,76 @@
+# Quickstart
+
+1. Install [docker](https://docs.docker.com/engine/install/)
+2. Init swarm: `docker swarm init`
+3. Add docker-configs from `fairdata-secrets` repo
+4. Update `/etc/hosts` with `0.0.0.0 metax.fd-dev.csc.fi`
+5. Deploy with `docker stack deploy -c docker-compose.yml --resolve-image=always --with-registry-auth metax-dev`
+6. Init Metax `docker exec $(docker ps -q -f name=metax-dev_metax-web) python manage.py first_time_setup`
+
 # Local development with Docker-swarm
 
-## Building metax-image
+After installing [Docker prerequisites](docker-prerequisites.md), Initialize the Swarm:
 
-After installing [Docker prerequisites](docker-prerequisites.md), build the metax-web docker image with the following command:
+`docker swarm init`
 
-`docker build -t fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-web .`
+## Running the stack locally
 
-<!-- ## Building httpd-image 
+Append your local `/etc/hosts` file with:
 
-`docker build -t fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-httpd -f containers/apache-image.Dockerfile .` -->
+`0.0.0.0 metax.fd-dev.csc.fi`
 
-## Pushing metax-image to Artifactory
+The default stack requires docker-configurations that can be found from `fairdata-secrets` repository. If there is no
+access to that repo, refer the next section. When the required configurations are created, the stack can be deployed
+from the repository root with:
 
- `docker push fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-web`
+`docker stack deploy -c docker-compose.yml --resolve-image=always --with-registry-auth metax-dev`
 
-<!-- ## Pushing httpd-image to Artifactory
+After all the services has been started, Metax is available from `metax.fd-dev.csc.fi`. This stack contains the common
+Fairdata nginx proxy.
 
-`docker push fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-httpd` -->
+__NOTE__: Docker for Mac has a bug that prevents the required configurations to be deployed to metax-web container. To
+work around this, refer the `Required environmental variables` section in [README](/README.md)
 
+## Running the stack without predefined docker-configs
 
-## Running the stack locally
+Development setup without pre-required docker-configurations can be run with:
 
-In the repository root, run
+`docker stack deploy -c config-swap-stack.yml --resolve-image=always --with-registry-auth metax-dev`
 
-`docker stack deploy -c docker-compose.yml --resolve-image=always --with-registry-auth metax-dev`
+After all the services has been started, Metax is available from `0.0.0.0:8008`. This stack allows the environment
+variables to be changed without re-deploying the whole stack. To add certain docker-config to metax-web container:
 
-This stack contains the common Fairdata nginx proxy.
+`docker service update --config-add source=<CONFIG-NAME>,target=/code/metax_api/settings/.env metax-dev_metax-web`
 
-## Running the stack without predefined docker-configs
+To change existing configuration in metax-web container:
 
-`docker stack deploy -c config-swap-stack.yml --resolve-image=always --with-registry-auth metax-dev`
+`docker service update --config-rm <CONFIG-NAME> --config-add source=<NEW-CONFIG-NAME>,target=/code/metax_api/settings/.env metax-dev_metax-web`
 
-## Running Metax management commands
+## Building related images
 
-To run Metax management commands, locate the running metax-dev_metax-web container and open terminal inside it with:
+Docker image for metax-web can be built with the following command:
 
-`docker exec -it <container-name> bash`
+`docker build -t fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-web .`
+
+Image for metax-httpd can be build with:
+
+`docker build -t fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-httpd -f httpd.dockerfile .`
+
+## Pushing images to Artifactory
+
+Ensure that you are logged in to Artifactory:
 
-## Adding docker-config to the stack
+`docker login fairdata-docker.artifactory.ci.csc.fi`
 
-`docker service update --config-add source=metax-web-stable-config,target=/code/metax_api/settings/.env metax-dev_metax-web`
+Push commands for metax-web and metax-httpd respectively:
 
-## Swapping docker-config in the stack
+`docker push fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-web`
 
-`docker service update --config-rm <docker-config-name> --config-add source=<docker-config-name>,target=/code/metax_api/settings/.env metax-dev_metax-web`
+`docker push fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-httpd`
+
+## Running Metax management commands
+
+To run Metax management commands, locate the running metax-dev_metax-web container (can be done with `docker container ls`)
+and open terminal inside it with:
+
+`docker exec -it <container-name> bash`

docs/manage-commands.md

@@ -1,5 +1,9 @@
 # Metax management commands
 
+## Run all relevant commands for first time setup
+
+`python manage.py first_time_setup`
+
 ## Create and migrate database
 
 `python manage.py migrate`
@@ -22,12 +26,14 @@
 
 ## Run all tests
 
-`DJANGO_ENV=test python manage.py test --parallel --failfast --keepdb -v 0`
+`DJANGO_ENV=unittests python manage.py test --parallel --failfast`
 
 ## Inspect current application settings
 
 `python manage.py diffsettings --output unified --force-color`
 
 ## Execute management commands against docker swarm metax-api container
 
+This command assumes the default name `metax-dev` for the stack.
+
 `docker exec $(docker ps -q -f name=metax-dev_metax-web) python manage.py check`

httpd.dockerfile

@@ -0,0 +1,7 @@
+FROM httpd:2.4
+
+RUN apt-get update && apt-get install -y ca-certificates libapache2-mod-auth-openidc
+
+RUN echo "Include /usr/local/apache2/conf/extra/httpd-metax.conf" >> /usr/local/apache2/conf/httpd.conf
+
+EXPOSE 8080

pyproject.toml

@@ -41,6 +41,7 @@ django-debug-toolbar = "^3.2"
 PyJWT = "^2.0.1"
 ipdb = "^0.13.7"
 isort = "^5.8.0"
+coverage = {version = "^5.5", extras = ["toml"]}
 
 [tool.poetry.extras]
 simplexquery = ["python-simplexquery"]