Merge branch 'stable' into 'master'

21/10/1 Production update

* Update mime-types in reference-data
* New endpoint for counting project files and counting files in multiple projects
* Make critical RPC-API operations atomic
* PAS-templates can now be fetched from the API
* Support filtering datasets by project
* Remove editor database model
* Deleted objects are now stored to database as JSON-blobs
* Refactor error-handling API to save errors on database
* Add management command to fix invalid file byte size values
* Swagger and docs can now be generated at runtime
* Improve various reference-data entries
* Improve organization reference-data update process
* Format the entire codebase with black-formatter
* Remove duplicate tests and tests failing on global state
* Add Matomo-integration to /swagger and /docs endpoints
* Refactor docs and swagger generation
* Major improvements on enduser documentation
* Change Matomo URL to new metrics.fairdata.fi
* Fix invalid authentication checks for anonymous users
* Improve actor filter for organizations
* Make django-debug-toolbar optional on runtime when DEBUG is true
* Fix PAS-template incorrect datacatalog

See merge request fairdata/fairdata-metax!86

.gitignore

@@ -6,6 +6,7 @@
 /src/__pycache__/*
 /src/static/*
 /src/.coverage
+/src/coverage_data
 /swagger/*index.html
 /swagger/*/index.html
 .idea/
@@ -18,6 +19,7 @@ ubuntu-xenial-16.04-cloudimg-console.log
 !.env.template
 *.pem
 /src/log
+/src/metax_api/templates/swagger
 
 # Unit test / coverage reports
 htmlcov/

.gitlab-ci.yml

@@ -49,11 +49,12 @@ clean_gitlab_env:
       when: manual
     - when: never
 
-update_proxy:
+update_metax:
   stage: update
   environment: $CI_COMMIT_REF_NAME
   script:
     - ansible-playbook -i $ANSIBLE_INVENTORY $UPDATE_PROXY_PLAYBOOK -e "build_id=$CI_COMMIT_SHORT_SHA"
+    - ansible-playbook -i $ANSIBLE_INVENTORY $MANAGE_PLAYBOOK -e "build_id=$CI_COMMIT_SHORT_SHA"
   rules:
     - if: $CI_COMMIT_BRANCH =~ /^(demo|stable|staging|test)$/
       when: always

Dockerfile

@@ -7,10 +7,11 @@ RUN mkdir -p /var/log/metax-api/errors && touch /var/log/metax-api/metax-api.jso
 
 RUN apt-get update && apt install xqilla libxerces-c-dev build-essential libssl-dev libffi-dev python-dev libxqilla-dev -y
 
-RUN pip install --upgrade pip wheel poetry
-COPY pyproject.toml poetry.lock /code/
+RUN pip install --upgrade pip wheel
+COPY requirements.txt /code/
 WORKDIR /code
-RUN poetry config virtualenvs.create false && poetry install --no-interaction --no-ansi --extras "docs simplexquery"
+
+RUN pip install -r requirements.txt
 
 EXPOSE 8008
 EXPOSE 8006

ENV_VARS.md

@@ -14,6 +14,7 @@ copy .env.template to .env and fill the required values from below table. Requir
 | DATACITE_URL                            | yes      |                                                                                       |
 | DATACITE_USERNAME                       | yes      |                                                                                       |
 | DEBUG                                   | no       | False                                                                                 |
+| DEBUG_TOOLBAR_ENABLED                   | not      | True                                                                                  | Enable debug toolbar on local environment                                                                  |
 | DJANGO_ENV                              | no       | local                                                                                 | Specifies the environment, corresponds with the environments found in src/metax_api/settings/environments/ |
 | DJANGO_SECRET_KEY                       | yes      |                                                                                       |
 | ELASTIC_SEARCH_HOSTS                    | no       | localhost                                                                             | Elastic Search instance IPs                                                                                |

README.md

@@ -16,11 +16,12 @@ The recommended way to run the development setup is to use [Docker-swarm setup](
 
 Install [Poetry](https://python-poetry.org/docs/) for your OS. Navigate to the repository root and run command `poetry install`. this will create and activate new Python virtualenv, installing all necessary Python packages to it.
 
-You can generate traditional requirements.txt file with `poetry export --dev -E simplexquery --without-hashes -f requirements.txt --output requirements.txt`
+
+You can generate traditional requirements.txt file with `poetry export --dev -E simplexquery -E docs -E swagger --without-hashes -o requirements.txt`
 
 ### Managing dependencies
 
-__NOTICE: Please remember to execute `poetry export --dev -E simplexquery --without-hashes -f requirements.txt --output requirements.txt` after any additions, updates or removals.__
+__NOTICE: Please remember to generate the requirements.txt file after any additions, updates or removals using the command above.__
 
 Developer dependencies can be added with command `poetry add -D <package>`
 Application dependencies can be added with command `poetry add <package>`
@@ -56,7 +57,11 @@ run the tests with command `DJANGO_ENV=unittests python manage.py test --paralle
 
 ### Running coverage (Docker)
 
-`docker exec -it -e DJANGO_ENV=unittests $(docker ps -q -f name="metax-web*") coverage run manage.py test --parallel`
+Collect data: `docker exec -it -e DJANGO_ENV=unittests $(docker ps -q -f name="metax-web*") coverage run manage.py test --parallel`
+
+Combine it: `docker exec -it $(docker ps -q -f name="metax-web*") coverage combine`
+
+Report: `docker exec -it $(docker ps -q -f name="metax-web*") coverage report`
 
 ### Generating coverage report
 

config-swap-stack.yml

@@ -51,6 +51,17 @@ services:
     volumes:
       - metax-rabbitmq:/var/lib/rabbitmq
 
+  metax-docs:
+    image: fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-sphinx
+    hostname: metax-docs
+    environment:
+      DOMAIN: 0.0.0.0:8008
+      REMS_ENABLED: "true"
+    ports:
+      - 8088:8000
+    volumes:
+      - ./docs/api:/sphinx
+
 volumes:
   metax-rabbitmq:
     external: true

docker-compose.yml

@@ -61,6 +61,17 @@ services:
         target: /usr/local/apache2/conf/extra/httpd-metax.conf
     hostname: metax-auth
 
+  metax-docs:
+    image: fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-sphinx
+    hostname: metax-docs
+    environment:
+      DOMAIN: metax.fd-dev.csc.fi
+      REMS_ENABLED: "true"
+    ports:
+      - 8088:8000
+    volumes:
+      - ./docs/api:/sphinx
+
   fairdata-nginx:
     image: nginx:latest
     configs:

docs/api/README.md

@@ -0,0 +1,47 @@
+# Developing documentation
+
+Metax has API documentation implemented with Swagger and Sphinx. Below are instructions how they can be improved.
+
+## Swagger
+
+The openapi/swagger specification version 2 is used to document the Metax REST API.
+Rest api descriptions are stored in the repository in /metax_api/swagger/v1/swagger.yaml
+ and /metax_api/swagger/v2/swagger.yaml, depending on the interface version. 
+From both yaml files, the corresponding html files are generated in the /metax_api/templates/swagger/ directory when Metax starts.
+
+Swagger documentation can be edited directly in PyCharm and VS Code. There are good openapi plugins for both. One good plugin for both is the OpenAPI (Swagger) Editor. Another option is to use [Swagger editor](https://editor.swagger.io).
+
+[VS Code plugin](https://marketplace.visualstudio.com/items?itemName=42Crunch.vscode-openapi&ssr=false#review-details)
+
+[PyCharm plugin](https://plugins.jetbrains.com/plugin/14837-openapi-swagger-editor)
+
+Although OpenApi is allowed to edit in json and yaml formats, in the case of Metax it has to be done in yaml format because otherwise the conversion to html format will not be possible.
+A good starting point for studying OpenApi is [OpenApi specification V2](https://swagger.io/specification/v2/)
+
+## Sphinx
+
+The repository provides a Sphinx autobuild server in Docker container for conveniently write and develop the API documentation.
+Below are the instructions how to use the server.
+
+### Building the image
+
+The autobuild server can be built with following command from repo root:
+
+`docker build -t fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-sphinx -f sphinx.dockerfile .`
+
+### Running the server in standalone container
+
+The server can be run with the following command, also from repo root:
+
+`docker run -it -v $PWD/docs/api:/sphinx/ -p 8088:8000 fairdata-docker.artifactory.ci.csc.fi/fairdata-metax-sphinx`
+
+### Running the server in stack
+
+The autobuild server is also present in both of the stacks provided in the repo. The default dev env domain name `metax.fd-dev.csc.fi`is used in the documentation
+so it should be added to `/etc/hosts` file to enable correct redirection in documentation links. When the domain name is added, the server is available from
+`http://metax.fd-dev.csc.fi:8088`. By disabling browser cache redirection errors can be prevented. If all other fail, `0.0.0.0:8088` should work. 
+
+### Additional notes
+
+To conditionally add parts of the documentation, use only -directive. See [This](https://github.com/sphinx-doc/sphinx/issues/1115) for known issue with this
+directive and headings.

docs/v1/source/conf.py → docs/api/conf.py

@@ -12,22 +12,25 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
+import os
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))
 
 
 # -- Project information -----------------------------------------------------
 
-project = 'Metax API'
-copyright = '2018, csc.fi'
-author = 'csc.fi'
+project = "Metax API"
+copyright = "2020, csc.fi"
+author = "csc.fi"
 
 # The short X.Y version
-version = ''
+version = ""
 # The full version, including alpha/beta/rc tags
-release = ''
+release = ""
 
+domain = os.getenv("DOMAIN", "metax.fairdata.fi")
+branch = os.getenv("BRANCH", "master")
+etsin_url = os.getenv("ETSIN_URL", "etsin.fairdata.fi")
 
 # -- General configuration ---------------------------------------------------
 
@@ -38,20 +41,18 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = [
-]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
 # source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+source_suffix = ".rst"
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = "index"
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -66,41 +67,50 @@
 exclude_patterns = []
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
 
+replacements = {
+    "__METAX_ENV_DOMAIN__": f"{domain}",
+    "__METAX_ENV_BRANCH__": f"{branch}",
+    "__ETSIN_ENV_BASE_URL__": f"{etsin_url}"
+}
+
+tags = {
+    'rems_enabled': os.getenv("REMS_ENABLED", 'False')
+}
 
 # -- Options for HTML output -------------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'sphinx_rtd_theme'
+html_theme = "sphinx_rtd_theme"
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
 # html_theme_options = {}
 html_theme_options = {
-    'canonical_url': '',
-    'analytics_id': '',
-    'logo_only': False,
-    'display_version': True,
-    'prev_next_buttons_location': 'bottom',
-    'style_external_links': False,
+    "canonical_url": "",
+    "analytics_id": "",
+    "logo_only": False,
+    "display_version": True,
+    "prev_next_buttons_location": "bottom",
+    "style_external_links": False,
     # Toc options
-    'collapse_navigation': True,
-    'sticky_navigation': True,
-    'navigation_depth': -1,
-    'includehidden': True,
-    'titles_only': False
+    "collapse_navigation": True,
+    "sticky_navigation": True,
+    "navigation_depth": -1,
+    "includehidden": True,
+    "titles_only": False,
 }
 
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ["_static"]
 
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.
@@ -116,7 +126,7 @@
 # -- Options for HTMLHelp output ---------------------------------------------
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'MetaxAPIdoc'
+htmlhelp_basename = "MetaxAPIdoc"
 
 
 # -- Options for LaTeX output ------------------------------------------------
@@ -125,15 +135,12 @@
     # The paper size ('letterpaper' or 'a4paper').
     #
     # 'papersize': 'letterpaper',
-
     # The font size ('10pt', '11pt' or '12pt').
     #
     # 'pointsize': '10pt',
-
     # Additional stuff for the LaTeX preamble.
     #
     # 'preamble': '',
-
     # Latex figure (float) alignment
     #
     # 'figure_align': 'htbp',
@@ -143,19 +150,15 @@
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'MetaxAPI.tex', 'Metax API Documentation',
-     'csc.fi', 'manual'),
+    (master_doc, "MetaxAPI.tex", "Metax API Documentation", "csc.fi", "manual"),
 ]
 
 
 # -- Options for manual page output ------------------------------------------
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'metaxapi', 'Metax API Documentation',
-     [author], 1)
-]
+man_pages = [(master_doc, "metaxapi", "Metax API Documentation", [author], 1)]
 
 
 # -- Options for Texinfo output ----------------------------------------------
@@ -164,10 +167,29 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'MetaxAPI', 'Metax API Documentation',
-     author, 'MetaxAPI', 'One line description of project.',
-     'Miscellaneous'),
+    (
+        master_doc,
+        "MetaxAPI",
+        "Metax API Documentation",
+        author,
+        "MetaxAPI",
+        "Part of the Fairdata services.",
+        "Miscellaneous",
+    ),
 ]
 
+# from https://github.com/sphinx-doc/sphinx/issues/4054
+def replace(app, docname, source):
+    for key, value in app.config.replacements.items():
+        source[0] = source[0].replace(key, value)
+
+def add_tags(app):
+    for key, value in tags.items():
+        if value.lower() in ('true'):
+            app.tags.add(key)
+
 def setup(app):
-    app.add_stylesheet('custom.css')
+    app.add_config_value('replacements', {}, True)
+    add_tags(app)
+    app.connect('source-read', replace)
+    app.add_css_file("custom.css")