diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst index a0465cade..e7bd58e97 100644 --- a/docs/source/configuration.rst +++ b/docs/source/configuration.rst @@ -1,83 +1,57 @@ .. _configuration: -# TODO: update all this - - ****************** -Configuration File +Configuration ****************** .. contents:: :local: :depth: 2 -After you have installed weaver you can customize the default weaver configuration by editing -the ``weaver/config/weaver.ini`` configuration file. This configuration file overwrites the default settings in the ``buildout.cfg``: +After you have installed `Weaver`, you can customize its behaviour using multiple configuration settings. -.. code-block:: cfg - $ vim custom.cfg - $ cat custom.cfg - [buildout] - extends = buildout.cfg +Configuration Settings +======================================= + +All settings are configured using a ``weaver.ini`` configuration file. An `weaver.ini.example`_ file is provided +with default values to help in the configuration process. - [settings] - hostname = localhost - http-port = 8083 - https-port = 5000 - log-level = WARN - username = - password = - workdir = - ows-security = true - ows-proxy = true - rpcinterface = true - wps = true - wps-cfg = /path/to/my/default/pywps.cfg +.. todo:: complete docs -After your have made a change in ``custom.cfg`` you *need to update* the installation and restart the weaver service: -.. code-block:: sh - $ make update - $ make restart - $ make status +Configuration of Data Sources +======================================= -Set hostname and port -===================== +.. todo:: complete docs -Edit the options ``hostname``, ``http-port`` and ``https-port``. +`data_sources.json.example`_ -Activate basic-auth for XML-RPC control interface -================================================= +Configuration of WPS Processes +======================================= -Set ``username`` and ``password``. +.. todo:: complete docs -Configure the default WPS configuration -======================================= +`wps_processes.yml.example`_ -Edit the ``wps-cfg`` option to set the default PyWPS configuration for the capabilities of the internal WPS (PyWPS) application. +Starting the Application +======================================= -Deactivate weaver components -============================== +.. todo:: complete docs -weaver has four components which by default are activated: +``make start`` (or similar command) -ows-security - The OWS security wsgi middleware -ows-proxy - A proxy wsgi application for OWS services -rpcinterface - An XML-RPC interface to control token generation and service registration -wps - An internal WPS wsgi application (PyWPS) +- need to start ``gunicorn/pserve`` (example `Dockerfile-manager`_) +- need to start ``celery`` worker (example `Dockerfile-worker`_) -By setting a component option to ``false`` you can deactivate it: -.. code-block:: sh - [settings] - ows-proxy = false +.. _weaver.ini.example: ../../../config/weaver.ini.example +.. _data_sources.json.example: ../../../config/data_sources.json.example +.. _wps_processes.yml.example: ../../../config/wps_processes.yml.example +.. _Dockerfile-manager: ../../../docker/Dockerfile-manager +.. _Dockerfile-worker: ../../../docker/Dockerfile-worker diff --git a/docs/source/tutorial.rst b/docs/source/tutorial.rst index 508dc1805..8e27d9896 100644 --- a/docs/source/tutorial.rst +++ b/docs/source/tutorial.rst @@ -12,13 +12,17 @@ Tutorial Using the WPS application included in Weaver ============================================== -Install Weaver (see: :ref:`installation`) and make sure it is started with ``make start`` (or similar command). +Install Weaver (see: :ref:`installation`) and make sure all required components +are started and running (see: :ref:`configuration`). Then, execute the desired `WPS`_ request according to desired operation mode and version. +For all following examples, ``${WEAVER_URL}`` is used to specify your application URL endpoint configuration. +By default, this value should be ``localhost:4001``. + .. _`WPS`: https://www.opengeospatial.org/standards/wps -WPS-1/WSP-2 requests +WPS-1/2 requests -------------------- Specifying the appropriate ``version=<1.0.0|2.0.0>`` parameter in the URL as required. @@ -27,7 +31,7 @@ Run a WPS-1/WPS-2 ``GetCapabilities`` request: .. code-block:: sh - $ curl -k "/ows/wps?service=wps&request=getcapabilities" + $ curl -k "${WEAVER_URL}/ows/wps?service=wps&request=getcapabilities" You should receive an XML response listing service details and available processes. @@ -35,42 +39,81 @@ Run a WPS-1/WPS-2 ``DescribeProcess`` request (built-in process ``jsonarray2netc .. code-block:: sh - $ curl -k "/ows/wps?service=wps&request=describeprocess&identifier=jsonarray2netcdf&version=1.0.0" + $ curl -k "${WEAVER_URL}/ows/wps?service=wps&request=describeprocess&identifier=jsonarray2netcdf&version=1.0.0" This will provide you with an XML response listing the specific process details such and inputs/outputs and description. -Run a WPS-1/WPS-2 ``Execute`` request (built-in process ``jsonarray2netcdf``): +We can now use the process to execute a WPS request. To do so, we will need some input data files to call it. +First, let's create a JSON file with some *dummy* NetCDF file reference for demonstration purpose. +.. code-block:: sh -# TODO: complete this demo + $ echo 'Test WPS' > /tmp/test.nc + $ echo '["file:///tmp/test.nc"]' > /tmp/test.json +Then, run the WPS-1/WPS-2 ``Execute`` request (built-in process ``jsonarray2netcdf``) as follow: .. code-block:: sh - $ curl -k "/ows/wps?service=wps&request=execute&identifier=jsonarray2netcdf&version=1.0.0" + $ curl -k "${WEAVER_URL}/ows/wps?service=wps&request=execute&identifier=jsonarray2netcdf&version=1.0.0 \ + &DataInputs=input=file:///tmp/test.json" +The execution of the process should read the JSON list with our dummy NetCDF file and make it available (as a copy) +on the output parameter named ``output`` with a path matching the configured output WPS path of the application. WPS-3 requests -------------- -# TODO: demo +All previous operations for listing available processes (``GetCapabilities``), describing or executing a WPS-1/2 +process can also be accomplished using the WPS-3 REST JSON interface. For instance, listing processes is done like so: + +.. code-block:: sh + + $ curl -k "${WEAVER_URL}/processes" + +Individual process details (``DescribeProcess``) can be obtained with the following method +(e.g.: built-in process ``jsonarray2netcdf`` in this case): + +.. code-block:: sh + + $ curl -k "${WEAVER_URL}/processes/jsonarray2netcdf" + + +And execution of this process can be accomplished with the following request: .. code-block:: sh + $ curl -X POST "${WEAVER_URL}/processes/jsonarray2netcdf/jobs" \ + -H "Content-Type: application/json" \ + -d '{"inputs": [{"id": "input", "href": "file:///tmp/test.json"}], + "outputs": [{"id": "output", "transmissionMode": "reference"}], + "response": "document", + "mode": "async"}' + + +The JSON response should provide a ``location`` field specifying where the job status can be verified. +Upon *successful* job completion, an ``output`` reference URL should have been generated just as with +the WPS-1/2 example. + + +The WPS-3 interface allows further operations such as job monitoring, specific output listing, log reporting, etc. +For all available operations and specific details about them, please refer to the *OpenAPI* schemas that will be +rendered on route ``${WEAVER_URL}/api`` when running `Weaver` application. + Managing WPS processes included in Weaver ADES/EMS ================================================== -Register a WPS service ----------------------- +Register a new WPS process +-------------------------- -# TODO: demo +.. todo:: complete demo docs -Access a registered service +Access a registered process --------------------------- -# TODO: demo +.. todo:: complete demo docs, stuff about process visibility diff --git a/requirements.txt b/requirements.txt index d8f1effaa..f36d8c7b0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,6 +6,7 @@ colander cornice cornice_swagger>=0.7.0 cwltool==1.0.20180820141117 +#cwltool==2.0.20200122124526; python_version >= "3.5" duration flufl.enum; python_version < "3" # gunicorn >20 breaks some config.ini loading parameters (paste) @@ -26,7 +27,10 @@ pywps pyyaml>=4.2b4 requests requests_file -ruamel.yaml>=0.15.78 +# let cwltool define ruamel.yaml version (<=16.5) +# ensure minimal 0.15.78 to solve install issue +# (https://bitbucket.org/ruamel/yaml/issues/261/error-while-installing-ruamelyaml-setuppy) +#ruamel.yaml>=0.15.78,<16 shapely six simplejson diff --git a/weaver/processes/utils.py b/weaver/processes/utils.py index 279a6290f..5446d6027 100644 --- a/weaver/processes/utils.py +++ b/weaver/processes/utils.py @@ -40,10 +40,8 @@ log_unhandled_exceptions ) from weaver.formats import CONTENT_TYPE_APP_JSON, CONTENT_TYPE_TEXT_PLAIN -from weaver.processes import wps_package from weaver.processes.constants import WPS_COMPLEX_DATA from weaver.processes.types import PROCESS_APPLICATION, PROCESS_WORKFLOW -from weaver.processes.wps_package import complex2json as jsonify_value from weaver.store.base import StoreProcesses from weaver.utils import get_sane_name, get_settings, get_url_without_query from weaver.wps_restapi import swagger_definitions as sd @@ -164,6 +162,8 @@ def convert_process_wps_to_db(service, process, container): """ Converts an owslib WPS Process to local storage Process. """ + from weaver.processes.wps_package import complex2json as jsonify_value + describe_process_url = "{base_url}/providers/{provider_id}/processes/{process_id}".format( base_url=get_wps_restapi_base_url(container), provider_id=service.get("name"), @@ -222,9 +222,10 @@ def _check_deploy(payload): is_request=False) def _get_deploy_process_info(process_info, reference, package): """Obtain the process definition from deploy payload with exception handling.""" + from weaver.processes.wps_package import get_process_definition try: # data_source `None` forces workflow process to search locally for deployed step applications - return wps_package.get_process_definition(process_info, reference, package, data_source=None) + return get_process_definition(process_info, reference, package, data_source=None) except PackageNotFound as ex: # raised when a workflow sub-process is not found (not deployed locally) raise HTTPNotFound(detail=str(ex))