more linting fixes + prepare basic doc updates (#61)

docs/source/configuration.rst

@@ -1,83 +1,57 @@
 .. _configuration:
 
-# TODO: update all this
-
-
 ******************
-Configuration File
+Configuration
 ******************
 
 .. contents::
     :local:
     :depth: 2
 
-After you have installed weaver you can customize the default weaver configuration by editing
-the ``weaver/config/weaver.ini`` configuration file. This configuration file overwrites the default settings in the ``buildout.cfg``:
+After you have installed `Weaver`, you can customize its behaviour using multiple configuration settings.
 
-.. code-block:: cfg
 
-   $ vim custom.cfg
-   $ cat custom.cfg
-   [buildout]
-   extends = buildout.cfg
+Configuration Settings
+=======================================
+
+All settings are configured using a ``weaver.ini`` configuration file. An `weaver.ini.example`_ file is provided
+with default values to help in the configuration process.
 
-   [settings]
-   hostname = localhost
-   http-port = 8083
-   https-port = 5000
-   log-level = WARN
-   username =
-   password =
-   workdir =
-   ows-security = true
-   ows-proxy = true
-   rpcinterface = true
-   wps = true
-   wps-cfg = /path/to/my/default/pywps.cfg
+.. todo:: complete docs
 
-After your have made a change in ``custom.cfg`` you *need to update* the installation and restart the weaver service:
 
-.. code-block:: sh
 
-   $ make update
-   $ make restart
-   $ make status
+Configuration of Data Sources
+=======================================
 
-Set hostname and port
-=====================
+.. todo:: complete docs
 
-Edit the options ``hostname``, ``http-port`` and ``https-port``.
+`data_sources.json.example`_
 
 
-Activate basic-auth for XML-RPC control interface
-=================================================
+Configuration of WPS Processes
+=======================================
 
-Set ``username`` and ``password``.
+.. todo:: complete docs
 
 
-Configure the default WPS configuration
-=======================================
+`wps_processes.yml.example`_
 
-Edit the ``wps-cfg`` option to set the default PyWPS configuration for the capabilities of the internal WPS (PyWPS) application.
 
+Starting the Application
+=======================================
 
-Deactivate weaver components
-==============================
+.. todo:: complete docs
 
-weaver has four components which by default are activated:
+``make start`` (or similar command)
 
-ows-security
-   The OWS security wsgi middleware
-ows-proxy
-   A proxy wsgi application for OWS services
-rpcinterface
-   An XML-RPC interface to control token generation and service registration
-wps
-   An internal WPS wsgi application (PyWPS)
+- need to start ``gunicorn/pserve`` (example `Dockerfile-manager`_)
+- need to start ``celery`` worker (example `Dockerfile-worker`_)
 
-By setting a component option to ``false`` you can deactivate it:
 
-.. code-block:: sh
 
-   [settings]
-   ows-proxy = false
+.. _weaver.ini.example: ../../../config/weaver.ini.example
+.. _data_sources.json.example: ../../../config/data_sources.json.example
+.. _wps_processes.yml.example: ../../../config/wps_processes.yml.example
+.. _Dockerfile-manager: ../../../docker/Dockerfile-manager
+.. _Dockerfile-worker: ../../../docker/Dockerfile-worker

docs/source/tutorial.rst

@@ -12,13 +12,17 @@ Tutorial
 Using the WPS application included in Weaver
 ==============================================
 
-Install Weaver (see: :ref:`installation`) and make sure it is started with ``make start`` (or similar command).
+Install Weaver (see: :ref:`installation`) and make sure all required components
+are started and running (see: :ref:`configuration`).
 
 Then, execute the desired `WPS`_ request according to desired operation mode and version.
 
+For all following examples, ``${WEAVER_URL}`` is used to specify your application URL endpoint configuration.
+By default, this value should be ``localhost:4001``.
+
 .. _`WPS`: https://www.opengeospatial.org/standards/wps
 
-WPS-1/WSP-2 requests
+WPS-1/2 requests
 --------------------
 
 Specifying the appropriate ``version=<1.0.0|2.0.0>`` parameter in the URL as required.
@@ -27,50 +31,89 @@ Run a WPS-1/WPS-2 ``GetCapabilities`` request:
 
 .. code-block:: sh
 
-    $ curl -k "<WEAVER_URL>/ows/wps?service=wps&request=getcapabilities"
+    $ curl -k "${WEAVER_URL}/ows/wps?service=wps&request=getcapabilities"
 
 You should receive an XML response listing service details and available processes.
 
 Run a WPS-1/WPS-2 ``DescribeProcess`` request (built-in process ``jsonarray2netcdf``):
 
 .. code-block:: sh
 
-    $ curl -k "<WEAVER_URL>/ows/wps?service=wps&request=describeprocess&identifier=jsonarray2netcdf&version=1.0.0"
+    $ curl -k "${WEAVER_URL}/ows/wps?service=wps&request=describeprocess&identifier=jsonarray2netcdf&version=1.0.0"
 
 This will provide you with an XML response listing the specific process details such and inputs/outputs and description.
 
-Run a WPS-1/WPS-2 ``Execute`` request (built-in process ``jsonarray2netcdf``):
+We can now use the process to execute a WPS request. To do so, we will need some input data files to call it.
+First, let's create a JSON file with some *dummy* NetCDF file reference for demonstration purpose.
 
+.. code-block:: sh
 
-# TODO: complete this demo
+    $ echo 'Test WPS' > /tmp/test.nc
+    $ echo '["file:///tmp/test.nc"]' > /tmp/test.json
 
+Then, run the WPS-1/WPS-2 ``Execute`` request (built-in process ``jsonarray2netcdf``) as follow:
 
 .. code-block:: sh
 
-    $ curl -k "<WEAVER_URL>/ows/wps?service=wps&request=execute&identifier=jsonarray2netcdf&version=1.0.0"
+    $ curl -k "${WEAVER_URL}/ows/wps?service=wps&request=execute&identifier=jsonarray2netcdf&version=1.0.0 \
+        &DataInputs=input=file:///tmp/test.json"
 
+The execution of the process should read the JSON list with our dummy NetCDF file and make it available (as a copy)
+on the output parameter named ``output`` with a path matching the configured output WPS path of the application.
 
 WPS-3 requests
 --------------
 
-# TODO: demo
+All previous operations for listing available processes (``GetCapabilities``), describing or executing a WPS-1/2
+process can also be accomplished using the WPS-3 REST JSON interface. For instance, listing processes is done like so:
+
+.. code-block:: sh
+
+    $ curl -k "${WEAVER_URL}/processes"
+
+Individual process details (``DescribeProcess``) can be obtained with the following method
+(e.g.: built-in process ``jsonarray2netcdf`` in this case):
+
+.. code-block:: sh
+
+    $ curl -k "${WEAVER_URL}/processes/jsonarray2netcdf"
+
+
+And execution of this process can be accomplished with the following request:
 
 .. code-block:: sh
 
+    $ curl -X POST "${WEAVER_URL}/processes/jsonarray2netcdf/jobs" \
+           -H "Content-Type: application/json" \
+           -d '{"inputs": [{"id": "input", "href": "file:///tmp/test.json"}],
+                "outputs": [{"id": "output", "transmissionMode": "reference"}],
+                "response": "document",
+                "mode": "async"}'
+
+
+The JSON response should provide a ``location`` field specifying where the job status can be verified.
+Upon *successful* job completion, an ``output`` reference URL should have been generated just as with
+the WPS-1/2 example.
+
+
+The WPS-3 interface allows further operations such as job monitoring, specific output listing, log reporting, etc.
+For all available operations and specific details about them, please refer to the *OpenAPI* schemas that will be
+rendered on route ``${WEAVER_URL}/api`` when running `Weaver` application.
+
 
 Managing WPS processes included in Weaver ADES/EMS
 ==================================================
 
-Register a WPS service
-----------------------
+Register a new WPS process
+--------------------------
 
-# TODO: demo
+.. todo:: complete demo docs
 
 
-Access a registered service
+Access a registered process
 ---------------------------
 
-# TODO: demo
+.. todo:: complete demo docs, stuff about process visibility
 
 
 

requirements.txt

@@ -6,6 +6,7 @@ colander
 cornice
 cornice_swagger>=0.7.0
 cwltool==1.0.20180820141117
+#cwltool==2.0.20200122124526; python_version >= "3.5"
 duration
 flufl.enum; python_version < "3"
 # gunicorn >20 breaks some config.ini loading parameters (paste)
@@ -26,7 +27,10 @@ pywps
 pyyaml>=4.2b4
 requests
 requests_file
-ruamel.yaml>=0.15.78
+# let cwltool define ruamel.yaml version (<=16.5)
+# ensure minimal 0.15.78 to solve install issue
+# (https://bitbucket.org/ruamel/yaml/issues/261/error-while-installing-ruamelyaml-setuppy)
+#ruamel.yaml>=0.15.78,<16
 shapely
 six
 simplejson

weaver/processes/utils.py

@@ -40,10 +40,8 @@
     log_unhandled_exceptions
 )
 from weaver.formats import CONTENT_TYPE_APP_JSON, CONTENT_TYPE_TEXT_PLAIN
-from weaver.processes import wps_package
 from weaver.processes.constants import WPS_COMPLEX_DATA
 from weaver.processes.types import PROCESS_APPLICATION, PROCESS_WORKFLOW
-from weaver.processes.wps_package import complex2json as jsonify_value
 from weaver.store.base import StoreProcesses
 from weaver.utils import get_sane_name, get_settings, get_url_without_query
 from weaver.wps_restapi import swagger_definitions as sd
@@ -164,6 +162,8 @@ def convert_process_wps_to_db(service, process, container):
     """
     Converts an owslib WPS Process to local storage Process.
     """
+    from weaver.processes.wps_package import complex2json as jsonify_value
+
     describe_process_url = "{base_url}/providers/{provider_id}/processes/{process_id}".format(
         base_url=get_wps_restapi_base_url(container),
         provider_id=service.get("name"),
@@ -222,9 +222,10 @@ def _check_deploy(payload):
                           is_request=False)
 def _get_deploy_process_info(process_info, reference, package):
     """Obtain the process definition from deploy payload with exception handling."""
+    from weaver.processes.wps_package import get_process_definition
     try:
         # data_source `None` forces workflow process to search locally for deployed step applications
-        return wps_package.get_process_definition(process_info, reference, package, data_source=None)
+        return get_process_definition(process_info, reference, package, data_source=None)
     except PackageNotFound as ex:
         # raised when a workflow sub-process is not found (not deployed locally)
         raise HTTPNotFound(detail=str(ex))