Skip to content

Commit

Permalink
Replace Camunda with Flowable
Browse files Browse the repository at this point in the history
  • Loading branch information
mario-winkler committed Oct 4, 2024
1 parent b68212c commit 4f8820f
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 11 deletions.
13 changes: 5 additions & 8 deletions docs/design/harvester/design.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,15 @@
# Harvester

The harvester is implemented as workflows in the Camunda BPMN platform. Camunda Platform is a flexible framework for workflow and process automation. Its core is a native Business Process Modelling Notation (BPMN) 2.0 process engine that runs inside the Java Virtual Machine. Workflows need to be defined with the BPMN specification that supports parallel workflow steps, branching, error handling, etc. A workflow step can be set to type “external” to allow an individual worker to fetch tasks for a specific workflow step. This worker connects to the REST API, fetches tasks, and returns the status (e.g., completed, failed, error). Failed workflow steps can be automatically retried from the Camunda process engine. Errors can be modelled in the BPMN diagram, e.g., asking an operator how to proceed as a manual user task.
The harvester is implemented as workflows in the Flowable BPMN platform. Flowable Platform is a flexible framework for workflow and process automation. Its core is a native Business Process Modelling Notation (BPMN) 2.0 process engine that runs inside the Java Virtual Machine. Workflows need to be defined with the BPMN specification that supports parallel workflow steps, branching, error handling, etc. A workflow step can be set to type “external” to allow an individual worker to fetch tasks for a specific workflow step. This worker connects to the REST API, fetches tasks, and returns the status (e.g., completed, failed, error). Failed workflow steps can be automatically retried from the Flowable process engine. Errors can be modelled in the BPMN diagram, e.g., asking an operator how to proceed as a manual user task.

The Camunda platform has the ability to separate the workflow engine from the workflow step execution, where the central workflow engine takes care of the workflow step orchestration. The workflow steps are implemented as Camunda Python Worker, allowing a seamless usage of various EO related upstream packages provided by Python's geospatial community.
The Flowable platform has the ability to separate the workflow engine from the workflow step execution, where the central workflow engine takes care of the workflow step orchestration. The workflow steps are implemented as Flowable Python Worker, allowing a seamless usage of various EO related upstream packages provided by Python's geospatial community.

On top of the workflow engine, a stack of tools for operations and monitoring are available. The following tools are used for the harvester:

- Camunda Engine as the core component responsible for executing BPMN workflows
- Flowable Engine as the core component responsible for executing BPMN workflows
- REST API provides remote access to running processes or to start processes
- Camunda Modeler as a standalone desktop application that allows users and developers to design and configure a workflow
- Camunda Cockpit as a web application tool for process operations
- Camunda Admin as a web application for managing users, groups, and their access permissions.
- Camunda Tasklist as a web application for managing and completing operator tasks in the context of processes.
- Camunda Python Worker as a Python library to communicate with the Camunda REST API, which can be launched anywhere with access to the Camunda REST API.
- Flowable Design as a web application that allows users and developers to design and configure a workflow
- Flowable External Client Python as a Python library to communicate with the Flowable REST API, which can be launched anywhere with access to the Flowable REST API.

Each workflow can be triggered by the Registration API and can utilize the Resource Discovery API for the registration of the harvested resources.

Expand Down
4 changes: 2 additions & 2 deletions docs/design/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ The components design of the Resource Registration Building Block is shown in th

The Registration API uses pygeoapi to publish processes according to the OGC API Processes specification. Processes for resource creation, update, and deletion are implemented for all resource types available in the resource catalogue. The processes make use of the OWSLib, components of the EOEPCA registrar as well as the newly combined and developed data ingestion library.

The Harvester component is built with workflows in Camunda allowing to easily operate the workflows conducted. Python is used to communicate with the Camunda core engine via REST API and to execute the workflow steps. The execution of the tasks for metadata harvesting, data download, and metadata creation is based on various software, such as EOX View Server’s harvester as well as OWSLib, pygeometa, EOMetadataTool, and stactools. A common data ingestion library is being developed to combine the functionalities from those various packages.
The Harvester component is built with workflows in Flowable allowing to easily operate the workflows conducted. Python is used to communicate with the Flowable core engine via REST API and to execute the workflow steps. The execution of the tasks for metadata harvesting, data download, and metadata creation is based on various software, such as EOX View Server’s harvester as well as OWSLib, pygeometa, EOMetadataTool, and stactools. A common data ingestion library is being developed to combine the functionalities from those various packages.

As part of the workflows in Camunda, notifications can be sent to the notification & automation building block and workflows can listen for notifications as well.
As part of the workflows in Flowable, notifications can be sent to the notification & automation building block and workflows can listen for notifications as well.

All APIs are protected with the API Gateway, which connects to the Identity Management building block for authentication and authorization. In addition, the API Gateway is able to conduct rate-limiting.
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ nav:
- API:
- design/registration-api/api/endpoint-specification.md
- design/registration-api/api/usage.md
- Harvester (Camunda):
- Harvester:
- Design: design/harvester/design.md
- API:
- design/harvester/api/endpoint-specification.md
Expand Down

0 comments on commit 4f8820f

Please sign in to comment.