diff --git a/airbyte-integrations/connectors/source-camunda-history/README.md b/airbyte-integrations/connectors/source-camunda-history/README.md new file mode 100644 index 000000000000..15dacfaf0796 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/README.md @@ -0,0 +1,115 @@ +# Camunda History Source + +This is the repository for the Camunda History configuration based source connector. +For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/camunda-history). + +## Local development + + +#### Create credentials +**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/camunda-history) +to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_camunda_history/spec.yaml` file. +Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information. +See `integration_tests/sample_config.json` for a sample config file. + +**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source camunda-history test creds` +and place them into `secrets/config.json`. + +### Locally running the connector docker image + +#### Use `airbyte-ci` to build your connector +The Airbyte way of building this connector is to use our `airbyte-ci` tool. +You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1). +Then running the following command will build your connector: + +```bash +airbyte-ci connectors --name source-camunda-history build +``` +Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-camunda-history:dev`. + +##### Customizing our build process +When contributing on our connector you might need to customize the build process to add a system dependency or set an env var. +You can customize our build process by adding a `build_customization.py` module to your connector. +This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively. +It will be imported at runtime by our build process and the functions will be called if they exist. + +Here is an example of a `build_customization.py` module: +```python +from __future__ import annotations + +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + # Feel free to check the dagger documentation for more information on the Container object and its methods. + # https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/ + from dagger import Container + + +async def pre_connector_install(base_image_container: Container) -> Container: + return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value") + +async def post_connector_install(connector_container: Container) -> Container: + return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value") +``` + +#### Build your own connector image +This connector is built using our dynamic built process in `airbyte-ci`. +The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`. +The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py). +It does not rely on a Dockerfile. + +If you would like to patch our connector and build your own a simple approach would be to: + +1. Create your own Dockerfile based on the latest version of the connector image. +```Dockerfile +FROM airbyte/source-camunda-history:latest + +COPY . ./airbyte/integration_code +RUN pip install ./airbyte/integration_code + +# The entrypoint and default env vars are already set in the base image +# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py" +# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"] +``` +Please use this as an example. This is not optimized. + +2. Build your image: +```bash +docker build -t airbyte/source-camunda-history:dev . +# Running the spec command against your patched connector +docker run airbyte/source-camunda-history:dev spec + +#### Run +Then run any of the connector commands as follows: +``` +docker run --rm airbyte/source-camunda-history:dev spec +docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-camunda-history:dev check --config /secrets/config.json +docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-camunda-history:dev discover --config /secrets/config.json +docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-camunda-history:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json +``` +## Testing + +### Acceptance Tests +Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information. +If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py. +Please run acceptance tests via [airbyte-ci](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#connectors-test-command): +```bash +airbyte-ci connectors --name source-camunda-history test +``` + +## Dependency Management +All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development. +We split dependencies between two groups, dependencies that are: +* required for your connector to work need to go to `MAIN_REQUIREMENTS` list. +* required for the testing need to go to `TEST_REQUIREMENTS` list + +### Publishing a new version of the connector +You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? +1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-camunda-history test` +2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors). +3. Make sure the `metadata.yaml` content is up to date. +4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/camunda-history.md`). +5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention). +6. Pat yourself on the back for being an awesome contributor. +7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. + diff --git a/airbyte-integrations/connectors/source-camunda-history/__init__.py b/airbyte-integrations/connectors/source-camunda-history/__init__.py new file mode 100644 index 000000000000..c941b3045795 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/__init__.py @@ -0,0 +1,3 @@ +# +# Copyright (c) 2023 Airbyte, Inc., all rights reserved. +# diff --git a/airbyte-integrations/connectors/source-camunda-history/acceptance-test-config.yml b/airbyte-integrations/connectors/source-camunda-history/acceptance-test-config.yml new file mode 100644 index 000000000000..8b83865b1981 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/acceptance-test-config.yml @@ -0,0 +1,53 @@ +connector_image: airbyte/source-camunda-history:dev +acceptance_tests: + spec: + tests: + - spec_path: "source_camunda_history/spec.yaml" + connection: + tests: + - config_path: "secrets/config.json" + status: "succeed" + - config_path: "integration_tests/invalid_config.json" + status: "failed" + discovery: + tests: + - config_path: "secrets/config.json" + basic_read: + tests: + - config_path: "secrets/config.json" + configured_catalog_path: "integration_tests/configured_catalog.json" + empty_streams: + - name: "batch" + bypass_reason: "Endpoint not allowed in integration account" + - name: "decision-instance" + bypass_reason: "Endpoint not allowed in integration account" + - name: "task-startedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "task-finishedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "case-instance-closedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "case-instance-createdAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "process-instance-startedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "activity-instance-startedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "process-instance-finishedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "activity-instance-finishedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "case-activity-instance-endedAfter" + bypass_reason: "Endpoint not allowed in integration account" + - name: "case-activity-instance-createdAfter" + bypass_reason: "Endpoint not allowed in integration account" + incremental: + tests: + - config_path: "secrets/config.json" + configured_catalog_path: "integration_tests/configured_catalog.json" + future_state: + future_state_path: "integration_tests/abnormal_state.json" + full_refresh: + tests: + - config_path: "secrets/config.json" + configured_catalog_path: "integration_tests/configured_catalog.json" diff --git a/airbyte-integrations/connectors/source-camunda-history/icon.svg b/airbyte-integrations/connectors/source-camunda-history/icon.svg new file mode 100644 index 000000000000..fdf3ba8b7c67 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/icon.svg @@ -0,0 +1,21 @@ + + + + + + + + + + + + + + + diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/__init__.py b/airbyte-integrations/connectors/source-camunda-history/integration_tests/__init__.py new file mode 100644 index 000000000000..c941b3045795 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/__init__.py @@ -0,0 +1,3 @@ +# +# Copyright (c) 2023 Airbyte, Inc., all rights reserved. +# diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/abnormal_state.json b/airbyte-integrations/connectors/source-camunda-history/integration_tests/abnormal_state.json new file mode 100644 index 000000000000..5b96b65af290 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/abnormal_state.json @@ -0,0 +1,17 @@ +{ + "activity-instance-startedAfter": { + "startTime": "2924-01-19T13:39:01.170+0000" + }, + "activity-instance-finishedAfter": { + "endTime": "2924-01-19T13:39:01.170+0000" + }, + "decision-instance": { + "evaluationTime": "2924-01-19T13:39:01.170+0000" + }, + "process-instance-startedAfter": { + "startTime": "2924-01-19T13:39:01.170+0000" + }, + "process-instance-finishedAfter": { + "endTime": "2924-01-19T13:39:01.170+0000" + } +} diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/acceptance.py b/airbyte-integrations/connectors/source-camunda-history/integration_tests/acceptance.py new file mode 100644 index 000000000000..9e6409236281 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/acceptance.py @@ -0,0 +1,16 @@ +# +# Copyright (c) 2023 Airbyte, Inc., all rights reserved. +# + + +import pytest + +pytest_plugins = ("connector_acceptance_test.plugin",) + + +@pytest.fixture(scope="session", autouse=True) +def connector_setup(): + """This fixture is a placeholder for external resources that acceptance test might require.""" + # TODO: setup test dependencies if needed. otherwise remove the TODO comments + yield + # TODO: clean up test dependencies diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/config.json b/airbyte-integrations/connectors/source-camunda-history/integration_tests/config.json new file mode 100644 index 000000000000..04e7e33ffdbb --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/config.json @@ -0,0 +1,6 @@ +{ + "baseurl": "http://localhost:8090/engine-rest", + "batchsize": "100", + "start_date": "2024-01-01T00:00:00Z", + "username": "anonymous" +} \ No newline at end of file diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/configured_catalog.json b/airbyte-integrations/connectors/source-camunda-history/integration_tests/configured_catalog.json new file mode 100644 index 000000000000..2dfafc38d0f5 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/configured_catalog.json @@ -0,0 +1,1051 @@ +{ + "streams": [ + { + "stream": { + "name": "activity-instance-startedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/schema#", + "properties": { + "id": { + "type": "string" + }, + "taskId": { + "type": [ + "null", + "string" + ] + }, + "endTime": { + "type": "string" + }, + "assignee": [ + "null", + "string" + ], + "canceled": { + "type": "boolean" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "startTime": { + "type": "string" + }, + "activityId": { + "type": "string" + }, + "executionId": { + "type": "string" + }, + "removalTime": { + "type": "string" + }, + "activityName": { + "type": "string" + }, + "activityType": { + "type": "string" + }, + "completeScope": { + "type": "boolean" + }, + "durationInMillis": { + "type": "number" + }, + "processInstanceId": { + "type": "string" + }, + "processDefinitionId": { + "type": "string" + }, + "calledCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "processDefinitionKey": { + "type": "string" + }, + "rootProcessInstanceId": { + "type": "string" + }, + "calledProcessInstanceId": { + "type": [ + "null", + "string" + ] + }, + "parentActivityInstanceId": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["startTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "startTime" ] + }, + { + "stream": { + "name": "activity-instance-finishedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/schema#", + "properties": { + "id": { + "type": "string" + }, + "taskId": { + "type": [ + "null", + "string" + ] + }, + "endTime": { + "type": "string" + }, + "assignee": [ + "null", + "string" + ], + "canceled": { + "type": "boolean" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "startTime": { + "type": "string" + }, + "activityId": { + "type": "string" + }, + "executionId": { + "type": "string" + }, + "removalTime": { + "type": "string" + }, + "activityName": { + "type": "string" + }, + "activityType": { + "type": "string" + }, + "completeScope": { + "type": "boolean" + }, + "durationInMillis": { + "type": "number" + }, + "processInstanceId": { + "type": "string" + }, + "processDefinitionId": { + "type": "string" + }, + "calledCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "processDefinitionKey": { + "type": "string" + }, + "rootProcessInstanceId": { + "type": "string" + }, + "calledProcessInstanceId": { + "type": [ + "null", + "string" + ] + }, + "parentActivityInstanceId": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["endTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "endTime" ] + }, + { + "stream": { + "name": "process-instance-startedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/schema#", + "properties": { + "id": { + "type": "string" + }, + "state": { + "type": "string" + }, + "endTime": { + "type": "string" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "startTime": { + "type": "string" + }, + "businessKey": { + "type": "string" + }, + "removalTime": { + "type": "string" + }, + "startUserId": { + "type": [ + "null", + "string" + ] + }, + "deleteReason": { + "type": [ + "null", + "string" + ] + }, + "caseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "startActivityId": { + "type": "string" + }, + "durationInMillis": { + "type": "number" + }, + "processDefinitionId": { + "type": "string" + }, + "superCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "processDefinitionKey": { + "type": "string" + }, + "processDefinitionName": { + "type": "string" + }, + "rootProcessInstanceId": { + "type": "string" + }, + "superProcessInstanceId": { + "type": [ + "null", + "string" + ] + }, + "processDefinitionVersion": { + "type": "number" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["startTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "startTime" ] + }, + { + "stream": { + "name": "process-instance-finishedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/schema#", + "properties": { + "id": { + "type": "string" + }, + "state": { + "type": "string" + }, + "endTime": { + "type": "string" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "startTime": { + "type": "string" + }, + "businessKey": { + "type": "string" + }, + "removalTime": { + "type": "string" + }, + "startUserId": { + "type": [ + "null", + "string" + ] + }, + "deleteReason": { + "type": [ + "null", + "string" + ] + }, + "caseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "startActivityId": { + "type": "string" + }, + "durationInMillis": { + "type": "number" + }, + "processDefinitionId": { + "type": "string" + }, + "superCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "processDefinitionKey": { + "type": "string" + }, + "processDefinitionName": { + "type": "string" + }, + "rootProcessInstanceId": { + "type": "string" + }, + "superProcessInstanceId": { + "type": [ + "null", + "string" + ] + }, + "processDefinitionVersion": { + "type": "number" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["endTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "endTime" ] + }, + { + "stream": { + "name": "batch", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "id": { + "type": "string" + }, + "type": { + "type": "string" + }, + "endTime": { + "type": "string" + }, + "tenantId": { + "type": "string" + }, + "startTime": { + "type": "string" + }, + "totalJobs": { + "type": "number" + }, + "removalTime": { + "type": "string" + }, + "createUserId": { + "type": "string" + }, + "batchJobsPerSeed": { + "type": "number" + }, + "seedJobDefinitionId": { + "type": "string" + }, + "batchJobDefinitionId": { + "type": "string" + }, + "invocationsPerBatchJob": { + "type": "number" + }, + "monitorJobDefinitionId": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh"], + "source_defined_cursor": false + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "append" + }, + { + "stream": { + "name": "case-instance-createdAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "id": { + "type": "string" + }, + "active": { + "type": "boolean" + }, + "closed": { + "type": "boolean" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "closeTime": { + "type": "string" + }, + "completed": { + "type": "boolean" + }, + "createTime": { + "type": "string" + }, + "terminated": { + "type": "boolean" + }, + "businessKey": { + "type": "string" + }, + "createUserId": { + "type": "string" + }, + "caseDefinitionId": { + "type": "string" + }, + "durationInMillis": { + "type": "number" + }, + "caseDefinitionKey": { + "type": "string" + }, + "caseDefinitionName": { + "type": "string" + }, + "superCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "superProcessInstanceId": { + "type": [ + "null", + "string" + ] + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["createTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "createTime" ] + }, + { + "stream": { + "name": "case-instance-closedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "id": { + "type": "string" + }, + "active": { + "type": "boolean" + }, + "closed": { + "type": "boolean" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "closeTime": { + "type": "string" + }, + "completed": { + "type": "boolean" + }, + "createTime": { + "type": "string" + }, + "terminated": { + "type": "boolean" + }, + "businessKey": { + "type": "string" + }, + "createUserId": { + "type": "string" + }, + "caseDefinitionId": { + "type": "string" + }, + "durationInMillis": { + "type": "number" + }, + "caseDefinitionKey": { + "type": "string" + }, + "caseDefinitionName": { + "type": "string" + }, + "superCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "superProcessInstanceId": { + "type": [ + "null", + "string" + ] + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["closeTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "closeTime" ] + }, + { + "stream": { + "name": "case-activity-instance-createdAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "id": { + "type": "string" + }, + "active": { + "type": "boolean" + }, + "taskId": { + "type": [ + "null", + "string" + ] + }, + "enabled": { + "type": "boolean" + }, + "disabled": { + "type": "boolean" + }, + "required": { + "type": "boolean" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "available": { + "type": "boolean" + }, + "endTime": { + "type": "string" + }, + "completed": { + "type": "boolean" + }, + "createTime": { + "type": "string" + }, + "repeatable": { + "type": "boolean" + }, + "repetition": { + "type": "boolean" + }, + "terminated": { + "type": "boolean" + }, + "caseActivityId": { + "type": "string" + }, + "caseInstanceId": { + "type": "string" + }, + "caseExecutionId": { + "type": "string" + }, + "caseActivityName": { + "type": "string" + }, + "caseActivityType": { + "type": "string" + }, + "caseDefinitionId": { + "type": "string" + }, + "durationInMillis": { + "type": "number" + }, + "calledCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "calledProcessInstanceId": { + "type": [ + "null", + "string" + ] + }, + "parentCaseActivityInstanceId": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["createTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "createTime" ] + }, + { + "stream": { + "name": "case-activity-instance-endedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "id": { + "type": "string" + }, + "active": { + "type": "boolean" + }, + "taskId": { + "type": [ + "null", + "string" + ] + }, + "enabled": { + "type": "boolean" + }, + "disabled": { + "type": "boolean" + }, + "required": { + "type": "boolean" + }, + "tenantId": { + "type": [ + "null", + "string" + ] + }, + "available": { + "type": "boolean" + }, + "completed": { + "type": "boolean" + }, + "createTime": { + "type": "string" + }, + "endTime": { + "type": "string" + }, + "repeatable": { + "type": "boolean" + }, + "repetition": { + "type": "boolean" + }, + "terminated": { + "type": "boolean" + }, + "caseActivityId": { + "type": "string" + }, + "caseInstanceId": { + "type": "string" + }, + "caseExecutionId": { + "type": "string" + }, + "caseActivityName": { + "type": "string" + }, + "caseActivityType": { + "type": "string" + }, + "caseDefinitionId": { + "type": "string" + }, + "durationInMillis": { + "type": "number" + }, + "calledCaseInstanceId": { + "type": [ + "null", + "string" + ] + }, + "calledProcessInstanceId": { + "type": [ + "null", + "string" + ] + }, + "parentCaseActivityInstanceId": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["endTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "endTime" ] + }, + { + "stream": { + "name": "decision-instance", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/schema#", + "properties": { + "id": { + "type": "string" + }, + "userId": { + "type": "string" + }, + "tenantId": { + "type": "string" + }, + "activityId": { + "type": "string" + }, + "removalTime": { + "type": "string" + }, + "caseInstanceId": { + "type": "string" + }, + "evaluationTime": { + "type": "string" + }, + "caseDefinitionId": { + "type": "string" + }, + "caseDefinitionKey": { + "type": "string" + }, + "processInstanceId": { + "type": "string" + }, + "activityInstanceId": { + "type": "string" + }, + "processDefinitionId": { + "type": "string" + }, + "decisionDefinitionId": { + "type": "string" + }, + "processDefinitionKey": { + "type": "string" + }, + "decisionDefinitionKey": { + "type": "string" + }, + "rootProcessInstanceId": { + "type": [ + "null", + "string" + ] + }, + "decisionDefinitionName": { + "type": "string" + }, + "rootDecisionInstanceId": { + "type": [ + "null", + "string" + ] + }, + "decisionRequirementsDefinitionId": { + "type": "string" + }, + "decisionRequirementsDefinitionKey": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["evaluationTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "evaluationTime" ] + }, + { + "stream": { + "name": "task-startedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/schema#", + "properties": { + "id": { + "type": "string" + }, + "due": { + "type": "string" + }, + "name": { + "type": "string" + }, + "owner": { + "type": "string" + }, + "endTime": { + "type": "string" + }, + "assignee": { + "type": "string" + }, + "duration": { + "type": "number" + }, + "followUp": { + "type": "string" + }, + "priority": { + "type": "number" + }, + "tenantId": { + "type": "string" + }, + "startTime": { + "type": "string" + }, + "description": { + "type": "string" + }, + "executionId": { + "type": "string" + }, + "removalTime": { + "type": "string" + }, + "deleteReason": { + "type": "string" + }, + "parentTaskId": { + "type": "string" + }, + "caseInstanceId": { + "type": "string" + }, + "caseExecutionId": { + "type": "string" + }, + "caseDefinitionId": { + "type": "string" + }, + "caseDefinitionKey": { + "type": "string" + }, + "processInstanceId": { + "type": "string" + }, + "taskDefinitionKey": { + "type": "string" + }, + "activityInstanceId": { + "type": "string" + }, + "processDefinitionId": { + "type": "string" + }, + "processDefinitionKey": { + "type": "string" + }, + "rootProcessInstanceId": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["startTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "startTime" ] + }, + { + "stream": { + "name": "task-finishedAfter", + "json_schema": { + "type": "object", + "$schema": "http://json-schema.org/schema#", + "properties": { + "id": { + "type": "string" + }, + "due": { + "type": "string" + }, + "name": { + "type": "string" + }, + "owner": { + "type": "string" + }, + "endTime": { + "type": "string" + }, + "assignee": { + "type": "string" + }, + "duration": { + "type": "number" + }, + "followUp": { + "type": "string" + }, + "priority": { + "type": "number" + }, + "tenantId": { + "type": "string" + }, + "startTime": { + "type": "string" + }, + "description": { + "type": "string" + }, + "executionId": { + "type": "string" + }, + "removalTime": { + "type": "string" + }, + "deleteReason": { + "type": "string" + }, + "parentTaskId": { + "type": "string" + }, + "caseInstanceId": { + "type": "string" + }, + "caseExecutionId": { + "type": "string" + }, + "caseDefinitionId": { + "type": "string" + }, + "caseDefinitionKey": { + "type": "string" + }, + "processInstanceId": { + "type": "string" + }, + "taskDefinitionKey": { + "type": "string" + }, + "activityInstanceId": { + "type": "string" + }, + "processDefinitionId": { + "type": "string" + }, + "processDefinitionKey": { + "type": "string" + }, + "rootProcessInstanceId": { + "type": "string" + } + } + }, + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "default_cursor_field": ["endTime"] + }, + "sync_mode": "incremental", + "destination_sync_mode": "append", + "cursor_field": [ "endTime" ] + } + ] +} diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/expected_records.jsonl b/airbyte-integrations/connectors/source-camunda-history/integration_tests/expected_records.jsonl new file mode 100644 index 000000000000..89299348a17e --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/expected_records.jsonl @@ -0,0 +1 @@ +{"stream": "task-finishedAfter", "data": {"id": "acb1d348-b6cd-11ee-8ba9-9ac115950248", "processDefinitionKey": "deathRegistrationProcess", "processDefinitionId": "deathRegistrationProcess:1:0baa8a0e-aeea-11ee-8ba9-9ac115950248", "processInstanceId": "acb1d345-b6cd-11ee-8ba9-9ac115950248", "executionId": "acb1d345-b6cd-11ee-8ba9-9ac115950248", "caseDefinitionKey": null, "caseDefinitionId": null, "caseInstanceId": null, "caseExecutionId": null, "activityInstanceId": "awaitingAppointment:acb1d347-b6cd-11ee-8ba9-9ac115950248", "name": "Awaiting appointment", "description": null, "deleteReason": "completed", "owner": null, "assignee": null, "startTime": "2024-01-19T13:21:37.696+0000", "endTime": "2024-01-19T13:27:36.817+0000", "duration": 359121, "taskDefinitionKey": "awaitingAppointment", "priority": 50, "due": null, "parentTaskId": null, "followUp": null, "tenantId": null, "removalTime": "2024-02-18T13:28:50.728+0000", "rootProcessInstanceId": "acb1d345-b6cd-11ee-8ba9-9ac115950248"}, "emitted_at": 1706282859372} diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/invalid_config.json b/airbyte-integrations/connectors/source-camunda-history/integration_tests/invalid_config.json new file mode 100644 index 000000000000..ba1dab9ebc15 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/invalid_config.json @@ -0,0 +1,6 @@ +{ + "baseurl": "http://localhost:8090/engine-rest", + "batchsize": "100", + "start_date": "2024-01-13T22:36:12.357+0000" + } + \ No newline at end of file diff --git a/airbyte-integrations/connectors/source-camunda-history/integration_tests/sample_state.json b/airbyte-integrations/connectors/source-camunda-history/integration_tests/sample_state.json new file mode 100644 index 000000000000..e2aa7e63be18 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/integration_tests/sample_state.json @@ -0,0 +1,18 @@ +{ + "activity-instance-startedAfter": { + "startTime": "2024-01-19T13:39:01.170+0000" + }, + "activity-instance-finishedAfter": { + "endTime": "2024-01-19T13:39:01.170+0000" + }, + "decision-instance": { + "evaluationTime": "2024-01-19T13:39:01.170+0000" + }, + "process-instance-startedAfter": { + "startTime": "2024-01-19T13:39:01.170+0000" + }, + "process-instance-finishedAfter": { + "endTime": "2024-01-19T13:39:01.170+0000" + } + +} diff --git a/airbyte-integrations/connectors/source-camunda-history/main.py b/airbyte-integrations/connectors/source-camunda-history/main.py new file mode 100644 index 000000000000..5b454324a63e --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/main.py @@ -0,0 +1,13 @@ +# +# Copyright (c) 2023 Airbyte, Inc., all rights reserved. +# + + +import sys + +from airbyte_cdk.entrypoint import launch +from source_camunda_history import SourceCamundaHistory + +if __name__ == "__main__": + source = SourceCamundaHistory() + launch(source, sys.argv[1:]) diff --git a/airbyte-integrations/connectors/source-camunda-history/metadata.yaml b/airbyte-integrations/connectors/source-camunda-history/metadata.yaml new file mode 100644 index 000000000000..734e5eab0364 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/metadata.yaml @@ -0,0 +1,25 @@ +data: + allowedHosts: + hosts: + - "*" + registries: + oss: + enabled: true + cloud: + enabled: true + connectorSubtype: api + connectorType: source + definitionId: 10d595c1-e10f-48a3-9fd3-cc197c74954f + dockerImageTag: 0.1.0 + dockerRepository: airbyte/source-camunda-history + githubIssueLabel: source-camunda-history + icon: camunda-history.svg + license: MIT + name: Camunda History + releaseDate: 2024-01-25 + releaseStage: alpha + supportLevel: community + documentationUrl: https://docs.airbyte.com/integrations/sources/camunda-history + tags: + - language:low-code +metadataSpecVersion: "1.0" diff --git a/airbyte-integrations/connectors/source-camunda-history/requirements.txt b/airbyte-integrations/connectors/source-camunda-history/requirements.txt new file mode 100644 index 000000000000..cc57334ef619 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/requirements.txt @@ -0,0 +1,2 @@ +-e ../../bases/connector-acceptance-test +-e . diff --git a/airbyte-integrations/connectors/source-camunda-history/setup.py b/airbyte-integrations/connectors/source-camunda-history/setup.py new file mode 100644 index 000000000000..4c5e32dd52b7 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/setup.py @@ -0,0 +1,28 @@ +# +# Copyright (c) 2023 Airbyte, Inc., all rights reserved. +# + + +from setuptools import find_packages, setup + +MAIN_REQUIREMENTS = [ + "airbyte-cdk~=0.1", +] + +TEST_REQUIREMENTS = [ + "pytest~=6.2", + "pytest-mock~=3.6.1", +] + +setup( + name="source_camunda_history", + description="Source implementation for Camunda History.", + author="MetaOps", + author_email="info@metaops.solutions", + packages=find_packages(), + install_requires=MAIN_REQUIREMENTS, + package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]}, + extras_require={ + "tests": TEST_REQUIREMENTS, + }, +) diff --git a/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/__init__.py b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/__init__.py new file mode 100644 index 000000000000..e95fde16c768 --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/__init__.py @@ -0,0 +1,8 @@ +# +# Copyright (c) 2023 Airbyte, Inc., all rights reserved. +# + + +from .source import SourceCamundaHistory + +__all__ = ["SourceCamundaHistory"] diff --git a/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/manifest.yaml b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/manifest.yaml new file mode 100644 index 000000000000..53be58b5319a --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/manifest.yaml @@ -0,0 +1,1289 @@ +spec: + type: Spec + connection_specification: + type: object + $schema: http://json-schema.org/draft-07/schema# + required: + - username + - start_date + - batchsize + - baseurl + properties: + baseurl: + type: string + order: 4 + title: baseurl + description: http://localhost:8090 + password: + type: string + order: 1 + title: Password + always_show: true + airbyte_secret: true + username: + type: string + order: 0 + title: Username + batchsize: + type: string + order: 3 + title: batchsize + default: '100' + description: Number of items to fetch at 1 rest call + start_date: + type: string + order: 2 + title: Start date + format: date-time + pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ + description: >- + Initial Start Date in format "2023-09-03T13:33:42.165+0100" from which + we are ingesting events + additionalProperties: true +type: DeclarativeSource +check: + type: CheckStream + stream_names: + - activity-instance-startedAfter +streams: + - name: activity-instance-startedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: activity-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: startTime + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + unfinished: 'true' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/schema# + properties: + id: + type: string + taskId: + type: + - 'null' + - string + endTime: + type: string + assignee: + type: + - 'null' + - string + canceled: + type: boolean + tenantId: + type: + - 'null' + - string + startTime: + type: string + activityId: + type: string + executionId: + type: string + removalTime: + type: string + activityName: + type: string + activityType: + type: string + completeScope: + type: boolean + durationInMillis: + type: number + processInstanceId: + type: string + processDefinitionId: + type: string + calledCaseInstanceId: + type: + - 'null' + - string + processDefinitionKey: + type: string + rootProcessInstanceId: + type: string + calledProcessInstanceId: + type: + - 'null' + - string + parentActivityInstanceId: + type: string + incremental_sync: + type: DatetimeBasedCursor + cursor_field: startTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: startedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: activity-instance-finishedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: activity-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: endTime + finished: 'true' + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/schema# + properties: + id: + type: string + taskId: + type: + - 'null' + - string + endTime: + type: string + assignee: + type: + - 'null' + - string + canceled: + type: boolean + tenantId: + type: + - 'null' + - string + startTime: + type: string + activityId: + type: string + executionId: + type: string + removalTime: + type: string + activityName: + type: string + activityType: + type: string + completeScope: + type: boolean + durationInMillis: + type: number + processInstanceId: + type: string + processDefinitionId: + type: string + calledCaseInstanceId: + type: + - 'null' + - string + processDefinitionKey: + type: string + rootProcessInstanceId: + type: string + calledProcessInstanceId: + type: + - 'null' + - string + parentActivityInstanceId: + type: string + incremental_sync: + type: DatetimeBasedCursor + cursor_field: endTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: finishedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: process-instance-startedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: process-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: startTime + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + unfinished: 'true' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/schema# + properties: + id: + type: string + state: + type: string + endTime: + type: string + tenantId: + type: + - 'null' + - string + startTime: + type: string + businessKey: + type: string + removalTime: + type: string + startUserId: + type: + - 'null' + - string + deleteReason: + type: + - 'null' + - string + caseInstanceId: + type: + - 'null' + - string + startActivityId: + type: string + durationInMillis: + type: number + processDefinitionId: + type: string + superCaseInstanceId: + type: + - 'null' + - string + processDefinitionKey: + type: string + processDefinitionName: + type: string + rootProcessInstanceId: + type: string + superProcessInstanceId: + type: + - 'null' + - string + processDefinitionVersion: + type: number + incremental_sync: + type: DatetimeBasedCursor + cursor_field: startTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: startedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: process-instance-finishedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: process-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: endTime + finished: 'true' + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/schema# + properties: + id: + type: string + state: + type: string + endTime: + type: string + tenantId: + type: + - 'null' + - string + startTime: + type: string + businessKey: + type: string + removalTime: + type: string + startUserId: + type: + - 'null' + - string + deleteReason: + type: + - 'null' + - string + caseInstanceId: + type: + - 'null' + - string + startActivityId: + type: string + durationInMillis: + type: number + processDefinitionId: + type: string + superCaseInstanceId: + type: + - 'null' + - string + processDefinitionKey: + type: string + processDefinitionName: + type: string + rootProcessInstanceId: + type: string + superProcessInstanceId: + type: + - 'null' + - string + processDefinitionVersion: + type: number + incremental_sync: + type: DatetimeBasedCursor + cursor_field: endTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: finishedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: batch + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: batch + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: startTime + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/draft-07/schema# + properties: + id: + type: string + type: + type: string + endTime: + type: string + tenantId: + type: string + startTime: + type: string + totalJobs: + type: number + removalTime: + type: string + createUserId: + type: string + batchJobsPerSeed: + type: number + seedJobDefinitionId: + type: string + batchJobDefinitionId: + type: string + invocationsPerBatchJob: + type: number + monitorJobDefinitionId: + type: string + additionalProperties: true + - name: case-instance-createdAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: case-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: createTime + notClosed: 'true' + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/draft-07/schema# + properties: + id: + type: string + active: + type: boolean + closed: + type: boolean + tenantId: + type: + - 'null' + - string + closeTime: + type: string + completed: + type: boolean + createTime: + type: string + terminated: + type: boolean + businessKey: + type: string + createUserId: + type: string + caseDefinitionId: + type: string + durationInMillis: + type: number + caseDefinitionKey: + type: string + caseDefinitionName: + type: string + superCaseInstanceId: + type: + - 'null' + - string + superProcessInstanceId: + type: + - 'null' + - string + additionalProperties: true + incremental_sync: + type: DatetimeBasedCursor + cursor_field: createTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: createdAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: case-instance-closedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: case-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: closeTime + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/draft-07/schema# + properties: + id: + type: string + active: + type: boolean + closed: + type: boolean + tenantId: + type: + - 'null' + - string + closeTime: + type: string + completed: + type: boolean + createTime: + type: string + terminated: + type: boolean + businessKey: + type: string + createUserId: + type: string + caseDefinitionId: + type: string + durationInMillis: + type: number + caseDefinitionKey: + type: string + caseDefinitionName: + type: string + superCaseInstanceId: + type: + - 'null' + - string + superProcessInstanceId: + type: + - 'null' + - string + additionalProperties: true + incremental_sync: + type: DatetimeBasedCursor + cursor_field: closeTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: closedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: case-activity-instance-createdAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: case-activity-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: createTime + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + unfinished: 'true' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/draft-07/schema# + properties: + id: + type: string + active: + type: boolean + taskId: + type: + - 'null' + - string + enabled: + type: boolean + disabled: + type: boolean + required: + type: boolean + tenantId: + type: + - 'null' + - string + available: + type: boolean + completed: + type: boolean + createTime: + type: string + endTime: + type: string + repeatable: + type: boolean + repetition: + type: boolean + terminated: + type: boolean + caseActivityId: + type: string + caseInstanceId: + type: string + caseExecutionId: + type: string + caseActivityName: + type: string + caseActivityType: + type: string + caseDefinitionId: + type: string + durationInMillis: + type: number + calledCaseInstanceId: + type: + - 'null' + - string + calledProcessInstanceId: + type: + - 'null' + - string + parentCaseActivityInstanceId: + type: string + additionalProperties: true + incremental_sync: + type: DatetimeBasedCursor + cursor_field: createTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: createdAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: case-activity-instance-endedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: case-activity-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: endTime + finished: 'true' + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/draft-07/schema# + properties: + id: + type: string + active: + type: boolean + taskId: + type: + - 'null' + - string + enabled: + type: boolean + disabled: + type: boolean + required: + type: boolean + tenantId: + type: + - 'null' + - string + available: + type: boolean + endTime: + type: string + completed: + type: boolean + createTime: + type: string + repeatable: + type: boolean + repetition: + type: boolean + terminated: + type: boolean + caseActivityId: + type: string + caseInstanceId: + type: string + caseExecutionId: + type: string + caseActivityName: + type: string + caseActivityType: + type: string + caseDefinitionId: + type: string + durationInMillis: + type: number + calledCaseInstanceId: + type: + - 'null' + - string + calledProcessInstanceId: + type: + - 'null' + - string + parentCaseActivityInstanceId: + type: string + additionalProperties: true + incremental_sync: + type: DatetimeBasedCursor + cursor_field: endTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: endedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: decision-instance + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: decision-instance + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: evaluationTime + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/schema# + properties: + id: + type: string + userId: + type: string + tenantId: + type: string + activityId: + type: string + removalTime: + type: string + caseInstanceId: + type: string + evaluationTime: + type: string + caseDefinitionId: + type: string + caseDefinitionKey: + type: string + processInstanceId: + type: string + activityInstanceId: + type: string + processDefinitionId: + type: string + decisionDefinitionId: + type: string + processDefinitionKey: + type: string + decisionDefinitionKey: + type: string + rootProcessInstanceId: + type: + - 'null' + - string + decisionDefinitionName: + type: string + rootDecisionInstanceId: + type: + - 'null' + - string + decisionRequirementsDefinitionId: + type: string + decisionRequirementsDefinitionKey: + type: string + incremental_sync: + type: DatetimeBasedCursor + cursor_field: evaluationTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: evaluatedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: task-startedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: task + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: startTime + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + unfinished: 'true' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/schema# + properties: + id: + type: string + due: + type: string + name: + type: string + owner: + type: string + endTime: + type: string + assignee: + type: string + duration: + type: number + followUp: + type: string + priority: + type: number + tenantId: + type: string + startTime: + type: string + description: + type: string + executionId: + type: string + removalTime: + type: string + deleteReason: + type: string + parentTaskId: + type: string + caseInstanceId: + type: string + caseExecutionId: + type: string + caseDefinitionId: + type: string + caseDefinitionKey: + type: string + processInstanceId: + type: string + taskDefinitionKey: + type: string + activityInstanceId: + type: string + processDefinitionId: + type: string + processDefinitionKey: + type: string + rootProcessInstanceId: + type: string + incremental_sync: + type: DatetimeBasedCursor + cursor_field: startTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: startedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' + - name: task-finishedAfter + type: DeclarativeStream + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: firstResult + inject_into: request_parameter + pagination_strategy: + type: OffsetIncrement + requester: + path: task + type: HttpRequester + url_base: '{{ config[''baseurl''] }}/history/' + http_method: GET + authenticator: + type: BasicHttpAuthenticator + password: '{{ config[''password''] }}' + username: '{{ config[''username''] }}' + request_headers: {} + request_body_json: {} + request_parameters: + sortBy: endTime + finished: 'true' + sortOrder: asc + maxResults: '{{ config[''batchsize''] }}' + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: [] + partition_router: [] + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + type: object + $schema: http://json-schema.org/schema# + properties: + id: + type: string + due: + type: string + name: + type: string + owner: + type: string + endTime: + type: string + assignee: + type: string + duration: + type: number + followUp: + type: string + priority: + type: number + tenantId: + type: string + startTime: + type: string + description: + type: string + executionId: + type: string + removalTime: + type: string + deleteReason: + type: string + parentTaskId: + type: string + caseInstanceId: + type: string + caseExecutionId: + type: string + caseDefinitionId: + type: string + caseDefinitionKey: + type: string + processInstanceId: + type: string + taskDefinitionKey: + type: string + activityInstanceId: + type: string + processDefinitionId: + type: string + processDefinitionKey: + type: string + rootProcessInstanceId: + type: string + incremental_sync: + type: DatetimeBasedCursor + cursor_field: endTime + start_datetime: + type: MinMaxDatetime + datetime: '{{ config[''start_date''] }}' + datetime_format: '%Y-%m-%dT%H:%M:%SZ' + datetime_format: '%Y-%m-%dT%H:%M:%S.%f%z' + start_time_option: + type: RequestOption + field_name: finishedAfter + inject_into: request_parameter + cursor_datetime_formats: + - '%Y-%m-%dT%H:%M:%S.%f%z' +version: 0.57.0 +metadata: + autoImportSchema: + batch: false + decision-instance: false + task-startedAfter: false + task-finishedAfter: false + case-instance-closedAfter: false + case-instance-createdAfter: false + process-instance-startedAfter: false + activity-instance-startedAfter: false + process-instance-finishedAfter: false + activity-instance-finishedAfter: false + case-activity-instance-endedAfter: false + case-activity-instance-createdAfter: false diff --git a/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/schemas/TODO.md b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/schemas/TODO.md new file mode 100644 index 000000000000..08bdb076d9fd --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/schemas/TODO.md @@ -0,0 +1,16 @@ +# TODO: Define your stream schemas +Your connector must describe the schema of each stream it can output using [JSONSchema](https://json-schema.org). + +You can describe the schema of your streams using one `.json` file per stream. + +## Static schemas +From the `camunda_history.yaml` configuration file, you read the `.json` files in the `schemas/` directory. You can refer to a schema in your configuration file using the `schema_loader` component's `file_path` field. For example: +``` +schema_loader: + type: JsonSchema + file_path: "./source_camunda_history/schemas/customers.json" +``` +Every stream specified in the configuration file should have a corresponding `.json` schema file. + +Delete this file once you're done. Or don't. Up to you :) + diff --git a/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/source.py b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/source.py new file mode 100644 index 000000000000..5e056f032e6a --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/source.py @@ -0,0 +1,18 @@ +# +# Copyright (c) 2023 Airbyte, Inc., all rights reserved. +# + +from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource + +""" +This file provides the necessary constructs to interpret a provided declarative YAML configuration file into +source connector. + +WARNING: Do not modify this file. +""" + + +# Declarative Source +class SourceCamundaHistory(YamlDeclarativeSource): + def __init__(self): + super().__init__(**{"path_to_yaml": "manifest.yaml"}) diff --git a/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/spec.yaml b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/spec.yaml new file mode 100644 index 000000000000..e182e2cd196b --- /dev/null +++ b/airbyte-integrations/connectors/source-camunda-history/source_camunda_history/spec.yaml @@ -0,0 +1,40 @@ +connectionSpecification: + type: object + $schema: http://json-schema.org/draft-07/schema# + required: + - username + - start_date + - batchsize + - baseurl + properties: + baseurl: + type: string + order: 4 + title: baseurl + description: http://localhost:8090 + password: + type: string + order: 1 + title: Password + always_show: true + airbyte_secret: true + username: + type: string + order: 0 + title: Username + batchsize: + type: string + order: 3 + title: batchsize + default: '100' + description: Number of items to fetch at 1 rest call + start_date: + type: string + order: 2 + title: Start date + format: date-time + pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ + description: >- + Initial Start Date in format "2023-09-03T13:33:42.165+0100" from which + we are ingesting events + additionalProperties: true diff --git a/docs/integrations/sources/camunda-history.md b/docs/integrations/sources/camunda-history.md new file mode 100644 index 000000000000..811e08e1e494 --- /dev/null +++ b/docs/integrations/sources/camunda-history.md @@ -0,0 +1,90 @@ +# Camunda History API + + + +This page contains the setup guide and reference information for the [Camunda History API](https://camunda.com/platform-7/) source connector. + + + +## Overview + +The Camunda History API integration allows to pull historical events from Camunda managed workflow. + +## Prerequisites + +- Camunda History should be enabled + +## Setup Guide + +### Step 1: Set up Camunda History API + +The History Event Stream provides audit information about executed process instances. +[Camunda History API](https://docs.camunda.org/manual/7.20/user-guide/process-engine/history) +History API enabled by default and doesn't require username/password. + +### Step 2: Set up the Camunda History connector in Airbyte + +1. Enter BaseUrl for your Camunda engine-rest API endpoint, i.e. http://x.x.x.x:8090/engine-rest +2. If your Camunda installation has Authentication for API enabled provide Username and Password.(If no authentication required Username and Password can be any values) +3. Enter Start date in a format YYYY-MM-DDTHH:mm:ssZ. This Timestamp will point to the beginning from where you start ingesting Camunda History. +4. Provide batchsize with a number of events pulled in a single rest call. 100 is a good default. Don't set it too high, it might overload the source system. + + + +## Supported sync modes + +| Feature | Supported? | +| :------------------------ | :--------- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | Yes | + +## Supported streams + +It contains several streams corresponding to Camunda History APIs: + - batch + - decision-instance + - task-startedAfter + - task-finishedAfter + - case-instance-closedAfter + - case-instance-createdAfter + - process-instance-startedAfter + - activity-instance-startedAfter + - process-instance-finishedAfter + - activity-instance-finishedAfter + - case-activity-instance-endedAfter + - case-activity-instance-createdAfter + +Each record in the stream contains many fields: + +The full ist of fileds and their type documented in [Camunda REST](https://docs.camunda.org/manual/7.13/reference/rest/history/) + +## joining streams with DBT + +Camunda APIs which export events with a start and end time being retrieved by 2 Streams: +-startedAfter and -finishedAfter or similar + +-startedAfter - events which have started but haven't yet finished. Such even might not have end date yet. +-finishedAfter - events which have finished and have both + +We pull both types of events as independent streams so we don't miss any event which havn't finished yet or which has finished after we already recived it with a start only. At the same time it helps avoiding full sync every time so we use cursor. + +Such streams should be merged and de-duped with preference given to completed events. +Example of the [DBT models](https://github.com/metaops-solutions/airbyte-camunda) + + +## Limitations & Troubleshooting + +Depending of the Camunda use some of the streams might be empty. It is best to keep the disabled. + +### Connector limitations + +Not all Camunda History APIs are implemented by the connector. Adding them could be quite simple task. + + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------ | +| 0.1.0 | 2024-01-26 | [2942](https://github.com/airbytehq/airbyte/pull/2942) | Implement Camunda API using the CDK | + + \ No newline at end of file