Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better support modification of existing adapters #1960

Closed
dominikriemer opened this issue Sep 24, 2023 · 0 comments · Fixed by #1979
Closed

Better support modification of existing adapters #1960

dominikriemer opened this issue Sep 24, 2023 · 0 comments · Fixed by #1979
Assignees
@dominikriemer
Labels
backend Everything that is related to the StreamPipes backend connect Related to the `connect` module (adapters) enhancement New feature or request

Comments

@dominikriemer
Copy link
Member

Body

Currently, it is quite hard to modify existing adapters:

  • Adapters can only be edited if no pipeline that uses the data stream is available
  • There are several bugs when opening the adapter modification view (e.g., timestamps are duplicated)
  • For users, it is quite time-consuming (and frustrating) to delete and re-create all existing pipelines even if the modified adapters doesn't remove existing fields.

To allow convenient modification of adapters, we need to migrate existing resources that depend on an already existing adapter:

  • Pipelines need to be re-validated and updated
  • Data Lake Measures, used in dashboards and data views, need to be re-validated and updated
  • Adapters need to be checked for obsolete transformation rules after modification

In case an adapter is modified which doesn't affect existing resources, all resources should be auto-migrated.
In case an adapter is modified and a dependent resource will break (e.g., a pipeline uses a field that is removed in the adapter), the resource should be flagged for manual investigation/migration.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.
@dominikriemer dominikriemer self-assigned this Sep 24, 2023
@dominikriemer dominikriemer added enhancement New feature or request connect Related to the `connect` module (adapters) backend Everything that is related to the StreamPipes backend labels Sep 24, 2023
dominikriemer added a commit that referenced this issue Sep 29, 2023
@dominikriemer
feat(#1960): Allow modification of adapters with pipelines
943e406
@dominikriemer dominikriemer linked a pull request Sep 29, 2023 that will close this issue
feat(#1960): Allow modification of adapters with pipelines #1979
Merged
dominikriemer added a commit that referenced this issue Oct 3, 2023
@dominikriemer
refactor(#1960): Refactor transformation rules, add schema restore
4afaf65
@dominikriemer dominikriemer mentioned this issue Oct 3, 2023
Merged
dominikriemer added a commit that referenced this issue Oct 5, 2023
@dominikriemer
refactor(#1960): Refactor transformation rules, add schema restore (#…
e680d47 
…1994)

* refactor(#1960): Refactor transformation rules, add schema restore

* Fix import order

* Extract transformation rule priorities to enum

* Add comment to skipped rules in stateful transformation visitor
@muyangye muyangye mentioned this issue Oct 22, 2023
Closed
3 tasks
@bossenti bossenti mentioned this issue Oct 23, 2023
Closed
dominikriemer added a commit that referenced this issue Oct 27, 2023
@dominikriemer @bossenti @Marcelfrueh
feat(#1960): Allow modification of adapters with pipelines (#1979)
f9f1bd5 
* feat(#1960): Allow modification of adapters with pipelines

* Add license header

* Add data model for schema diff management

* Improve schema editor and static value handling

* Fix schema preview of nested fields

* Improve schema updates

* Improve download dialog

* Remove unused stylesheets

* Redirect to adapter overview after adapter creation

* Improve modification of measurement units

* Fix update of measurement units

* Properly display labels and runtime type changes in schema editor

* Fix modification of conversion rules

* Show selected timestamp transformation rules

* Add warning hint and value validation to correction value rule

* Fix NullPointer when adding static rules

* Fix unit transformation

* Remove unused component

* Add metadata object to properties

* modified tests (#2079)

* Update typescript model

* Remove obsolete service

* Add logging message

---------

Co-authored-by: Tim Bossenmaier <[email protected]>
Co-authored-by: Marcelfrueh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dominikriemer dominikriemer

Labels
backend Everything that is related to the StreamPipes backend connect Related to the `connect` module (adapters) enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(#1960): Allow modification of adapters with pipelines apache/streampipes
1 participant
@dominikriemer