Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NR-347971 | OpenLineage event consumer writer #1887

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
5 changes: 5 additions & 0 deletions entity-types/agent-openlineage/definition.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
domain: AGENT
type: OPENLINEAGE
configuration:
entityExpirationTime: MANUAL
alertable: false
27 changes: 27 additions & 0 deletions entity-types/dpo-dataset_versions/definition.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
domain: DPO
type: DATASET_VERSIONS
synthesis:
rules:
- compositeIdentifier:
separator: ":"
attributes:
- dataset.namespace
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest you prepend those tags with something specific of your domain/type e.g. openlineage.dataset.namespace.

dataset, job and pipeline are too broad and may end up matching unrelated metrics.

Copy link
Contributor

@otaviocarvalho otaviocarvalho Feb 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to change it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@otaviocarvalho Thanks for pointing this.
I have updated the PR with unique tags.

- dataset.name
- updatedAt
name: dataset.name
encodeIdentifierInGUID: true
conditions:
- attribute: dataset.name
tags:
dataset.namespace:
entityTagName: olin.dataset.namespace
multiValue: false
dataset.name:
entityTagName: olin.dataset.name
multiValue: false
dataset.updatedAt:
entityTagName: olin.dataset.updatedAt
multiValue: false
configuration:
entityExpirationTime: MANUAL
alertable: true
23 changes: 23 additions & 0 deletions entity-types/dpo-datasets/definition.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
domain: DPO
type: DATASETS
synthesis:
rules:
- compositeIdentifier:
separator: ":"
attributes:
- dataset.namespace
- dataset.name
name: dataset.name
encodeIdentifierInGUID: true
conditions:
- attribute: dataset.name
tags:
dataset.namespace:
entityTagName: olin.dataset.namespace
multiValue: false
dataset.name:
entityTagName: olin.dataset.name
multiValue: false
configuration:
entityExpirationTime: MANUAL
alertable: true
30 changes: 30 additions & 0 deletions entity-types/dpo-job_versions/definition.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
domain: DPO
type: JOB_VERSIONS
synthesis:
rules:
- compositeIdentifier:
separator: ":"
attributes:
- job.namespace
- job.name
- updatedAt
name: job.name
encodeIdentifierInGUID: true
conditions:
- attribute: job.name
tags:
job.namespace:
entityTagName: olin.job.namespace
multiValue: false
job.name:
entityTagName: olin.job.name
multiValue: false
job.updatedAt:
entityTagName: olin.job.updatedAt
multiValue: false
pipeline.name:
entityTagName: olin.pipeline.name
multiValue: false
configuration:
entityExpirationTime: MANUAL
alertable: true
27 changes: 27 additions & 0 deletions entity-types/dpo-jobs/definition.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
domain: DPO
type: JOBS
synthesis:
rules:
- compositeIdentifier:
separator: ":"
attributes:
- job.namespace
- job.name
- eventTime
name: job.name
encodeIdentifierInGUID: true
conditions:
- attribute: job.name
tags:
job.namespace:
entityTagName: olin.job.namespace
multiValue: false
job.name:
entityTagName: olin.job.name
multiValue: false
pipeline.name:
entityTagName: olin.pipeline.name
multiValue: false
configuration:
entityExpirationTime: MANUAL
alertable: true
23 changes: 23 additions & 0 deletions entity-types/dpo-pipelines/definition.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
domain: DPO
type: PIPELINES
synthesis:
rules:
- compositeIdentifier:
separator: ":"
attributes:
- pipeline.namespace
- pipeline.name
name: pipeline.name
encodeIdentifierInGUID: true
conditions:
- attribute: pipeline.name
tags:
pipeline.namespace:
entityTagName: olin.pipeline.namespace
multiValue: false
pipeline.name:
entityTagName: olin.pipeline.name
multiValue: false
configuration:
entityExpirationTime: MANUAL
alertable: true