Add ExternalDeploymentSensor #1472
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pankajastro
changed the title
pankajastro
force-pushed
the
enhance_external_deployment_task_sesnor
branch
6 times, most recently
from
df7d0c8 to
cad8ba6
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1472 +/- ##
==========================================
+ Coverage 98.28% 98.33% +0.05%
==========================================
Files 91 94 +3
Lines 4485 4637 +152
==========================================
+ Hits 4408 4560 +152
Misses 77 77 ☔ View full report in Codecov by Sentry. |
pankajastro
commented
Feb 7, 2024
pankajastro
force-pushed
the
enhance_external_deployment_task_sesnor
branch
from
5e2e341 to
b460f77
Compare
pankajastro
changed the title
|
PR to fix CI: #1473 |
pankajastro
requested review from
phanikumv,
pankajkoti,
Lee-W,
sunank200,
utkarsharma2 and
vatsrahul1001
as code owners
pankajastro
force-pushed
the
enhance_external_deployment_task_sesnor
branch
3 times, most recently
from
ea363e7 to
5ae9682
Compare
BasPH
reviewed
Feb 12, 2024
| """ | ||
| base_url, _ = self.get_conn() | ||
| path = f"/api/v1/dags/{external_dag_id}/dagRuns" | ||
| params: dict[str, int | str | list[str]] = {"limit": 1, "state": ["running", "queued"]} |
There was a problem hiding this comment.
Unsure why limit is set to 1 here?
There was a problem hiding this comment.
just to fetch last dagrun based on execution date
| state = dag_runs[0].get("state") | ||
| if state == "success": | ||
| return True | ||
| return False |
There was a problem hiding this comment.
Do I understand correctly this fetches the latest DAG run and checks the status?
There was a problem hiding this comment.
yes, it check the status of last dagrun based on execution date
pankajastro
force-pushed
the
enhance_external_deployment_task_sesnor
branch
from
7a83e83 to
d578ccb
Compare
Lee-W
reviewed
Feb 15, 2024
phanikumv
force-pushed
the
enhance_external_deployment_task_sesnor
branch
from
9ffdcf9 to
0c41551
Compare
pankajastro
force-pushed
the
enhance_external_deployment_task_sesnor
branch
from
0c41551 to
ac900b1
Compare
phanikumv
approved these changes
Feb 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sensor expects external_dag_id as a required param and external_task_id as an optional param if external_task_id is provided then it monitors a task instance of task external_task_id otherwise monitors the overall status of external_dag_id.
Once the sensor execution starts it fetches the dag run for external_dag_id with status running or queued if found then waits for it to succeed or fail in the trigger component. if not found then return immediately and mark it a success.
TODO;Test caseDocs/exampleprobably monitor task group too: This can't be done at moment because task group is just UI feature and not exposed externallyAssumption
When this sensor starts running it assumes that the dag it is monitoring is either in a running or queued state