Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kill jobs on workers #147

Merged
merged 6 commits into from
Jan 27, 2025
Merged

Kill jobs on workers #147

merged 6 commits into from
Jan 27, 2025

Conversation

francesconazzaro
Copy link
Collaborator

No description provided.

Copy link
Contributor

@gbiavati gbiavati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice implementation.
However, some times I have seen duplicated request_uid running on the scheduler (not often, but from time to time)
So an handling for errors should be in place to avoid crashes of the broker.
Consider for instance the race condition of a job finishing by itself just before sending the kill command

@francesconazzaro
Copy link
Collaborator Author

I've updated the PR to address multiple processes and no processes cases

Copy link
Contributor

@gbiavati gbiavati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The methods implemented are properly killing the jobs in the workers pod. This will free resources immediately.
Next task is to log this event on events table.

@francesconazzaro francesconazzaro merged commit d9ad9a1 into main Jan 27, 2025
7 of 9 checks passed
@francesconazzaro francesconazzaro deleted the kill-jobs-on-workers branch January 27, 2025 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants