Proposal of dynamic GPU slice plugin #3820
Conversation
|
Welcome @sailorvii! |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
volcano-sh-bot
added
the
size/L
|
Hi, please squash to one commit and sign off. |
sailorvii
force-pushed
the
master
branch
2 times, most recently
from
e3ffd7e to
0000b26
Compare
There was a problem hiding this comment.
Where is the logic of this AddPod, in the mig-agent of nos? I'm wondering whether our dynamic GPU slice plugin is strongly dependent on the nos project. You can see that the annotation has the watermark of nos, and nos project is not updated frequently.
There was a problem hiding this comment.
- AddPod is in volcano/pkg/scheduler/api/node_info.go addResource.
- 3 functions can be reused from nos project: mig agent, mps agent and mps device plugin. They are not the most important part. If needed, we could rewrite them.
There was a problem hiding this comment.
/cc @Monokaix , I think we'd better rewrite them as part of volcano and evolve with us.
sailorvii
force-pushed
the
master
branch
2 times, most recently
from
9c752ad to
7e85873
Compare
|
A nice feature, but i have a few recommends:
|
Refine as JesseStutler's comments Address the comments by archlitchi. Signed-off-by: sailorvii <[email protected]> Signed-off-by: chenw66 <[email protected]>
|
Thanks for your time and review.
|
| actions: "enqueue, allocate, backfill" | ||
| tiers: | ||
| - plugins: | ||
| - name: dynamicgpuslice |
There was a problem hiding this comment.
How about use the deviceshare plugin?
|
We should clarify which dp the user should deploy and the relationship between dynamic mig slice and vgpu. The semantics of vgpu and dynamic mig slice are not completely consistent. Whether to use nvidia dp or hami needs to be discussed again. |
|
Let’s discuss it again how to evolve this feature at the weekly meeting? Currently, it seems that there are three repos: volcano does the scheduling, hami does the dp, and nos does the mig/mps agent. It is too fragmented. @sailorvii @archlitchi @Monokaix |
Thank you all for your time. It's good to discuss the details in the meeting. |
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
stale
bot
added
the
lifecycle/stale
|
|
||
| ## Introduction | ||
|
|
||
| The NVIDIA GPU build-in sharing method includes: time-slice, MPS and MIG. The context switch for time slice sharing would waste some time, so we chose the MPS and MIG. The GPU MIG profile is variable, the user could acquire the MIG device in the profile definition, but current implementation only defines the dedicated profile before the user requirement. That limits the usage of MIG. We want to develop an automatic slice plugin and create the slice when the user require it. And we also add the MPS support. |
There was a problem hiding this comment.
Please use upper case MIG uniformly.
NVIDIA official GPU sharing includes time-slice, MPS and MIG. Currently the MPS and MIG dynamic is not supported, we want to add this into volcano scheduler plugin