Limit clients to certain tasks

[JacobCummins]

Python version (python3 -V)

3.10

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.1

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 22.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I have 5 clients, (1,2,3,4,5). Clients 1 and 2 should be working on task A (Brain Cancer Image Segmentation) while clients 3,4,5 should be working on task B (Lung Cancer Image Segmentation). Both of these tasks are completely independent.

Currently if I connect all 5 clients to the central server, then execute a training job, all 5 clients will be part of that job.
Is there a way that when I initiate task A, only clients 1 and 2 are part of the job, and similarly only clients 3,4,5 are used on task B?

[YuanTingHsieh]

@JacobCummins thanks for the question!
So essentially you have 2 independent tasks you want to do.

In NVFlare's word, you can just prepare two "job".
Say your client name is "site-1, site-2, site-3, site-4, site-5"
We can easily control which job should be run on which client using the "meta.json" for each job.

For example, your first job to do Brain Cancer Image Segmentation can have a "meta.json":

{
    "name": "brain_cancer_seg",
    "resource_spec": {  
    },
    "deploy_map": {
        "app": [
            "server", "site-1", "site-2"
        ]
    },
    "min_clients": 2,
    "mandatory_clients": [
        "site-1", "site-2"
    ]
}

While your second job have a meta.json:

{
    "name": "brain_cancer_seg",
    "resource_spec": {  
    },
    "deploy_map": {
        "app": [
            "server", "site-3", "site-4", "site-5"
        ]
    },
    "min_clients": 2,
    "mandatory_clients": [
        "site-3", "site-4", "site-5"
    ]
}

You can refer to our documentation for more information: https://nvflare.readthedocs.io/en/stable/real_world_fl/job.html

[YuanTingHsieh]

@JacobCummins
This deploy_map has to be defined in each "JOB".
Job can be defined by anyone, it is not necessary the server side.

We have the "authorization" techniques that allows each site to control which "job" can be run/or which to reject

For example in here: https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/job-level-authorization
We showed how "site-a" use a "CustomSecurityHandler" to reject job "FL Demo Job1" to run. Any other jobs with different names will be able to deploy and run on site_a.

[JacobCummins]

@YuanTingHsieh

Thank you for this. I tried out the job-level-authorisation example and tried a few different scenarios with variants of this in my implementation.

One caveat with this is that it seems like while the security mechanism does prevent the code from running unapproved jobs by the site, it doesn't free it from the job until the job has finished.

So going back to the example above: If site-1,2,3,4,5 are all waiting for a job to execute, then when brain_cancer_seg job is started, site-1,2,3,4,5 all join that job. site-1 and site-2 correctly start running the job and site-3, site-4 and site-5 all correctly reject running the job. But site-3,4,5 don't leave the job completely. This means that while brain_cancer_seg job is running, site-3,4,5 won't be able to join lung_cancer_seg until the brain_cancer_seg job finishes and free's them.

This makes job parallelisation more difficult.

This isn't due to max_jobs limit as I have the limit set to 4.

Is this intended behaviour/do you know of a way to free up site-3,4,5 after it rejects running the job?

[chesterxgchen]

@JacobCummins
When you say " ... But site-3,4,5 don't leave the job completely. ... ", I don't understand what it means, each site doesn't "join" the job, we don't have such concept. The job's app specified on the deploy_map will be deployed to the target sites. If the job is rejected by the site, it won't run that job.

Are you saying the "lung_cancer_seg" scheduled for the site-3,4,5 are not in the "running" status, but only "submitted" status ?
This shouldn't related to the max_jobs_limit , as you only have two jobs ( brain_cancer_seg and lung_cancer_seg).

Please let us know if that's what you are saying or not.

One possibility is resource (GPU) allocated for the brain_cancer_seg is not released, even though the job is rejected, the resource is still marked used. There is not enough resources to schedule the next job. We can check on this

[YuanTingHsieh]

@JacobCummins if you follow the job-level-authorisation example,
you will see that site-a does NOT participate in the job2 training.

So follow the same logic, your site-3, site-4, site-5 will not even start the job. So there should not be a situation where "But site-3,4,5 don't leave the job completely."

[YuanTingHsieh]

@IsaacYangSLA ^ is this problem resolved? thanks