[Q&A] Retrieve files from server workspace while job is running

[ianh-apheris]

Python version (python3 -V)

3.10

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.0

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.4

Operating system

Ubuntu 20.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

We are using NVFlare in a closed environment, where general access to the outside internet is blocked from both the server and the client containers.

I'd like to use some experiment tracking tools to retrieve live metrics (e.g. wandb), but the default setup for wandb expects to have access to the wandb server from the flare server. In our case, this is blocked by a network policy in our environment.

We can set wandb into offline mode, and write the tracking logs into the job directory (by passing in the dir parameter in the init config to include {JOB_ID}), but this is only available at the end of the run, which makes tracking the progress of a multi-day training run quite challenging.

We've thought about polling and parsing the server logs, using session.download_job_result, but it's not ideal as we'd have to write our own wandb wrapper, and would ideally like to use flare built-ins.

I wondered if there's a better way to periodically download the wandb directory from the flare server while the job is still running?

[chesterxgchen]

WanB is not designed to be used in house. As far as I know it is designed as SaaS.
instead of using W&B, you could use Tensorboard or MLflow in this case, with MLflow, you can simply setup a Mlflow server with mySQL database internally. and you can specify the tracking URI to point to the MLFlow server. Then you can access the MLflow within your own network.

[ianh-apheris]

Hi @chesterxgchen , this is not about using wandb in-house, so much as providing a way to stream live job data from the flare server environment back to an authorised user (i.e. the person who submitted the job).

Pulling W&B data and syncing it to their web server is one use-case for this, but actually there are other applications, like allowing the user to download the latest global checkpoint during a run.

[chesterxgchen]

I see, I miss the title. but not sure pulling the global model snapshot is a valid use case. The global model is already broadcast to each client. If the client needs the model, the client can always save the snapshot, it might be a round delay. The aggregate the model will be broadcast again, user likes to pull while job is running instead of waiting for few second to get the new finished model ? the aggregation usually really fast. Not sure this the valid use case.

Let me know if there other use cases you would like to do this

[YuanTingHsieh]

@ianh-apheris thanks for this discussion.

@chesterxgchen @yhwen @yanchengnv asides from W&B design, I think we do have two general enhancement directions we can think of:

1. If users are writing custom files to the JOB directory (format other than the ones we supported by default), how can we enable them to download those files (with some authorization polices)? Right now we only allow downloading of ".txt", ".log"
2. Should we enable the admin CLI to download from the server at locations OTHER than the job directory (with some authorization polices)? Right now we only allow downloading files inside JOB.