[Q&A] NVFlare=2.5.0 and Python 3.10: COMMUNICATION_ERROR

[kun0906]

Python version (python3 -V)

3.10

NVFlare version (python3 -m pip list | grep "nvflare")

2.5.0

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.5

Operating system

Ubuntu 22.04.4 LTS

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Hello,

I would like to use tensorboard and follow this tutorial:
https://github.com/NVIDIA/NVFlare/blob/2.5.0/examples/advanced/experiment-tracking/tensorboard/jobs/tensorboard-streaming/app/config/config_fed_client.json

If I increase "epochs": from 5 to 10, I got the following error: COMMUNICATION_ERROR.

2024-09-10 23:08:51,324 - Cell - INFO - broadcast: channel='aux_communication', topic='task_check', targets=['server.bcd0ef1d-5e70-4b95-9d3e-1f998822bbe0'], timeout=5.0
2024-09-10 23:08:53,526 - ClientRunner - ERROR - [identity=site-4, run=bcd0ef1d-5e70-4b95-9d3e-1f998822bbe0, peer=example_project, peer_run=bcd0ef1d-5e70-4b95-9d3e-1f998822bbe0, task_name=train, task_id=610bd6e5-b1ce-45b0-b6ac-1ce29eefeb52]: failed task_check: COMMUNICATION_ERROR
2024-09-10 23:08:55,222 - ClientRunner - ERROR - [identity=site-7, run=bcd0ef1d-5e70-4b95-9d3e-1f998822bbe0, peer=example_project, peer_run=bcd0ef1d-5e70-4b95-9d3e-1f998822bbe0, task_name=train, task_id=a66d1352-027c-4180-a8ed-2f3b6d7fe3d5]: failed task_check: COMMUNICATION_ERROR
2024-09-10 23:08:55,787 - ClientRunner - ERROR - [identity=site-6, run=bcd0ef1d-5e70-4b95-9d3e-1f998822bbe0, peer=example_project, peer_run=bcd0ef1d-5e70-4b95-9d3e-1f998822bbe0, task_name=train, task_id=812b7baf-9262-423b-aa20-a0cc89adcbd0]: failed task_check: COMMUNICATION_ERROR

I have no idea for this issue.
Any help would be greatly appreciated!

[YuanTingHsieh]

Hi @kun0906 thanks for the question!

Could you attach your full log or full workspace.
Have you check memory usage / CPU / GPU usage?
Could there be resource issue? (for example out of memory or out of storage spaces)