[Q&A] Need help debugging this error: 'ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'

[aayush-kapoor]

Python version (python3 -V)

3.9.18

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.1

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Rocky Linux 9.3

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I run into the following issue and don't know what to do when I was trying to run the nvflare simulator.

Here is the error log file that I encountered

2024-05-24 16:41:16,424 - SimulatorRunner - INFO - Create the Simulator Server.
2024-05-24 16:41:16,435 - CoreCell - INFO - server: creating listener on tcp://0:55319
2024-05-24 16:41:16,611 - CoreCell - INFO - server: created backbone external listener for tcp://0:55319
2024-05-24 16:41:16,612 - ConnectorManager - INFO - 3810396: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-05-24 16:41:16,616 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:17615] is starting
2024-05-24 16:41:17,119 - CoreCell - INFO - server: created backbone internal listener for tcp://localhost:17615
2024-05-24 16:41:17,120 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 PASSIVE tcp://0:55319] is starting
2024-05-24 16:41:17,266 - nvflare.fuel.hci.server.hci - INFO - Starting Admin Server localhost on Port 48767
2024-05-24 16:41:17,266 - SimulatorRunner - INFO - Deploy the Apps.
2024-05-24 16:41:17,483 - SimulatorRunner - INFO - Create the simulate clients.
2024-05-24 16:41:17,502 - ClientManager - INFO - Client: New client [email protected] joined. Sent token: 3da5f730-c3cc-49f7-8d19-dbe66c8d6ce3.  Total clients: 1
2024-05-24 16:41:17,504 - FederatedClient - INFO - Successfully registered client:site-1 for project simulator_server. Token:3da5f730-c3cc-49f7-8d19-dbe66c8d6ce3 SSID:
2024-05-24 16:41:17,507 - ClientManager - INFO - Client: New client [email protected] joined. Sent token: 5cea90a6-8878-4e28-9ce9-2eeed4c65b55.  Total clients: 2
2024-05-24 16:41:17,509 - FederatedClient - INFO - Successfully registered client:site-2 for project simulator_server. Token:5cea90a6-8878-4e28-9ce9-2eeed4c65b55 SSID:
2024-05-24 16:41:17,510 - SimulatorRunner - INFO - Set the client status ready.
2024-05-24 16:41:17,511 - SimulatorRunner - INFO - Deploy and start the Server App.
2024-05-24 16:41:17,515 - Cell - INFO - Register blob CB for channel='server_command', topic='*'
2024-05-24 16:41:17,516 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2024-05-24 16:41:17,517 - ServerCommandAgent - INFO - ServerCommandAgent cell register_request_cb: server.simulate_job
Using cache found in /home/akapo004/.cache/torch/hub/NVIDIA_DeepLearningExamples_torchhub
/home/akapo004/.cache/torch/hub/NVIDIA_DeepLearningExamples_torchhub/PyTorch/Classification/ConvNets/image_classification/models/common.py:13: UserWarning: pytorch_quantization module not found, quantization will not be available
  warnings.warn(
/home/akapo004/.cache/torch/hub/NVIDIA_DeepLearningExamples_torchhub/PyTorch/Classification/ConvNets/image_classification/models/efficientnet.py:17: UserWarning: pytorch_quantization module not found, quantization will not be available
  warnings.warn(
2024-05-24 16:41:27,881 - JsonScanner - ERROR - Traceback (most recent call last):
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 54, in instantiate_class
    instance = c(**init_params)
TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 99, in _do_scan
    node.processor.process_element(node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 150, in process_element
    self.process_config_element(self.config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_json_config.py", line 101, in process_config_element
    FedJsonConfigurator.process_config_element(self, config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed_json_config.py", line 91, in process_config_element
    c = self.authorize_and_build_component(element, config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 103, in authorize_and_build_component
    return self.build_component(config_dict)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_json_config.py", line 157, in build_component
    t = super().build_component(config_dict)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/component_builder.py", line 86, in build_component
    return instantiate_class(class_path, class_args)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 58, in instantiate_class
    raise ValueError(f"Class {class_path} has parameters error: {secure_format_exception(e)}.")
ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'.

Error processing config ['/home/akapo004/baremetal/./workspace/simulate_job/app_server/config/config_fed_server.json']: ConfigError: Error processing '/home/akapo004/baremetal/workspace/simulate_job/app_server/config/config_fed_server.conf' in element 'id = "json_generator"
path = "nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator"
args {
  num_rounds = 10
}': path: 'components.#5', exception: 'ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'.'
2024-05-24 16:41:27,924 - SimulatorRunner - ERROR - FL server execution exception: ConfigError: Error processing '/home/akapo004/baremetal/workspace/simulate_job/app_server/config/config_fed_server.conf' in element 'id = "json_generator"
path = "nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator"
args {
  num_rounds = 10
}': path: 'components.#5', exception: 'ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'.'
Traceback (most recent call last):
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 54, in instantiate_class
    instance = c(**init_params)
TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 99, in _do_scan
    node.processor.process_element(node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 150, in process_element
    self.process_config_element(self.config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_json_config.py", line 101, in process_config_element
    FedJsonConfigurator.process_config_element(self, config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed_json_config.py", line 91, in process_config_element
    c = self.authorize_and_build_component(element, config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 103, in authorize_and_build_component
    return self.build_component(config_dict)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_json_config.py", line 157, in build_component
    t = super().build_component(config_dict)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/component_builder.py", line 86, in build_component
    return instantiate_class(class_path, class_args)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 58, in instantiate_class
    raise ValueError(f"Class {class_path} has parameters error: {secure_format_exception(e)}.")
ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_app_runner.py", line 72, in start_server_app
    conf.configure()
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 147, in configure
    raise e
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 144, in configure
    self._do_configure()
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 137, in _do_configure
    self.json_scanner.scan(self)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 159, in scan
    self._do_scan(node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 129, in _do_scan
    self._do_scan(_child_node(node, k, 0, v))
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 132, in _do_scan
    self._do_scan(_child_node(node, node.key, i + 1, element[i]))
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 121, in _do_scan
    raise ConfigError(self.get_process_err_msg(e, elmt_str, location, node))
nvflare.fuel.common.excepts.ConfigError: Error processing '/home/akapo004/baremetal/workspace/simulate_job/app_server/config/config_fed_server.conf' in element 'id = "json_generator"
path = "nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator"
args {
  num_rounds = 10
}': path: 'components.#5', exception: 'ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'.'
2024-05-24 16:41:27,926 - SimulatorServer - INFO - Server app stopped.


Exception in thread Thread-7 (start_server_app):
Traceback (most recent call last):
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 54, in instantiate_class
    instance = c(**init_params)
TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 99, in _do_scan
    node.processor.process_element(node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 150, in process_element
    self.process_config_element(self.config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_json_config.py", line 101, in process_config_element
    FedJsonConfigurator.process_config_element(self, config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed_json_config.py", line 91, in process_config_element
    c = self.authorize_and_build_component(element, config_ctx, node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 103, in authorize_and_build_component
    return self.build_component(config_dict)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_json_config.py", line 157, in build_component
    t = super().build_component(config_dict)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/component_builder.py", line 86, in build_component
    return instantiate_class(class_path, class_args)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 58, in instantiate_class
    raise ValueError(f"Class {class_path} has parameters error: {secure_format_exception(e)}.")
ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/app/simulator/simulator_runner.py", line 481, in start_server_app
    server_app_runner.start_server_app(
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_app_runner.py", line 85, in start_server_app
    raise e
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/fed/server/server_app_runner.py", line 72, in start_server_app
    conf.configure()
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 147, in configure
    raise e
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 144, in configure
    self._do_configure()
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 137, in _do_configure
    self.json_scanner.scan(self)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 159, in scan
    self._do_scan(node)
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 129, in _do_scan
    self._do_scan(_child_node(node, k, 0, v))
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 132, in _do_scan
    self._do_scan(_child_node(node, node.key, i + 1, element[i]))
  File "/home/akapo004/envs/ppfl/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 121, in _do_scan
    raise ConfigError(self.get_process_err_msg(e, elmt_str, location, node))
nvflare.fuel.common.excepts.ConfigError: Error processing '/home/akapo004/baremetal/workspace/simulate_job/app_server/config/config_fed_server.conf' in element 'id = "json_generator"
path = "nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator"
args {
  num_rounds = 10
}': path: 'components.#5', exception: 'ValueError: Class nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator has parameters error: TypeError: ValidationJsonGenerator.__init__() got an unexpected keyword argument 'num_rounds'.'
2024-05-24 16:41:27,962 - SimulatorRunner - ERROR - Simulator run error: RuntimeError: Could not start the Server App.
2024-05-24 16:41:31,502 - MPM - INFO - MPM: Good Bye!

The simulator is trying to train efficientnet model for CIFAR10 dataset and I'm running it for 50 rounds.

All the config files are generated automatically when I run this create_job.sh file :

#!/bin/bash
#SBATCH -p gpu --gres=gpu:1
module load container_env pytorch-gpu


model_name=${31:-efficientnet}  # Default value is efficientnet if not provided
dataset_name=${2:-CIFAR10}  # Default value is CIFAR10 if not provided
num_rounds=${3:-50}  # Default value is 10 if not provided

crun -p ~/envs/ppfl nvflare config -jt ./job_templates

crun -p ~/envs/ppfl nvflare job create -force \
-j ./jobs -w sag_custom \
-sd ./code_job/ \
-f config_fed_client.conf \
executors[0].executor.args.model_name="$model_name" \
executors[0].executor.args.dataset_name="$dataset_name" \
executors[1].executor.args.model_name="$model_name" \
executors[1].executor.args.dataset_name="$dataset_name" \
-f config_fed_server.conf \
components[3].args.model_name="$model_name" \
components[4].args.num_rounds=$num_rounds

and then to run the simulation, I run the following file:

#!/bin/bash
#SBATCH -p gpu --gres=gpu:1
module load container_env pytorch-gpu



N=${1:-2}
t=${2:-2}

crun -p ~/envs/ppfl nvflare simulator -n $N -t $t ./jobs -w ./workspace

and then I get the error. I'm confused as to why this is happening, would love some help to figure out if it is because of me or NVFlare?

[YuanTingHsieh]

@aayush-kapoor Thanks for the question.
I see you are using our JOB CLI with your own "sag_custom" template.

The error message there means that, you try to pass the "num_rounds" to the component "ValidationJsonGenerator".
Seeing from your command, you are using components[4].args.num_rounds=$num_rounds
I think you pass the "num_rounds" to an incorrect component.

If you want, you can share your "sag_custom" template and we can check which one is the right component that you want to pass num_rounds to