[Q&A] Tracking of communication time

[LeoDuda]

Python version (python3 -V)

3.8

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.0rc7+187.g56c9c4d2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.4

Operating system

MacOS 14.5

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Hello,
I am trying to somehow track the communication time between clients and server, and also just between clients when using decentralized approaches like Swarm Learning. I am aware of the admin commands 'list_pools' and 'show_pool' and also reading from 'stats_pool_summary.json' which I don't find that trivial to understand. Is there something I haven't found yet that would help me getting those statistics? Thanks a lot.

Best,
Leo

[YuanTingHsieh]

@LeoDuda thank you very much for brining this up!

We will enhance our documentation on this.

Meanwhile, please find below explanation:

Command: list_pools

Stats pools collect real-time performance statistics of the cell net.

There are two types of stats pools: histogram pools and counter pools.

Hist pools are mainly used for time-based measurement: how many messages are processed within time ranges. They are also used for measuring message sizes: how many messages are within defined size ranges.

Counter pools are used to keep counters: how many different types of errors have happened.

The system predefined many pools. The “list_pools” command shows all defined pools.

> help list_pools
Server Commands
---------------
Command: cellnet.list_pools
Description: list stats pools
Usage: list_pools target

> list_pools server
--------------------------------------------------------------------------------------------------
| POOL                  | TYPE    | DESCRIPTION                                                  |
--------------------------------------------------------------------------------------------------
| msg_travel            | hist    | Time taken (secs) to get here (receiver)                     |
| received_msg_counters | counter | Result counters of received messages                         |
| received_msg_sizes    | hist    | Sizes of messages received (MBs)                             |
| request_processing    | hist    | Time spent (secs) by request processing callbacks (receiver) |
| request_response      | hist    | Request/response time in secs (sender)                       |
| sent_msg_counters     | counter | Result counters of sent messages                             |
| sent_msg_sizes        | hist    | Sizes of messages sent (MBs)                                 |
| sfm_send_frame        | hist    | SFM send_frame time in secs                                  |
--------------------------------------------------------------------------------------------------

Command: show_pool

With this command, you can show the details of a specified pool on a target cell.

> help show_pool
Server Commands
---------------
Command: cellnet.show_pool
Description: show stats pool detail
Usage: show_pool target pool_name [mode]

Examples

> show_pool server.2793f5d8-70e4-4dc8-888b-2f23df8c97cb msg_travel
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| CATEGORY                                                                     | 0.0001-0.0005 | 0.0005-0.001 | 0.001-0.002 | 0.002-0.004 | 0.004-0.008 | 0.008-0.01 | 0.01-0.02 |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| blue.2793f5d8-70e4-4dc8-888b-2f23df8c97cb#req:server_command:get_task        |               | 10           | 51          | 156         | 20          | 23         | 15        |
| blue.2793f5d8-70e4-4dc8-888b-2f23df8c97cb#req:server_command:submit_update   | 1             |              |             |             |             |            |           |
| red.2793f5d8-70e4-4dc8-888b-2f23df8c97cb#req:server_command:get_task         |               | 9            | 66          | 142         | 29          | 16         | 13        |
| red.2793f5d8-70e4-4dc8-888b-2f23df8c97cb#req:server_command:submit_update    |               | 2            |             |             |             |            |           |
| server#reply:server_parent_listener:get_clients                              | 1             |              |             |             |             |            |           |
| server#req:_net_manager:cells                                                |               | 1            |             |             |             |            |           |
| server#req:_net_manager:show_pool                                            | 1             |              |             |             |             |            |           |
| silver.2793f5d8-70e4-4dc8-888b-2f23df8c97cb#req:server_command:get_task      |               | 16           | 85          | 116         | 36          | 13         | 9         |
| silver.2793f5d8-70e4-4dc8-888b-2f23df8c97cb#req:server_command:submit_update | 1             |              |             |             |             |            |           |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Done [8872 usecs] 2023-08-20 12:21:13.112811

The msg_travel is an important pool that shows, from the target cell’s perspective, how fast messages travel to it.

The CATEGORY column encodes the types of the message:

<from_cell_fqcn>#<msg_type>:<msg_channel>:<msg_topic>

For example, the category “blue.2793f5d8-70e4-4dc8-888b-2f23df8c97cb#req:server_command:get_task”
Represents the “get_task” request sent to the server’s job cell from the blue client’s job cell.
The [mode] parameter (one of ['count', 'percent', 'avg', 'max', 'min']) of the command lets you see the stats in different ways:

count - count (the default)
percent - percentage
avg - average
max - maximum
min - minimum