A simple math in the reported number, the MsgRate and #iterations don't match

[shenweihai1]

I ran write_bw for 60s with a small message size 8bytes, and got statistics as below

                    RDMA_Write Post List BW Test
 Dual-port       : OFF		Device         : ib0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: Unsupported
 ibv_wr* API     : OFF
 TX depth        : 256
 Post List       : 128
 CQ Moderation   : 128
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 8          224958464        0.00               0.48   		   7.498617
---------------------------------------------------------------------------------------

According to the #iterations, we can do a simple math to get a MsgRate: 224958464/60/1000/1000=3.749M packet per second. It's just half of 7.498617. Can anyone tell me why I'm wrong?

[sshaulnv]

can you please share the full cmd's you used?

[yangrudan]

224958464
Did you test othter message size， such as 16, 32 bytes?

[shenweihai1]

I reran 4 times with different msg size: 8, 64, 512 and 65536, commands for msg-size:8 are:

# server
ib_write_bw -d ib0 -x 2 -p 50001 -D 64 -s 8 -q 1 --report_gbits -F
# client
ib_write_bw -d ib0 -x 2 -p 50001 -D 60 -s 8 -l 16 -n 16 -q 1 --report_gbits -F  1000:aaa:3456:1234::1

All logs are

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 8          466502432        0.00               1.00   		   15.550118
------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 64         466590640        0.00               7.96   		   15.553004
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 512        448422656        0.00               61.22  		   14.947421
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      10847344         0.00               189.57 		   0.361578
---------------------------------------------------------------------------------------

All iterations are half of the MsgRate. This is uni-directional workload.

----- UPDATE
I took a look at the code, perftest uses sum_of_test_cycles/cycles_to_units to count the runtime; however this (~30) can differ significantly from the actual 60-s duration.

I will try to saturate the Processing Unit to see if it makes any difference. I'm now curious whether this is just a coincidence that the measured time by cycles is just half of the duration when unsaturated.