Iperf3 server is getting stuck after printing the result #819
Comments
|
I have a slightly different behaviour: |
|
@janmaris: You should probably try your tests with a (much) newer version of iperf3. 3.6 is current as of this writing. @Kara88: If you are still having this problem I would need to know what version of iperf3 you're running, some infomration about the networking environment, and the client and server command-line arguments you are using. I am pretty sure these are two separate problems. |
|
Closing for lack of feedback. |
|
Hi, I think I have reproduced the same issue as janmaris on recent iperf3 3.12 version. Problem seems to only occur when we test over a mobile network (laptop -> OpenWRT Router with mobile -> internet -> Wired PC iperf3 server). Currently tested only on this one setup and can reproduce within a couple minutes, but I can make another one if needed. First we had this problem when using iperf3 3.7, but updating did not solve this issue. Issue does not seem to occur when using "normal" download mode and Laptop client: Laptop client netstat: PC server: PC server netstat: Issue is always reproduced on the end of the test, a prolonged test does not hang as often. I do not really see anyone else having any similar problems, so I would not be too surprised if the router software is at fault, however iperf should still preferably not hang forever. |
davidBar-On
added a commit
to davidBar-On/iperf
that referenced
this issue
May 28, 2023
|
Hi @ffolkes1911, It seems that the problem is caused because the client is not able to receive a control message from the server. This may be because network buffers are full somewhere (I noticed that for each 128KB packet sent, only about 60KB are really sent). I prepared a temporary iperf3 version with some enhancements and test messages. Can you build this version and run it both as client and server? If you can run this version can you share the results and the output? |
|
Hi @davidBar-On , I have reproduced issue with your version. There are a ton of logs, but I am refraining from putting them into files, so that people can find it from a search engine easier. The setup is identical to my previous comment. Laptop client okPC server okLaptop client failPC server failed |
|
Hi @ffolkes1911, thanks for the logs (and sorry for the previous message I sent by mistake which I deleted). |
|
If you're talking about PC server failed logs, then it should due to the added |
|
OK. Thanks. Unfortunately, I don't see much info that can help. One possibility is that this is because not all the test message are printed to the terminal. I therefore, added Can you test with this version? Sending the logs for the failed tests is enough. Also, it is sufficient to send the log records only at the end of the test:
|
Laptop clientPC serverI also added a messages around I added a print before |
|
The issue is that for some reason the client does not receive the What I don't understand is what was received by the client at the end. The message shows "AFTER select() state=4, result=1". "result=1" means that an input was received. Either we don't see all the messages or the client somehow got stuck when reading that input. I added additional test messages (including showing the fd that you suggested). Especially, added the list of the sockets (fd's) where input was received by |
|
Hello, here are the logs with new version. PC server.Laptop client. |
|
Hopefully I now understand why the client is stuck. At the end of the test, the client's sockets are set to blocking mode (I am not sure why). Since only about 60KB from each 128KB packet are sent by the server, the last packet that the client is receiving is not the full 128KB (TCP is bytes stream so each time the read get the next 128KB sent). In the new version I removed the blocking mode setting, so the client should be able to read the last partial packet without getting stack. Can you try this version? Note that I am not sure whether the test will succeed, as the |
|
Hi, from some initial testing it seems that iperf3 with @davidBar-On fix does not get stuck anymore, but I will leave a test running overnight to have more information. |
|
:-) well done!! |
|
Had 600 attempts on one test using iperf3 with latest fix, issue was not reproduced, where previously I could reproduce within ~10 attempts. At one point server seemed to randomly timeout or reject connections between tests, but it seems gone now. So fix seems to work, however cannot be certain about side effects, as we do not know why the client's sockets were set to blocking mode in the first place (mentioned by @davidBar-On ) |
|
@ffolkes1911, thanks a lot for your testings effort. I will submit a PR based on your test results and partial simulation of the issue I did (after understanding its cause).
I also run several tests and there was no issue with this change. My assumption that when blocking/non-blocking handling was added, this was added to the client to keep the functionality at the end of the test as it was before, without taking into account that the client process ends at the end of the test (the server on the other hand continues to the next test). In any case, I hope that when the iperf3 team will handle the PR they will be able to tell whether this change is o.k. |
|
Hi @ffolkes1911, I submitted PR #1527 with the changes you already tested but with some additional changes per my internal test (and removing the "[TEST]" messages). Can you test this version to make sure that the additional changes and cleanup do not cause issues in your environment and that the server and client still does not get stuck? Note that I added some of the test messages as debug messages in this PR. Also, hen testing with debug messages, the option |
|
Hi @davidBar-On, I have updated to your latest iperf3 version, ran 21k iterations with |
|
Hi @ffolkes1911, that's great. Thank a lot for your effort to test the changes. I hope too that the the changes will be merged soon. |
|
I tested with another application and I think your changes may have broken JSON functionality, as I get a segfault right after printing final results if I add Click this to collapse/fold.Problem is also reproduced without
Seems that the segfault also happens when |
|
@ffolkes1911, thanks for finding issue! The problem was that I added timers canceling, so they will not interrupt with the receive timeout, but forgot to mark the timers pointers as free. I don't remember if I forgot to test with UDP or didn't notice the error at the end ... The issue is now fixed. It will be great if you can run your original tests to make sure that the original issue is still fixed. (The change now is canceling the timers one by one instead of all active, so I hope no time remained active.) By the way, if you find other issues, it is better to report it in PR #1527 and not here. |
|
Hi, sorry for the delay, the setup is at work and I had a couple days off, but I'm also trying not miss anything during testing. I have checked the tests and exit codes using Also sorry for not following the correct procedure, first time contributing to opensource. |
|
@ffolkes1911, no problem. Your efforts for testing iperf3 are really appreciated. Also, if by "not following the correct procedure" you mean that you didn't log the last bug in the PR, this is really not an issue. Regarding the "unable to receive control message ..." or "Connection timeout", it may be related to the stability of the network you use. To further evaluate such issue, tcpdump and/or Wireshark should be used to evaluate the network traffic between the client and server. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After
rcv_tcp_congestion cubic, the iperf3 server is getting hanged.The text was updated successfully, but these errors were encountered: