Unexpectedly low test results with Firefox upload test

[untoldone]

We have a user we're supporting who mentioned they're seeing consistently slower upload results on specifically Firefox. I asked them to try speed.measurementlab.net specifically to test to remove any possibility of the bug being on our side of things/ with our js-based client but they are still consistently seeing upload test results at about 1/3rd of what they see with Chrome and Edge (Chromium based latest versions). They are on a link which normally sees 600-800mbps and are currently getting uploads in the 150-250mbps range
Any other possibilities here as well? I can try to find some more locations to test this at as well but curious if there's currently known on this front or something perhaps we're doing wrong or known network configuration that might cause this type of issue.

This same result was seen in the Google Speed Test widget when you search for "speed test"

I don't currently have specific Firefox version details but it was tested on two different windows machine -- one being a clean installation of windows / firefox. Let me know if you need more specifics to reproduce this.

Note this was also discussed on the m-lab slack at: https://measurementlab.slack.com/archives/C01SMA30M7W/p1683662733546979

[robertodauria]

I've been investigating this for a while now, and there is indeed a performance issue with Firefox on Windows. While noting that both Firefox and Chrome are much slower on Windows that they are on Linux when running against a server running on localhost, in my testing on Windows Firefox was limited to ~600/700 Mb/s while Chrome reached ~1100/1200 Mb/s.

I do not have a solution yet, so just to summarize what I know so far:

I initially thought it could be due to the slower event loop rescheduling of the uploader function: while Chrome runs it every ~4ms (with some sporadic 60ms or so peaks), Firefox schedules the function every ~15ms. This could theoretically limit how often sock.send() gets called, since it's at most one call per iteration, and also gave me ideas on how to check for cpu-bound clients.

However:

1. Even considering a sock.send() every 15ms, with 8MB messages we'd still be able to support over 500 MBytes/s
2. I instrumented that code and found that except at the very beginning of the measurement (before message size is scaled up) the WebSocket send buffer is almost always full - i.e., sock.send() is only called every N iterations, and the buffer contains plenty of data to send at all times.

These seem to rule out the idea that the reduced performance could be due to the delayed schedule of uploader().

Playing with message size / buffer size only seems to make things worse -- not too surprisingly, as the current values were chosen after many hours of testing.

I also tried rewriting the uploader code splitting it into functions scheduled with setInterval() less aggressively (except for the main sock.send()) which, IIUC, basically takes advantage of the JS event scheduling loop instead of trying to run our own on top of it. This increased the performance on both Firefox and Chrome by maybe 10% and it's something I think worth merging into ndt7-js, but the difference between the two browsers was still there.

We may have reached the point where there isn't much more we can do on the Javascript side, and we might consider opening a bug report similarly to how we did with Safari/Webkit under similar circumstances in the past.

Finally, I went through pretty much every version of Firefox released in the past 3 years:

1. My modified uploader code ran much faster (almost on par with Chrome) between version 73.0.1 (Feb 2020) and version 81. Unfortunately that performance improvement disappears with Firefox >= 83. Unclear why, as the release notes don't seem to mention anything WebSocket-related. I'd also guess one uses these pretty old versions anymore.
2. There were no significant performance differences in ndt7-js between Firefox 73.0.1 (Feb 2020) and the most recent version.