Testbed failure in test_window_state_rapid_assignment

[mhsmith]

This test was merged 2 days ago from #2473 by @proneon267.

It's failing intermittently on GitHub Actions:

• https://github.com/beeware/toga/actions/runs/12153792178/
• https://github.com/beeware/toga/actions/runs/12142884949/attempts/1

But it's 100% reproducible locally for me:

% briefcase run --test -ur -- tests/window/test_window.py -k 'rapid'
...
[testbed] Starting test_suite...
===========================================================================
tests/window/test_window.py: Don't know how to open documents with extension .py
Waiting for app to be ready for testing... ready.
============================= test session starts ==============================
collecting ... collected 50 items / 46 deselected / 4 selected

tests/window/test_window.py::test_window_state_rapid_assignment[MainWindow-second_window_kwargs0-states0] PASSED [ 25%]
tests/window/test_window.py::test_window_state_rapid_assignment[MainWindow-second_window_kwargs0-states1] FAILED [ 50%]
tests/window/test_window.py::test_window_state_rapid_assignment[MainWindow-second_window_kwargs0-states2] PASSED [ 75%]
tests/window/test_window.py::test_window_state_rapid_assignment[MainWindow-second_window_kwargs0-states3] PASSED [100%]

=================================== FAILURES ===================================
_ test_window_state_rapid_assignment[MainWindow-second_window_kwargs0-states1] _
Traceback (most recent call last):
...
  File "/Users/msmith/git/beeware/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app/tests/window/test_window.py", line 812, in test_window_state_rapid_assignment
    assert second_window_probe.instantaneous_state == states[-1]
AssertionError: assert <WindowState.NORMAL: 0> == <WindowState.MINIMIZED: 1>
 +  where <WindowState.NORMAL: 0> = <tests_backend.window.WindowProbe object at 0x12c71adc0>.instantaneous_state
---------------------------- Captured stdout setup -----------------------------
Closing window
Resetting main_window
Window (Secondary Window) has been created
----------------------------- Captured stdout call -----------------------------
Secondary window is visible
Secondary window is in WindowState.MINIMIZED
=========================== short test summary info ============================
FAILED tests/window/test_window.py::test_window_state_rapid_assignment[MainWindow-second_window_kwargs0-states1] - assert <WindowState.NORMAL: 0> == <WindowState.MINIMIZED: 1>
 +  where <WindowState.NORMAL: 0> = <tests_backend.window.WindowProbe object at 0x12c71adc0>.instantaneous_state
================= 1 failed, 3 passed, 46 deselected in 10.00s ==================

When the test runs, I get an error sound effect, the same one as if you press Cmd-C in the TextEdit app with nothing selected.

[proneon267]

I am looking into this and will report back soon.

[mhsmith]

Now a different but related test has failed on macOS x86_64:

• https://github.com/beeware/toga/actions/runs/12188827849/job/34003382310

[proneon267]

I couldn't reproduce this on my machine locally. On the CI also, I couldn't reproduce it: https://github.com/proneon267/toga/actions/runs/12190500356/job/34007856201. Could you re-run the failed job to make sure that the failure wasn't random?

Also, the test passes successfully on the CI from another PR: https://github.com/beeware/toga/actions/runs/12190718820/job/34008472836

If the tests still fail then the solution would be to do the window state cleanup at the end of the test itself, instead of relying on window_cleanup to reset the window state. I'll try this once we confirm that the failure still exists.

[freakboy3742]

It's definitely an intermittent failure; being hard to reproduce doesn't mean it doesn't exist, though.

I've done a bunch of rebuilds on the pass that failed, and there's been a couple of other CI passes since then, and I haven't seen another failure.

It's also interesting to note that although the general failure mode is similar, the failure was on the slow test, not the rapid one. This suggests to me that the issue might have been a momentary slowdown on the machine (paging something out of cache, or something), so the GUI didn't respond inside the expected time window.

In the process of investigating the "rapid" fix, it occurred to me that especially for these resize operations, we might be better served implementing wait_for_window as a polling operation, checking every (say) 0.1s, but with a very generous timeout (say 5s). That way, the test will complete as fast as the GUI allows, but will wait if there's a momentary slowdown. The current solution of a fixed interval works in most cases, but clearly can fail in the worst case, and is (by necessity) not as fast as it could be in the best case.

[proneon267]

I agree, polling based waiting seems to be the ideal solution to these types of intermittent failures. I'll research more about it and report back.

[mhsmith]

Here's another CI failure, this time on Linux Wayland:

• https://github.com/beeware/toga/actions/runs/12285892648/job/34285064242

  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/noble/testbed-0.0.1/usr/lib/testbed/app/tests/window/test_window.py", line 812, in test_window_state_rapid_assignment
    assert second_window_probe.instantaneous_state == states[-1]
AssertionError: assert <WindowState.NORMAL: 0> == <WindowState.PRESENTATION: 4>
 +  where <WindowState.NORMAL: 0> = <tests_backend.window.WindowProbe object at 0x7ffabc2c1430>.instantaneous_state
---------------------------- Captured stdout setup -----------------------------
Closing window
Resetting main_window
Window (Secondary Window) has been created
----------------------------- Captured stdout call -----------------------------
Secondary window is visible
Secondary window is in WindowState.PRESENTATION
=========================== short test summary info ============================
FAILED tests/window/test_window.py::test_window_state_rapid_assignment[MainWindow-second_window_kwargs0-states2] - assert <WindowState.NORMAL: 0> == <WindowState.PRESENTATION: 4>
 +  where <WindowState.NORMAL: 0> = <tests_backend.window.WindowProbe object at 0x7ffabc2c1430>.instantaneous_state
====== 1 failed, 489 passed, 70 skipped, 34 xfailed in 211.05s (0:03:31) =======