chore(web): karma is deprecated, find an alternative test runner

[mcdurdin]

See also #10494

[jahorton]

https://github.com/karma-runner/karma#karma-is-deprecated-and-is-not-accepting-new-features-or-general-bug-fixes

[jahorton]

Sadly, as of this moment at least, BrowserStack's tutorials for establishing JS unit testing still utilize Karma.

https://www.browserstack.com/docs/automate/javascript-testing/getting-started

That is, there's no quick and easy reference for migrating to an alternative approach compatible with their service.

[jahorton]

One of the leads from Karma's deprecation page - Web Test Runner - does have a BrowserStack plugin: https://modern-web.dev/docs/test-runner/browser-launchers/browserstack/

Web Test Runner says:

For modern browsers, we recommend using other browser launchers, as they are a lot faster. Browserstack is a good option for testing on older browser versions.

Looking into some of its other recommendations, a testing engine called Playwright is notably compatible. It's very focused on running the "latest and greatest", though - it won't be helpful for testing legacy browsers. That said, it could allow us to offload our "modern browser" section of tests onto the build agents, bypassing any Browserstack connectivity issues. It also has full Chromium, WebKit, and Firefox-engine support - runnable from all major platforms. Apparently, it can run tests under simulated mobile environments with WebKit and Chromium, too.

The biggest issue I can see: it looks like it'll take some work to convert from the Karma test-specification format. But... that's just the nature of tech-debt, I guess.

I suppose the option always did exist with Karma, but I just realized that it's totally possible to make one config that sets up Playwright launchers and BrowserStack launchers - it's not like we'd have to specify only one. It's not clear how it handles cases where BrowserStack launching fails to connect yet, though.

[jahorton]

Extra thing of coolness for Web Test Runner: https://modern-web.dev/guides/test-runner/code-coverage/#coverage-browser-support

Runs against Chromium can give code coverage reports, and it does seem like the format is either c8-compatible directly or easily convertable to the same (based upon istanbul): https://modern-web.dev/docs/test-runner/writing-tests/code-coverage/

[jahorton]

A thought occurs: we have regular user testing now. Not only do we already have user & regression tests against Android 5.0 via Android Studio emulation... but BrowserStack does offer live testing, not just automation. We could just schedule semi-regular "test this live via BrowserStack" user tests / regression tests against specific legacy targets in order to validate that things still work on older devices. Essentially, it'd result in less-frequent legacy testing but still give us a semi-regular 'canary' for legacy-compat issues. From there, we could shift to only automatically testing against modern targets.

I'm not saying it's ideal, but... it would certainly be a practical decision, if naught else, and keeping the tests fully on our build agents should certainly help with flaky connectivity issues.

[jahorton]

Another caveat: there doesn't seem to be a public teamcity-reporter for Web Test Runner. That said, they've got decent docs on how to DIY one, so it's not the end of the world.

If we're willing to take a stronger shift in testing paradigm, use of playwright directly can come with some notable benefits. One really interesting one: automatic test generation via recording.

• https://playwright.dev/docs/codegen#introduction
• We could then write the tests with Mocha as the runner, issuing commands via playwright to execute tests.
• Mocha has a pre-existing teamcity-reporter, so test-output formatting wouldn't be a big concern.
• Playwright can also connect for code-coverage stuff: https://playwright.dev/docs/api/class-coverage
• They have some fairly strong VS Code integration via plugins; it's even possible to use VS Code as the code inspector, it seems.

Huh. And, it turns out that playwright... is published on GitHub by Microsoft.

They also have their own mocha-like test-definition structure we could use instead of driving things within mocha test-specs, though that of course means we hit the TC-reporter issue again. That said, their structure does make some nice guarantees about making sure the document is fully settled at each step, removing a lot of flakiness issues we've seen at times.

[mcdurdin]

Side Note: Karma sometimes eats all available memory on our build agents, for example today on ba-win10-64-s1-601:

Name	PID	Status	User name	CPU	Working set (memory)	Memory (active private working set)	Commit size	Command line	UAC virtualization
node.exe	8028	Running	bob	03 	4,540 K	4,372 K	1,156,644 K	"C:\Program Files\nodejs\node.exe" C:\BuildAgent\work\99b311828f4ee7c\keyman\node_modules\.bin/../karma/bin/karma start --log-level=debug --reporters teamcity,BrowserStack ./in_browser/CI.conf.cjs	Not allowed