Migrating away from the @sand4rt/web-ct-testing package to use native Playwright

[JoshuaNg2332]

This discussion serves as initial documentation and discussion around the migration from the use of the @sand4rt/experimental-ct-web package to the native playwright package.

The Problems

We implemented this package back in the early stages of pie when we needed to mount components quickly and be able to test them natively using their props. We chose this package because we saw the potential of this package being merged into the main playwright package which was created by a user who has a long lasting track record of having their ideas and experimental packages adopted directly by the Playwright team.

However, 6 months on, this has not happened and we are having balance dependancy managment across all our test related packages. This includes (but not limited to), Playwright test packages, Axe core packages and Percy dependancies. All these packages are reliant on each others versions lining up and this caused many issues with keep certain packages updated.

Updating packages caused further issues on running tests which is what brings us to this point.

The Solution

As our current implementation of mounting the component has yet to be officially adopted by the Playwright team, we've decided to move away from this form of loading our components and would go towards the official playwright option and leveraging our Storybook instances and testing on components by loading the components onto the page via our Storybook deployments.

This will give us some leverage over the easy tools we've got built into Storybook for loading component variants.
This also means that we can use more page object related refactors to simplify some of our code meaning that the tests will be more legible and easier to understand. We can extract key functionality that has been written into the tests into a component object making for easier implementation of functionality within the tests.

The potential cons

The main issue with this option would be how we load during CI. This would mean that we rely heavily on our deployments working as expected and we've had issues with our deployments in the past which could potentially cause issues. How we've been able to mitigate as much time lost as possible is by re-ordering our CI steps so that we only run these tests if Storybook is deployed. Component tests can't be run without the deployment working so this seems like the main change we've had to make within the CI steps.

Through our initial testing and implementation using Cookie Banner, we've been able to get this working as expected by using the base page object which then all components will then be able to use.

Where are we now and what comes next?

We're currently at a stage where we are looking to get hidden storybook instances in place. This will be for the visual tests and will allow us to, 1. Remove the active status banner on the top right of components when loaded in Storybook and 2. Create custom storybook wrapped components for listing our component variations for visual tests.
We do this in our current implementation whereby we wrap our components in a gridded wrapper for easier organising and easier identification of variations in the visual tests.

We need to do these things before we can proceed with updating the Cookie Banner Visual tests and then begin implementing the other tests across the other components.

Currently only 2 components are using component page objects

Tickets have been written to at least create the page objects and refactor tests. Creating component page objects for components that don't already have them can be done independently without any of the above changes.

[jamieomaguire]

Thank you for the write-up!

If I have understood correctly, we are stating that component browser tests will require Storybook to be deployed before they can run. I have some concerns about the potential impact this might have on the speed of our feedback loop. I hope this isn't a silly question, but could we not spin up a local Storybook instance during the test step on CI?

Also, just to confirm, do we need hidden Stories for the browser tests as well?

[JoshuaNg2332]

Totally not a silly question. Points are completely valid. I think during our initial conversations were that it would be better to test on our deployments however, it should be completely doable to create a local storybook instance in the test CI step. I think the only potential issue could be when it comes to spinning up multiple instances when tests are potentially running in parallel. I'm unsure of the behaviour in that instance but we can totally try!
If we go this route, alongside making hidden stories for browser tests, it would mean that during test instances it would be completely isolated which isn't a bad thing.

I do believe though that we don't necessarily need hidden stories for browser tests as we are primarily testing functionality which is something that should behave exactly the same as what we would see in our Storybook instances.

[JoshuaNg2332]

I've spun up a PR just to test the above of having CI load localhost instances for tests. You can check the progress and we can compare speeds here.