PIE Automated Testing Frameworks RFC #223
Replies: 2 comments 1 reply
-
|
Have you considered bringing Consumer Web QAs to the discussion? I believe they use a different toolset and might also have some opinions. |
Beta Was this translation helpful? Give feedback.
-
|
During discussions between the QEs we've acknowledged the other frameworks used by Consumer Web QAs and are planning to build this framework out in such a way that exposes selectors so that consuming teams are able to use these selectors without needing to use our framework if that makes sense? These discussions are primarily for engineers on Design Systems and if other engineers want to use their own frameworks, we are helping them with these exposed selectors. |
Beta Was this translation helpful? Give feedback.
-
|
@davidpn11 Heya, just a heads up, we're looking to add this to Pie. Just wanted to loop you into this and give you and the Skip team a heads up. Let us know if you have any questions on the topic. |
Beta Was this translation helpful? Give feedback.
-
PIE Automated Testing Frameworks RFC
Status: In Review
Authors: Ben Siggery, Josh Ng, Amarpreet Sandhu
Problem Statement / Needs
Current status
With the new PIE design system project, we should look to re-evaluate our testing toolset. Currently, we’re using WebDriverIO to perform automated browser testing within Fozzie Components. Since this project's creation, there have been advancements in alternate testing tools. The testing requirements of PIE differ from Fozzie Components, so we should take this opportunity to look at which tools are better aligned with our test strategy.
We currently have WebDriverIO in the PIE monorepo serving our visual and system tests, however this was mainly added as a quick win to add testing capabilities due to the increase in developer activity.
Purpose of RFC
Which tool is best suited for the PIE monorepo, specifically the testing of PIE Microsite and Lit Web Components.
Whether alternative tools are more suitable for our existing test strategy.
Specific Proposal
Frameworks
We’ve chosen to do a comparison on three of the most popular JS testing libraries for browser testing.
Playwright
WebdriverIO
Cypress
Whilst other libraries exist, we believe the ones listed strike a balance between industry adoption, and having a growing and active user / contributor base.
Requirements
Musts
Syntax must be similar to our existing testing tooling with a minimal learning curve
While all testing tools have similar capabilities and work in much the same way, given our team's familiarity with frameworks that use the WebDriver protocol, we would prefer to work with a library that is more closely aligned to this API.
Must be compatible with 3rd party integrations
We use a number of third-party tools to facilitate testing within our existing stack. Our new framework must be compatible with, but not limited to:
Percy
Axe Core
Browserstack
GH Actions
TypeScript support
Given the adoption of TypeScript within the PIE monorepo, we’d like our chosen framework to have native support, as it provides consistency, and makes our Page Objects more robust, giving us the ability to provide static typing.
Cross-Browser compatibility
Given the variance in browser usage across our applications, it’s important that we have the capability to test against our supported browsers / versions.
Network mocking/interception
Our chosen framework should have mocking / interception capabilities so that we’re able to achieve reliable and repeatable tests that aren’t dependent on hitting real API endpoints that our components / Microsite may be dependent on.
Ability to generate HTML reports
Test reporting is crucial for assessing the health of a test suite, and provides useful information when identifying flaky tests, as well as diagnosing and troubleshooting failures, particularly within CI. We expect our chosen framework to provide test tooling, with screenshot / video playback as a minimum.
Thorough documentation with an active user base
It’s impossible to determine if a technology is ‘future-proof’, however the chances of longevity are increased if our chosen framework is backed by an active and growing user-base, with frequent contributions and reliable documentation.
Be able to provide reusable code snippets for consuming teams to construct system / E2E tests
Given our components are going to be used across multiple consuming applications, it’s important that we’re able to provide test utilities these teams can utilise to aid testing in their repos. These include publishable page objects, JSON files containing test selectors, and other sharable packages / config to reduce duplication and promote standardisation across consuming test frameworks.
Component testing support (mounting UI components)
Given how the modern web is architectured by constructing pages through UI components, our chosen test framework should provide functionality that allows us to mount components in isolation.
Automatic download of specified browser version
Our tests should be able to run independently both during local testing as well as through CI. As such, the tooling when run should automatically download a specified browser version that we have set.
Extensive debugging tooling
During testing, an extensive debugging tool is useful to help find out what happened during the test as well as to help engineers understand how their changes might have affected the tests.
Stable and reusable code using the PageObjects pattern. With the ability to expose via NPM packages for consumers using Playwright.
PageObjects are key for allowing our tests to work for all consuming teams. The testing platform needs to be able to support this so that when it comes to testing, it allows consuming teams to use the PageObjects in whatever testing environment they may be using.
Nice to have
Highly and fairly easy scalable solution.
Allowing us to have a scalable tool will allow us to prepare best for the future of the repo and be able to scale our testing accordingly based on the repo’s needs and potential future scope.
VSCode extensions (Debugging tooling / Test Runner)
Using VSCode extensions would allow engineers and QEs easy access to running specific tests on specific components across the monorepo as well as allow us to have a clean and easy debugging tool built into VSCode for ease of use.
Real device support
Real device testing support would allow us to test on actual devices meaning that we can have a clear idea of how our components are working and displaying on these devices.
Frameworks/Drivers
Comparison of drivers
Browsers
❗Cypress: Support is currently experimental in beta
Maintainers
Adherence to our requirements
** Not familiar with Cypress APIs
*** Requires 3rd party library
Main risks
Cypress
Can only test in a single tab.
Unfamiliar APIs.
Some features are behind a paywall.
No iFrame support.
Webkit support is still experimental.
No real device support.
No parallel support.
Cypress only supports JavaScript for creating test cases.
Cypress suggests to not use PageObjects.
No support for native ‘tab’ key presses without plugins.
Playwright
No real device support for iOS, meaning that Playwright does not have the ability to run the tests natively on a real iOS device if we were to run using tools such as Browserstack.
Library is still in its infancy.
Currently doesn’t have a support for natively mounting web components
Currently being worked on.
WebdriverIO
Since WebdriverIOs decision to deprecate Node12, changes to their API have resulted in instability.
Doesn’t have context of the browser such as waiting for network activity.
Design System example use cases
Salesforce Lightning
Salesforce’s Lightning Design system uses Playwright for its AXE accessibility testing.
IBM Carbon
IBM’s Carbon Design System uses Playwright for its component and e2e testing.
Supported CI Platforms
When considering our tools, we need to consider what CI platforms are supported by these tools.
Previously, we were using CircleCI and TeamCity for our CI platforms and more recently, due to the partnership we have with Github, a majority of the business is moving towards adopting GitHub Actions as its primary CI tooling.
All of the listed tools we are considering support the main CI platforms used at JET. For Pie, in line with the rest of the business and with the recommendation of CDE, we’re adopting the use of GitHub Actions as our primary CI tooling which all listed tools support.
Our proposal
Framework
We’re looking into Playwright as our testing framework of choice. This tooling is most closely aligned with our testing strategy and needs. It works most similarly to WebdriverIO and meets our criteria across our various requirements. It also has extensive support from Microsoft in the form of being well documented as well as has many plugins that allow us to add additional functionality such as AXE testing into the repo.
The full list of pros and cons of Playwright can seen here.
Pros
Most closely aligned with our testing strategy and needs.
Extensive support from Microsoft with lots of documentation.
Plugins/integrations support
Axe
Percy
Browserstack
Trace reporting
Breakdown report of each test
Step by step instructions of what each test is doing
Metadata of each action (i.e. how long each action took)
Screenshots with a timeframe viewer
Network activity monitor at each step
Console output
Cons
Does not support Apple devices natively
Recently added the functionality to support android devices.
Does not natively support Teamcity
This means that we won’t be able to adopt Playwright into any tool that uses this CI platform unless using a 3rd party package and Playwright’s docker image implementation.
When looking at the alternative tools, there were several points that were of key detriment to us considering these tools within the PIE and web component repo.
Whilst WebdriverIO has served us well in our previous projects, when we looked into the requirements,
the documentation and the recent updates and releases made to WDIO introduced some instability and flakiness to our testing which ultimately led to us deciding against WDIO at this point.
Also when looking at our requirements, some of the requirements we have can only be solved using 3rd party libraries, which aren’t officially supported (or maintained) by WebdriverIO.
When considering Cypress, ultimately,
There were too many limitations to the toolset such as potential paywall blockages for capabilities,
The limitations to parallel running of tests,
No iframe support,
As well as no production ready Webkit support for browsers such as Safari.
This along with the fact that the API is vastly different from WebdriverIO/Playwright, is why we chose not to go with this option.
Whilst we’ve chosen Playwright as our testing platform, this doesn’t mean that consuming teams are unable to use the alternative tools. During the implementation, we’ll be looking to expose selectors and objects in a way that allows the consuming teams to use tools such as cypress within their teams giving ultimate freedom to consuming teams to use what they deem is best for their use case.
Given the research that has been done into these tools above, we feel that Playwright is the tool that best fits our needs on PIE and can allow us the widest opportunity to expand the testing suite in the future as it will currently support all the required testing we currently need as well as cover additional testing areas in the future for both us and also for our consuming teams.
Framework Limitations
Real device Testing
When deciding on Playwright, we still considered the additional features we would like (see section, ‘Nice to haves’) in our tooling. Whilst Playwright does have some of these, it does lack real device support.
Historically as part of our test strategy, real device testing via tools such as browserstack have always been a requirement before JET test frameworks. Given the nature of our design system, it is unlikely consumers will ever view our components in isolation. From our research looking at other design systems, we believe that emulating mobile devices and testing relevant viewports via visual regression is a suitable compromise.
Whilst we won’t be doing real device testing within our repo, we see this as something that would be covered by consuming teams during their testing stages once components have been integrated onto a page within an application.
API differences
Within our other repos, we use testing libraries that utilise the Webdriver protocol, such as WebdriverIO. One of the things we considered during this process is ease of adoption from engineers. Whilst Playwright doesn’t utilise the Webdriver protocol, and may require a small amount of learning, its API is closer aligned when compared to alternate tools such as Cypress.
Working examples in PIE
System Tests
PIE Button tests
PIE Microsite navigation tests
Visual Tests
Axe Tests
Beta Was this translation helpful? Give feedback.
All reactions