Skip to content

Commit

Permalink
Add e2e infrastructure (#337)
Browse files Browse the repository at this point in the history
  • Loading branch information
swissspidy authored Feb 25, 2020
1 parent 62dd638 commit 23e6783
Show file tree
Hide file tree
Showing 15 changed files with 456 additions and 30 deletions.
1 change: 0 additions & 1 deletion .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,4 @@
**/vendor/**
**/assets/js/*.js
build/*
tests/*
!.storybook
1 change: 0 additions & 1 deletion .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
/docs
includes/data/fonts.json
/node_modules
/tests
/vendor
/assets/css
/assets/js
Expand Down
14 changes: 12 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,12 @@ env:

install:
- |
if [ "$LINT" = "1" ] || [ "$JS" = "1" ] || [ "$DEPLOY_PLUGIN" = "1" ]; then
if [ "$LINT" = "1" ] || [ "$JS" = "1" ] || [ "$E2E" = "1" ] || [ "$DEPLOY_PLUGIN" = "1" ]; then
nvm install
npm ci
fi
- |
if [ "$LINT" = "1" ] || [ "$PHP" = "1" ] || [ "$DEPLOY_PLUGIN" = "1" ]; then
if [ "$LINT" = "1" ] || [ "$PHP" = "1" ] || [ "$E2E" = "1" ] || [ "$DEPLOY_PLUGIN" = "1" ]; then
composer install --no-interaction
fi
- |
Expand Down Expand Up @@ -102,6 +102,13 @@ script:
if [ "$ANALYSIS" = "1" ]; then
vendor/bin/phpstan analyze --memory-limit=256M
fi
- |
if [ "$E2E" = "1" ]; then
npm run build:js
npm run env:start
npm run env:e2e:ci
npm run env:stop
fi
jobs:
fast_finish: true
Expand Down Expand Up @@ -136,6 +143,9 @@ jobs:
if: branch = master AND type = push
php: 7.4
env: JS=1 PHP=1 COVERAGE=1 WP_VERSION=latest
- name: E2E tests
php: 7.4
env: E2E=1 WP_VERSION=latest PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=
- stage: deploy
if: (NOT type IN (pull_request)) AND (branch = master)
php: 7.3
Expand Down
1 change: 1 addition & 0 deletions bin/local-env/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ services:
- wordpress_data:/var/www/html
- ./uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
- ../../:/var/www/html/wp-content/plugins/web-stories
- ../../tests/e2e/plugins:/var/www/html/wp-content/plugins/web-stories-test-plugins
depends_on:
- mysql

Expand Down
39 changes: 39 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,45 @@ npm run test:js:watch

See `npm run test:js:help` to get a list of additional options that can be passed to the test runner.

#### End-to-End Tests

This project leverages the local Docker-based environment to facilitate end-to-end (e2e) testing using Puppeteer.

To run the full test suite, you can use the following command:

```bash
npm run test:e2e
```

You can also watch for any file changes and only run tests that failed or have been modified:

```bash
npm run test:e2e:watch
```

Not using the built-in local environment? You can also pass any other URL to run the tests against. Example:

```bash
npm run test:e2e -- --wordpress-base-url=https://my-amp-dev-site.local
```

For debugging purposes, you can also run the E2E tests in non-headless mode:

```bash
npm run test:e2e:interactive
```

Note that this will also slow down all interactions during tests by 80ms. You can control these values individually too:

```bash
PUPPETEER_HEADLESS=false npm run test:e2e # Interactive mode, normal speed.
PUPPETEER_SLOWMO=200 npm run test:e2e # Headless mode, slowed down by 200ms.
```

Sometimes one might want to test additional scenarios that aren't possible in a WordPress installation out of the box. That's why the test setup allows for for adding some utility plugins that can be activated during E2E tests.

These plugins can be added to `tests/e2e/plugins` and then activated via the WordPress admin.

#### Storybook

The latest version of the project's [storybook](https://storybook.js.org/) can be found at [https://google.github.io/web-stories-wp/](https://google.github.io/web-stories-wp/).
Expand Down
54 changes: 54 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 8 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,11 @@
"@wordpress/babel-preset-default": "^4.10.0",
"@wordpress/components": "^9.2.2",
"@wordpress/dom-ready": "^2.7.0",
"@wordpress/e2e-test-utils": "^4.3.0",
"@wordpress/element": "^2.11.0",
"@wordpress/eslint-plugin": "^4.0.0",
"@wordpress/i18n": "^3.9.0",
"@wordpress/jest-puppeteer-axe": "^1.6.0",
"@wordpress/scripts": "^7.1.2",
"@wordpress/url": "^2.11.0",
"babel-eslint": "^10.0.3",
Expand Down Expand Up @@ -134,7 +136,12 @@
"test:js:help": "wp-scripts test-unit-js --help",
"test:js:watch": "npm run test:js -- --watch",
"test:php": "vendor/bin/phpunit",
"test:php:help": "npm run test:php -- --help"
"test:php:help": "npm run test:php -- --help",
"test:e2e": "WP_BASE_URL=http://localhost:8899 wp-scripts test-e2e --config=tests/e2e/jest.config.js",
"test:e2e:help": "npm run test:e2e -- --help",
"test:e2e:watch": "npm run test:e2e -- --watch",
"test:e2e:interactive": "npm run test:e2e -- --puppeteer-interactive",
"test:e2e:ci": "npm run test:e2e -- --runInBand"
},
"husky": {
"hooks": {
Expand Down
185 changes: 185 additions & 0 deletions tests/e2e/config/bootstrap.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
/*
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

/**
* External dependencies
*/
// eslint-disable-next-line import/no-extraneous-dependencies
import { setDefaultOptions } from 'expect-puppeteer';

/**
* WordPress dependencies
*/
import {
enablePageDialogAccept,
setBrowserViewport,
} from '@wordpress/e2e-test-utils';

/**
* Environment variables
*/
const { PUPPETEER_TIMEOUT, EXPECT_PUPPETEER_TIMEOUT } = process.env;

/**
* Set of console logging types observed to protect against unexpected yet
* handled (i.e. not catastrophic) errors or warnings. Each key corresponds
* to the Puppeteer ConsoleMessage type, its value the corresponding function
* on the console global object.
*
* @type {Object<string,string>}
*/
const OBSERVED_CONSOLE_MESSAGE_TYPES = {
warning: 'warn',
error: 'error',
};

/**
* Array of page event tuples of [ eventName, handler ].
*
* @type {Array}
*/
const pageEvents = [];

// The Jest timeout is increased because these tests are a bit slow
jest.setTimeout(PUPPETEER_TIMEOUT || 100000);

// Set default timeout for individual expect-puppeteer assertions. (Default: 500)
setDefaultOptions({ timeout: EXPECT_PUPPETEER_TIMEOUT || 500 });

async function setupBrowser() {
// 15inch screen.
await setBrowserViewport({
width: 1680,
height: 948,
});
}

/**
* Adds an event listener to the page to handle additions of page event
* handlers, to assure that they are removed at test teardown.
*/
function capturePageEventsForTearDown() {
page.on('newListener', (eventName, listener) => {
pageEvents.push([eventName, listener]);
});
}

/**
* Removes all bound page event handlers.
*/
function removePageEvents() {
pageEvents.forEach(([eventName, handler]) => {
page.removeListener(eventName, handler);
});
}

/**
* Adds a page event handler to emit uncaught exception to process if one of
* the observed console logging types is encountered.
*/
function observeConsoleLogging() {
page.on('console', (message) => {
const type = message.type();
if (!OBSERVED_CONSOLE_MESSAGE_TYPES.hasOwnProperty(type)) {
return;
}

let text = message.text();

// styled-components warns about dynamically created components.
// @todo Fix issues.
if (text.includes(' has been created dynamically.')) {
return;
}

const logFunction = OBSERVED_CONSOLE_MESSAGE_TYPES[type];

// As of Puppeteer 1.6.1, `message.text()` wrongly returns an object of
// type JSHandle for error logging, instead of the expected string.
//
// See: https://github.com/GoogleChrome/puppeteer/issues/3397
//
// The recommendation there to asynchronously resolve the error value
// upon a console event may be prone to a race condition with the test
// completion, leaving a possibility of an error not being surfaced
// correctly. Instead, the logic here synchronously inspects the
// internal object shape of the JSHandle to find the error text. If it
// cannot be found, the default text value is used instead.
text = message.args()?.[0]?._remoteObject?.description || text;

// Disable reason: We intentionally bubble up the console message
// which, unless the test explicitly anticipates the logging via
// @wordpress/jest-console matchers, will cause the intended test
// failure.

// eslint-disable-next-line no-console
console[logFunction](text);
});
}

/**
* Runs Axe tests when the story editor is found on the current page.
*
* @return {?Promise} Promise resolving once Axe texts are finished.
*/
async function runAxeTestsForStoriesEditor() {
if (!(await page.$('body.edit-story'))) {
return;
}

await expect(page).toPassAxeTests({
// Temporary disabled rules to enable initial integration.
disabledRules: [
'aria-allowed-role',
'aria-input-field-name',
'aria-required-parent',
'button-name',
'color-contrast',
'label',
'landmark-banner-is-top-level',
'landmark-no-duplicate-banner',
'landmark-unique',
'page-has-heading-one',
'region',
],
});
}

/**
* Before every test suite run, delete all content created by the test. This ensures
* other posts/comments/etc. aren't dirtying tests and tests don't depend on
* each other's side-effects.
*/
// eslint-disable-next-line jest/require-top-level-describe
beforeAll(async () => {
capturePageEventsForTearDown();
enablePageDialogAccept();
observeConsoleLogging();
await setupBrowser();
await page.setDefaultNavigationTimeout(10000);
await page.setDefaultTimeout(3000);
});

// eslint-disable-next-line jest/require-top-level-describe
afterEach(async () => {
await runAxeTestsForStoriesEditor();
await setupBrowser();
});

// eslint-disable-next-line jest/require-top-level-describe
afterAll(() => {
removePageEvents();
});
Loading

0 comments on commit 23e6783

Please sign in to comment.