diff --git a/docs/cli.md b/docs/cli.md index 19d176865..01d3487fe 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -4,59 +4,66 @@ ### Contents -- [Overview](#overview) -- [Reporters](#reporters) -- [Require modules](#require-modules) +- [`testplane` command](#testplane-command) + - [Options](#options) + - [`--reporter=`](#--reportername) + - [`--require=`](#--requiremodule) + - [`--inspect`](#--inspect) + - [`--repl`](#--repl) + - [switchToRepl](#switchtorepl) + - [Test development in runtime](#test-development-in-runtime) + - [How to set up using VSCode](#how-to-set-up-using-vscode) + - [How to set up using Webstorm](#how-to-set-up-using-webstorm) +- [`list-tests` command](#list-tests-command) + - [Options](#options-1) + - [`--formatter=`](#--formattername) - [Overriding settings](#overriding-settings) -- [Debug mode](#debug-mode) -- [REPL mode](#repl-mode) - - [switchToRepl](#switchtorepl) - - [Test development in runtime](#test-development-in-runtime) - - [How to set up using VSCode](#how-to-set-up-using-vscode) - - [How to set up using Webstorm](#how-to-set-up-using-webstorm) - [Environment variables](#environment-variables) - [TESTPLANE_SKIP_BROWSERS](#testplane_skip_browsers) - [TESTPLANE_SETS](#testplane_sets) -### Overview +### `testplane` command -``` -testplane --help -``` +Main command to run tests. -shows the following +```bash +> testplane --help -``` - Usage: testplane [options] [paths...] + Usage: testplane [options] [command] [paths...] + + Run tests Options: - -V, --version output the version number - -c, --config path to configuration file - -b, --browser run tests only in specified browser - -s, --set run tests only in the specified set - -r, --require require a module before running `testplane` - --reporter test reporters - --grep run only tests matching the pattern - --update-refs update screenshot references or gather them if they do not exist ("assertView" command) - --inspect [inspect] nodejs inspector on [=[host:]port] - --inspect-brk [inspect-brk] nodejs inspector with break at the start - --repl [type] run one test, call `browser.switchToRepl` in test code to open repl interface (default: false) - --repl-before-test [type] open repl interface before test run (default: false) - --repl-on-fail [type] open repl interface on test fail only (default: false) - -h, --help output usage information + -V, --version output the version number + -c, --config path to configuration file + -b, --browser run tests only in specified browser + -s, --set run tests only in the specified set + -r, --require require module + --grep run only tests matching the pattern + --reporter test reporters + --update-refs update screenshot references or gather them if they do not exist ("assertView" command) + --inspect [inspect] nodejs inspector on [=[host:]port] + --inspect-brk [inspect-brk] nodejs inspector with break at the start + --repl [type] run one test, call `browser.switchToRepl` in test code to open repl interface (default: false) + --repl-before-test [type] open repl interface before test run (default: false) + --repl-on-fail [type] open repl interface on test fail only (default: false) + --devtools switches the browser to the devtools mode with using CDP protocol + -h, --help output usage information ``` For example, ``` -testplane --config ./config.js --reporter flat --browser firefox --grep name +npx testplane --config ./config.js --reporter flat --browser firefox --grep name ``` **Note.** All CLI options override config values. -### Reporters +#### Options + +##### `--reporter=` You can choose `flat`, `plain` or `jsonl` reporter by option `--reporter`. Default is `flat`. Information about test results is displayed to the command line by default. But there is an ability to redirect the output to a file, for example: @@ -74,45 +81,14 @@ Information about each report type: * `plain` – information about fails and retries would be placed after each test; * `jsonl` - displays detailed information about each test result in [jsonl](https://jsonlines.org/) format. -### Require modules +##### `--require=` Using `-r` or `--require` option you can load external modules, which exists in your local machine, before running `testplane`. This is useful for: - compilers such as TypeScript via [ts-node](https://www.npmjs.com/package/ts-node) (using `--require ts-node/register`) or Babel via [@babel/register](https://www.npmjs.com/package/@babel/register) (using `--require @babel/register`); - loaders such as ECMAScript modules via [esm](https://www.npmjs.com/package/esm). -### Overriding settings - -All options can also be overridden via command-line flags or environment variables. Priorities are the following: - -* A command-line option has the highest priority. It overrides the environment variable and config file value. - -* An environment variable has second priority. It overrides the config file value. - -* A config file value has the lowest priority. - -* If there isn't a command-line option, environment variable or config file option specified, the default is used. - -To override a config setting with a CLI option, convert the full option path to `--kebab-case`. For example, if you want to run tests against a different base URL, call: - -``` -testplane path/to/mytest.js --base-url http://example.com -``` - -To change the number of sessions for Firefox (assuming you have a browser with the `firefox` id in the config): - -``` -testplane path/to/mytest.js --browsers-firefox-sessions-per-browser 7 -``` - -To override a setting with an environment variable, convert its full path to `snake_case` and add the `testplane_` prefix. The above examples can be rewritten to use environment variables instead of CLI options: - -``` -testplane_base_url=http://example.com testplane path/to/mytest.js -testplane_browsers_firefox_sessions_per_browser=7 testplane path/to/mytest.js -``` - -### Debug mode +##### `--inspect` In order to understand what is going on in the test step by step, there is a debug mode. You can run tests in this mode using these options: `--inspect` and `--inspect-brk`. The difference between them is that the second one stops before executing the code. @@ -124,7 +100,7 @@ testplane path/to/mytest.js --inspect **Note**: In the debugging mode, only one worker is started and all tests are performed only in it. Use this mode with option `sessionsPerBrowser=1` in order to debug tests one at a time. -### REPL mode +##### `--repl` Testplane provides a [REPL](https://en.wikipedia.org/wiki/Read–eval–print_loop) implementation that helps you not only to learn the framework API, but also to debug and inspect your tests. In this mode, there is no timeout for the duration of the test (it means that there will be enough time to debug the test). It can be used by specifying the CLI options: @@ -132,7 +108,7 @@ Testplane provides a [REPL](https://en.wikipedia.org/wiki/Read–eval–print_lo - `--repl-before-test` - the same as `--repl` option except that REPL interface opens automatically before run test. Disabled by default; - `--repl-on-fail` - the same as `--repl` option except that REPL interface opens automatically on test fail. Disabled by default. -#### switchToRepl +###### switchToRepl Browser command that stops the test execution and opens REPL interface in order to communicate with browser. For example: @@ -200,7 +176,7 @@ Another command features: // foo: 1 ``` -#### Test development in runtime +###### Test development in runtime For quick test development without restarting the test or the browser, you can run the test in the terminal of IDE with enabled REPL mode: @@ -214,16 +190,184 @@ Also, during the test development process, it may be necessary to execute comman - [clearSession](#clearsession) - clears session state (deletes cookies, clears local and session storages). In some cases, the environment may contain side effects from already executed commands; - [reloadSession](https://webdriver.io/docs/api/browser/reloadSession/) - creates a new session with a completely clean environment. -##### How to set up using VSCode +###### How to set up using VSCode 1. Open `Code` -> `Settings...` -> `Keyboard Shortcuts` and print `run selected text` to search input. After that, you can specify the desired key combination 2. Run `testplane` in repl mode (examples were above) 3. Select one or mode lines of code and press created hotkey -##### How to set up using Webstorm +###### How to set up using Webstorm Ability to run selected text in terminal will be available after this [issue](https://youtrack.jetbrains.com/issue/WEB-49916/Debug-JS-file-selection) will be resolved. +### `list-tests` command + +Command to get list of tests in one of available formats (list or tree). + +```bash +> testplane list-tests --help + + Usage: list-tests [options] [paths...] + + Lists all tests info in one of available formats + + Options: + + -c, --config path to configuration file + -b, --browser list tests only in specified browser + -s, --set list tests only in the specified set + -r, --require require module + --grep list only tests matching the pattern + --ignore exclude paths from tests read + --silent [type] flag to disable events emitting while reading tests (default: false) + --output-file save results to specified file + --formatter [name] return tests in specified format (default: list) + -h, --help output usage information +``` + +For example, +``` +npx testplane list-tests --config ./config.js --browser firefox --grep name --formatter tree +``` + +**Note.** All CLI options override config values. + +#### Options + +##### `--formatter=` + +Return tests in specified format. Available formatters: `list` (default) and `tree`. +Let's see how the output of the tests in the yandex and chrome browsers will differ. For example, we have the following tests: + +```js +// example.hermione.js +it('test1', () => {}); + +describe('suite1', () => { + it('test2', () => {}); +}); +``` + +When using the `list` formatter (`npx testplane list-tests --formatter=list`), we get the following output: +```json +[ + { + "id": "5a105e8", + "titlePath": [ + "test1" + ], + "browserIds": [ + "yandex", + "chrome" + ], + "file": "tests/second.hermione.js" + }, + { + "id": "d2b3179", + "titlePath": [ + "suite", + "test2" + ], + "browserIds": [ + "yandex", + "chrome" + ], + "file": "tests/second.hermione.js" + } +] +``` + +Here, we got plain list of unique tests, where: +- `id` (`String`) - unique identifier of the test; +- `titlePath` (`String[]`) - full name of the test. Each element of the array is the title of a suite or test. To get the full title, you need just join `titlePath` with single whitespace; +- `browserIds` (`String[]`) - list of browsers in which the test will be launched; +- `file` (`String`) - path to the file relative to the working directory. + +When using the `tree` formatter (`npx testplane list-tests --formatter=tree`), we get the following output: +```json +[ + { + "id": "36749990", + "title": "suite", + "line": 3, + "column": 1, + "file": "example.hermione.js", + "suites": [], + "tests": [ + { + "id": "d2b3179", + "title": "test2", + "line": 4, + "column": 5, + "browserIds": [ + "yandex" + ] + } + ] + }, + { + "id": "5a105e8", + "title": "test1", + "line": 1, + "column": 1, + "browserIds": [ + "yandex" + ], + "file": "example.hermione.js" + } +] +``` + +Here, we got list of unique tests in the form of a tree structure (with parent suites), where `Suite` has following options: +- `id` (`String`) - unique identifier of the suite; +- `title` (`String`) - unique identifier of the suite; +- `line` (`Number`) - line on which the suite was called; +- `column` (`Number`) - column on which the suite was called; +- `file` (`String`, only in topmost suite) - path to the file relative to the working directory; +- `suites` (`Suite[]`, exists only in suite) - list of child suites; +- `tests` (`Test[]`) - list of tests; + +And `Test` has following options: +- `id` (`String`) - unique identifier of the test; +- `title` (`String`) - unique identifier of the test; +- `line` (`Number`) - line on which the test was called; +- `column` (`Number`) - column on which the test was called; +- `browserIds` (`String[]`) - list of browsers in which the test will be launched; +- `file` (`String`, only in tests without parent suites) - path to the file relative to the working directory. + +### Overriding settings + +All options can also be overridden via command-line flags or environment variables. Priorities are the following: + +* A command-line option has the highest priority. It overrides the environment variable and config file value. + +* An environment variable has second priority. It overrides the config file value. + +* A config file value has the lowest priority. + +* If there isn't a command-line option, environment variable or config file option specified, the default is used. + +To override a config setting with a CLI option, convert the full option path to `--kebab-case`. For example, if you want to run tests against a different base URL, call: + +``` +testplane path/to/mytest.js --base-url http://example.com +``` + +To change the number of sessions for Firefox (assuming you have a browser with the `firefox` id in the config): + +``` +testplane path/to/mytest.js --browsers-firefox-sessions-per-browser 7 +``` + +To override a setting with an environment variable, convert its full path to `snake_case` and add the `testplane_` prefix. The above examples can be rewritten to use environment variables instead of CLI options: + +``` +testplane_base_url=http://example.com testplane path/to/mytest.js +testplane_browsers_firefox_sessions_per_browser=7 testplane path/to/mytest.js +``` + + + ### Environment variables #### TESTPLANE_SKIP_BROWSERS diff --git a/docs/component-testing.md b/docs/component-testing.md index cdd2f279d..bccf60bfe 100644 --- a/docs/component-testing.md +++ b/docs/component-testing.md @@ -1,3 +1,17 @@ + + +### Contents + +- [Testplane Component Testing (experimental)](#testplane-component-testing-experimental) + - [Implementation options for component testing](#implementation-options-for-component-testing) + - [How to use it?](#how-to-use-it) + - [What additional features are supported?](#what-additional-features-are-supported) + - [Hot Module Replacement (HMR)](#hot-module-replacement-hmr) + - [Using the browser and expect instances directly in the browser DevTools](#using-the-browser-and-expect-instances-directly-in-the-browser-devtools) + - [Logs from the browser console in the terminal](#logs-from-the-browser-console-in-the-terminal) + + + ## Testplane Component Testing (experimental) Almost every modern web interfaces diff --git a/docs/config.md b/docs/config.md index 666ab5aec..b2e835419 100644 --- a/docs/config.md +++ b/docs/config.md @@ -73,7 +73,11 @@ - [List of useful plugins](#list-of-useful-plugins) - [prepareBrowser](#preparebrowser) - [prepareEnvironment](#prepareenvironment) +<<<<<<< HEAD - [lastFailed](#lastfailed) +======= +- [devServer](#devserver) +>>>>>>> a9b9a9ec (feat: implement new cli command to read tests) diff --git a/docs/programmatic-api.md b/docs/programmatic-api.md index e04db90a2..f49cd307a 100644 --- a/docs/programmatic-api.md +++ b/docs/programmatic-api.md @@ -255,6 +255,9 @@ await testplane.readTests(testPaths, options); * **ignore** (optional) `String|Glob|Array` - patterns to exclude paths from the test search. * **sets** (optional) `String[]`– Sets to run tests in. * **grep** (optional) `RegExp` – Pattern that defines which tests to run. + * **replMode** (optional) `{enabled: boolean; beforeTest: boolean; onFail: boolean;}` - [Test development mode using REPL](./cli.md#--repl). When reading the tests, it checks that only one test is running in one browser. + * **runnableOpts** (optional): + * **saveLocations** (optional) `Boolean` - flag to save `location` (`line` and `column`) to suites and tests. Allows to determine where the suite or test is declared in the file. ### isFailed diff --git a/docs/writing-tests.md b/docs/writing-tests.md index 4b5c6c1e6..3d8628334 100644 --- a/docs/writing-tests.md +++ b/docs/writing-tests.md @@ -8,6 +8,7 @@ - [Hooks](#hooks) - [Skip](#skip) - [Only](#only) +- [Also](#also) - [Config overriding](#config-overriding) - [testTimeout](#testtimeout) - [WebdriverIO extensions](#webdriverio-extensions) diff --git a/package-lock.json b/package-lock.json index 885e66311..caf4a1ed9 100644 --- a/package-lock.json +++ b/package-lock.json @@ -23,7 +23,6 @@ "debug": "2.6.9", "devtools": "8.21.0", "error-stack-parser": "2.1.4", - "escape-string-regexp": "1.0.5", "expect-webdriverio": "3.5.3", "fastq": "1.13.0", "fs-extra": "5.0.0", @@ -65,6 +64,7 @@ "@babel/preset-typescript": "7.24.1", "@commitlint/cli": "^19.0.3", "@commitlint/config-conventional": "^19.0.3", + "@cspotcode/source-map-support": "0.8.0", "@sinonjs/fake-timers": "10.3.0", "@swc/core": "1.3.40", "@types/babel__code-frame": "7.0.6", @@ -117,9 +117,13 @@ "node": ">= 18.0.0" }, "peerDependencies": { + "@cspotcode/source-map-support": ">=0.7.0", "ts-node": ">=10.5.0" }, "peerDependenciesMeta": { + "@cspotcode/source-map-support": { + "optional": true + }, "ts-node": { "optional": true } @@ -1218,9 +1222,9 @@ } }, "node_modules/@cspotcode/source-map-support": { - "version": "0.8.1", - "resolved": "https://registry.npmjs.org/@cspotcode/source-map-support/-/source-map-support-0.8.1.tgz", - "integrity": "sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==", + "version": "0.8.0", + "resolved": "https://registry.npmjs.org/@cspotcode/source-map-support/-/source-map-support-0.8.0.tgz", + "integrity": "sha512-pOQRG+w/T1KogjiuO4uqqa+dw/IIb8kDY0ctYfiJstWv7TOTmtuAkx8ZB4YgauDNn2huHR33oruOgi45VcatOg==", "dev": true, "dependencies": { "@jridgewell/trace-mapping": "0.3.9" @@ -18547,9 +18551,9 @@ } }, "@cspotcode/source-map-support": { - "version": "0.8.1", - "resolved": "https://registry.npmjs.org/@cspotcode/source-map-support/-/source-map-support-0.8.1.tgz", - "integrity": "sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==", + "version": "0.8.0", + "resolved": "https://registry.npmjs.org/@cspotcode/source-map-support/-/source-map-support-0.8.0.tgz", + "integrity": "sha512-pOQRG+w/T1KogjiuO4uqqa+dw/IIb8kDY0ctYfiJstWv7TOTmtuAkx8ZB4YgauDNn2huHR33oruOgi45VcatOg==", "dev": true, "requires": { "@jridgewell/trace-mapping": "0.3.9" diff --git a/package.json b/package.json index c3af0114f..fded2e811 100644 --- a/package.json +++ b/package.json @@ -65,7 +65,6 @@ "debug": "2.6.9", "devtools": "8.21.0", "error-stack-parser": "2.1.4", - "escape-string-regexp": "1.0.5", "expect-webdriverio": "3.5.3", "fastq": "1.13.0", "fs-extra": "5.0.0", @@ -103,6 +102,7 @@ "@babel/preset-typescript": "7.24.1", "@commitlint/cli": "^19.0.3", "@commitlint/config-conventional": "^19.0.3", + "@cspotcode/source-map-support": "0.8.0", "@sinonjs/fake-timers": "10.3.0", "@swc/core": "1.3.40", "@types/babel__code-frame": "7.0.6", @@ -152,11 +152,15 @@ "uglifyify": "3.0.4" }, "peerDependencies": { + "@cspotcode/source-map-support": ">=0.7.0", "ts-node": ">=10.5.0" }, "peerDependenciesMeta": { "ts-node": { "optional": true + }, + "@cspotcode/source-map-support": { + "optional": true } } } diff --git a/src/cli/commands/list-tests/constants.ts b/src/cli/commands/list-tests/constants.ts new file mode 100644 index 000000000..5cb0dd80b --- /dev/null +++ b/src/cli/commands/list-tests/constants.ts @@ -0,0 +1,6 @@ +export const Formatters = { + LIST: "list", + TREE: "tree", +} as const; + +export const AVAILABLE_FORMATTERS = Object.values(Formatters); diff --git a/src/cli/commands/list-tests/formatters/list.ts b/src/cli/commands/list-tests/formatters/list.ts new file mode 100644 index 000000000..8d96ab411 --- /dev/null +++ b/src/cli/commands/list-tests/formatters/list.ts @@ -0,0 +1,38 @@ +import path from "node:path"; +import type { TestCollection, TestDisabled } from "../../../../test-collection"; + +type TestInfo = { + id: string; + titlePath: string[]; + file: string; + browserIds: string[]; +}; + +export const format = (testCollection: TestCollection): TestInfo[] => { + const allTestsById = new Map(); + + testCollection.eachTest((test, browserId) => { + if ((test as TestDisabled).disabled) { + return; + } + + if (allTestsById.has(test.id)) { + const foundTest = allTestsById.get(test.id)!; + + if (!foundTest.browserIds.includes(browserId)) { + foundTest.browserIds.push(browserId); + } + + return; + } + + allTestsById.set(test.id, { + id: test.id, + titlePath: test.titlePath(), + browserIds: [browserId], + file: path.relative(process.cwd(), test.file as string), + }); + }); + + return [...allTestsById.values()]; +}; diff --git a/src/cli/commands/list-tests/formatters/tree.ts b/src/cli/commands/list-tests/formatters/tree.ts new file mode 100644 index 000000000..3520af689 --- /dev/null +++ b/src/cli/commands/list-tests/formatters/tree.ts @@ -0,0 +1,129 @@ +import path from "node:path"; + +import type { TestCollection, TestDisabled } from "../../../../test-collection"; +import type { Suite, Test } from "../../../../types"; + +type TreeSuite = { + id: string; + title: string; + line: number; + column: number; + suites: TreeSuite[]; + // eslint-disable-next-line no-use-before-define + tests: TreeTest[]; +}; + +type TreeTest = Omit & { + browserIds: string[]; +}; + +type MainTreeRunnable = (TreeSuite | TreeTest) & { + file: string; +}; + +export const format = (testCollection: TestCollection): MainTreeRunnable[] => { + const allSuitesById = new Map(); + const allTestsById = new Map(); + + testCollection.eachTest((test, browserId) => { + if ((test as TestDisabled).disabled) { + return; + } + + if (allTestsById.has(test.id)) { + const treeTest = allTestsById.get(test.id)!; + + if (!treeTest.browserIds.includes(browserId)) { + treeTest.browserIds.push(browserId); + } + + return; + } + + const treeTest = createTreeTest(test, browserId); + allTestsById.set(treeTest.id, treeTest); + + collectSuites(test.parent!, treeTest, allSuitesById); + }); + + return getTreeRunnables(allSuitesById, allTestsById); +}; + +function collectSuites(suite: Suite, child: TreeTest | TreeSuite, allSuitesById: Map): void { + if (allSuitesById.has(suite.id)) { + const treeSuite = allSuitesById.get(suite.id)!; + addChild(treeSuite, child); + + return; + } + + if (!suite.parent) { + return; + } + + const treeSuite = createTreeSuite(suite); + addChild(treeSuite, child); + + allSuitesById.set(treeSuite.id, treeSuite); + + collectSuites(suite.parent, treeSuite, allSuitesById); +} + +function isTreeTest(runnable: unknown): runnable is TreeTest { + return Boolean((runnable as TreeTest).browserIds); +} + +function createTreeTest(test: Test, browserId: string): TreeTest { + return { + id: test.id, + title: test.title, + ...test.location!, + browserIds: [browserId], + ...getMainRunanbleFields(test), + }; +} + +function createTreeSuite(suite: Suite): TreeSuite { + return { + id: suite.id, + title: suite.title, + ...suite.location!, + ...getMainRunanbleFields(suite), + suites: [], + tests: [], + }; +} + +function addChild(treeSuite: TreeSuite, child: TreeTest | TreeSuite): void { + const fieldName = isTreeTest(child) ? "tests" : "suites"; + const foundRunnable = treeSuite[fieldName].find(test => test.id === child.id); + + if (!foundRunnable) { + isTreeTest(child) ? addTest(treeSuite, child) : addSuite(treeSuite, child); + } +} + +function addTest(treeSuite: TreeSuite, child: TreeTest): void { + treeSuite.tests.push(child); +} + +function addSuite(treeSuite: TreeSuite, child: TreeSuite): void { + treeSuite.suites.push(child); +} + +function getMainRunanbleFields(runanble: Suite | Test): Partial> { + const isMain = runanble.parent && runanble.parent.root; + + return { + ...(isMain ? { file: path.relative(process.cwd(), runanble.file) } : {}), + }; +} + +function getTreeRunnables( + allSuitesById: Map, + allTestsById: Map, +): MainTreeRunnable[] { + return [...allSuitesById.values(), ...allTestsById.values()].filter( + suite => (suite as MainTreeRunnable).file, + ) as MainTreeRunnable[]; +} diff --git a/src/cli/commands/list-tests/index.ts b/src/cli/commands/list-tests/index.ts new file mode 100644 index 000000000..78da7ead7 --- /dev/null +++ b/src/cli/commands/list-tests/index.ts @@ -0,0 +1,72 @@ +import path from "node:path"; +import fs from "fs-extra"; +import { ValueOf } from "type-fest"; + +import { Testplane } from "../../../testplane"; +import { Formatters, AVAILABLE_FORMATTERS } from "./constants"; +import { CliCommands } from "../../constants"; +import { withCommonCliOptions, collectCliValues, handleRequires, type CommonCmdOpts } from "../../../utils/cli"; +import logger from "../../../utils/logger"; + +const { LIST_TESTS: commandName } = CliCommands; + +type ListTestsCmdOpts = { + ignore?: Array; + silent?: boolean; + outputFile?: string; + formatter: ValueOf; +}; + +export type ListTestsCmd = typeof commander & CommonCmdOpts; + +export const registerCmd = (cliTool: ListTestsCmd, testplane: Testplane): void => { + withCommonCliOptions({ cmd: cliTool.command(`${commandName}`), actionName: "list" }) + .description("Lists all tests info in one of available formats") + .option("--ignore ", "exclude paths from tests read", collectCliValues) + .option("--silent [type]", "flag to disable events emitting while reading tests", Boolean, false) + .option("--output-file ", "save results to specified file") + .option("--formatter [name]", "return tests in specified format", String, Formatters.LIST) + .arguments("[paths...]") + .action(async (paths: string[], options: ListTestsCmdOpts) => { + const { grep, browser: browsers, set: sets, require: requireModules } = cliTool; + const { ignore, silent, outputFile, formatter } = options; + + try { + validateFormatters(formatter); + handleRequires(requireModules); + + const testCollection = await testplane.readTests(paths, { + browsers, + sets, + grep, + ignore, + silent, + runnableOpts: { + saveLocations: formatter === Formatters.TREE, + }, + }); + + // eslint-disable-next-line @typescript-eslint/no-var-requires + const { format } = require(path.resolve(__dirname, "./formatters", formatter)); + const result = format(testCollection); + + if (outputFile) { + await fs.ensureDir(path.dirname(outputFile)); + await fs.writeJson(outputFile, result); + } else { + console.info(JSON.stringify(result)); + } + + process.exit(0); + } catch (err) { + logger.error((err as Error).stack || err); + process.exit(1); + } + }); +}; + +function validateFormatters(formatter: ValueOf): void { + if (!AVAILABLE_FORMATTERS.includes(formatter)) { + throw new Error(`"formatter" option must be one of: ${AVAILABLE_FORMATTERS.join(", ")}`); + } +} diff --git a/src/cli/constants.ts b/src/cli/constants.ts new file mode 100644 index 000000000..22fa8ec7a --- /dev/null +++ b/src/cli/constants.ts @@ -0,0 +1,3 @@ +export const CliCommands = { + LIST_TESTS: "list-tests", +} as const; diff --git a/src/cli/index.js b/src/cli/index.js index 47c6bac6e..529ad8401 100644 --- a/src/cli/index.js +++ b/src/cli/index.js @@ -1,16 +1,17 @@ "use strict"; +const path = require("node:path"); const _ = require("lodash"); const { Command } = require("@gemini-testing/commander"); -const escapeRe = require("escape-string-regexp"); const defaults = require("../config/defaults"); const info = require("./info"); const { Testplane } = require("../testplane"); const pkg = require("../../package.json"); const logger = require("../utils/logger"); -const { requireModule } = require("../utils/module"); const { shouldIgnoreUnhandledRejection } = require("../utils/errors"); +const { CliCommands } = require("./constants"); +const { withCommonCliOptions, collectCliValues, handleRequires } = require("../utils/cli"); let testplane; @@ -47,13 +48,10 @@ exports.run = (opts = {}) => { const configPath = preparseOption(program, "config"); testplane = Testplane.create(configPath); - program - .on("--help", () => logger.log(info.configOverriding(opts))) - .option("-b, --browser ", "run tests only in specified browser", collect) - .option("-s, --set ", "run tests only in the specified set", collect) - .option("-r, --require ", "require module", collect) - .option("--reporter ", "test reporters", collect) - .option("--grep ", "run only tests matching the pattern", compileGrep) + withCommonCliOptions({ cmd: program, actionName: "run" }) + .on("--help", () => console.log(`\n${info.configOverriding(opts)}`)) + .description("Run tests") + .option("--reporter ", "test reporters", collectCliValues) .option( "--update-refs", 'update screenshot references or gather them if they do not exist ("assertView" command)', @@ -112,15 +110,17 @@ exports.run = (opts = {}) => { } }); + for (const commandName of Object.values(CliCommands)) { + const { registerCmd } = require(path.resolve(__dirname, "./commands", commandName)); + + registerCmd(program, testplane); + } + testplane.extendCli(program); program.parse(process.argv); }; -function collect(newValue, array = []) { - return array.concat(newValue); -} - function preparseOption(program, option) { // do not display any help, do not exit const configFileParser = Object.create(program); @@ -130,18 +130,3 @@ function preparseOption(program, option) { configFileParser.parse(process.argv); return configFileParser[option]; } - -function compileGrep(grep) { - try { - return new RegExp(`(${grep})|(${escapeRe(grep)})`); - } catch (error) { - logger.warn(`Invalid regexp provided to grep, searching by its string representation. ${error}`); - return new RegExp(escapeRe(grep)); - } -} - -async function handleRequires(requires = []) { - for (const modulePath of requires) { - await requireModule(modulePath); - } -} diff --git a/src/test-collection.ts b/src/test-collection.ts index c26f41f78..b10837e8f 100644 --- a/src/test-collection.ts +++ b/src/test-collection.ts @@ -2,7 +2,7 @@ import _ from "lodash"; import type { Suite, RootSuite, Test } from "./types"; -type TestDisabled = Test & { disabled: true }; +export type TestDisabled = Test & { disabled: true }; type TestsCallback = (test: Test, browserId: string) => T; type SortTestsCallback = (test1: Test, test2: Test) => number; diff --git a/src/test-reader/index.js b/src/test-reader/index.js index 464b0a7a4..e1a801ac2 100644 --- a/src/test-reader/index.js +++ b/src/test-reader/index.js @@ -20,7 +20,7 @@ module.exports = class TestReader extends EventEmitter { } async read(options = {}) { - const { paths, browsers, ignore, sets, grep } = options; + const { paths, browsers, ignore, sets, grep, runnableOpts } = options; const { fileExtensions } = this.#config.system; const envSets = env.parseCommaSeparatedValue(["TESTPLANE_SETS", "HERMIONE_SETS"]).value; @@ -37,7 +37,7 @@ module.exports = class TestReader extends EventEmitter { const parser = new TestParser({ testRunEnv }); passthroughEvent(parser, this, [MasterEvents.BEFORE_FILE_READ, MasterEvents.AFTER_FILE_READ]); - await parser.loadFiles(setCollection.getAllFiles(), this.#config); + await parser.loadFiles(setCollection.getAllFiles(), { config: this.#config, runnableOpts }); const filesByBro = setCollection.groupByBrowser(); const testsByBro = _.mapValues(filesByBro, (files, browserId) => diff --git a/src/test-reader/mocha-reader/index.js b/src/test-reader/mocha-reader/index.js index 7ff5ea5ee..8de0ad9fe 100644 --- a/src/test-reader/mocha-reader/index.js +++ b/src/test-reader/mocha-reader/index.js @@ -1,17 +1,20 @@ "use strict"; +const _ = require("lodash"); +const Mocha = require("mocha"); + const { MochaEventBus } = require("./mocha-event-bus"); const { TreeBuilderDecorator } = require("./tree-builder-decorator"); const { TestReaderEvents } = require("../../events"); const { MasterEvents } = require("../../events"); -const Mocha = require("mocha"); +const { getMethodsByInterface } = require("./utils"); -async function readFiles(files, { esmDecorator, config, eventBus }) { +async function readFiles(files, { esmDecorator, config, eventBus, runnableOpts }) { const mocha = new Mocha(config); mocha.fullTrace(); initBuildContext(eventBus); - initEventListeners(mocha.suite, eventBus); + initEventListeners({ rootSuite: mocha.suite, outBus: eventBus, config, runnableOpts }); files.forEach(f => mocha.addFile(f)); await mocha.loadFilesAsync({ esmDecorator }); @@ -25,11 +28,12 @@ function initBuildContext(outBus) { }); } -function initEventListeners(rootSuite, outBus) { +function initEventListeners({ rootSuite, outBus, config, runnableOpts }) { const inBus = MochaEventBus.create(rootSuite); forbidSuiteHooks(inBus); passthroughFileEvents(inBus, outBus); + addLocationToRunnables(inBus, config, runnableOpts); registerTestObjects(inBus, outBus); inBus.emit(MochaEventBus.events.EVENT_SUITE_ADD_SUITE, rootSuite); @@ -95,6 +99,94 @@ function applyOnly(rootSuite, eventBus) { }); } +function addLocationToRunnables(inBus, config, runnableOpts) { + if (!runnableOpts || !runnableOpts.saveLocations) { + return; + } + + const sourceMapSupport = tryToRequireSourceMapSupport(); + const { suiteMethods, testMethods } = getMethodsByInterface(config.ui); + + inBus.on(MochaEventBus.events.EVENT_FILE_PRE_REQUIRE, ctx => { + [ + { + methods: suiteMethods, + eventName: MochaEventBus.events.EVENT_SUITE_ADD_SUITE, + }, + { + methods: testMethods, + eventName: MochaEventBus.events.EVENT_SUITE_ADD_TEST, + }, + ].forEach(({ methods, eventName }) => { + methods.forEach(methodName => { + ctx[methodName] = withLocation(ctx[methodName], { inBus, eventName, sourceMapSupport }); + + if (ctx[methodName]) { + ctx[methodName].only = withLocation(ctx[methodName].only, { inBus, eventName, sourceMapSupport }); + ctx[methodName].skip = withLocation(ctx[methodName].skip, { inBus, eventName, sourceMapSupport }); + } + + if (!config.ui || config.ui === "bdd") { + const pendingMethodName = `x${methodName}`; + ctx[pendingMethodName] = withLocation(ctx[pendingMethodName], { + inBus, + eventName, + sourceMapSupport, + }); + } + }); + }); + }); +} + +function withLocation(origFn, { inBus, eventName, sourceMapSupport }) { + if (!_.isFunction(origFn)) { + return origFn; + } + + const wrappedFn = (...args) => { + const origStackTraceLimit = Error.stackTraceLimit; + const origPrepareStackTrace = Error.prepareStackTrace; + + Error.stackTraceLimit = 2; + Error.prepareStackTrace = (error, stackFrames) => { + const frame = sourceMapSupport ? sourceMapSupport.wrapCallSite(stackFrames[1]) : stackFrames[1]; + + return { + line: frame.getLineNumber(), + column: frame.getColumnNumber(), + }; + }; + + const obj = {}; + Error.captureStackTrace(obj); + + const location = obj.stack; + Error.stackTraceLimit = origStackTraceLimit; + Error.prepareStackTrace = origPrepareStackTrace; + + inBus.once(eventName, runnable => { + if (!runnable.location) { + runnable.location = location; + } + }); + + return origFn(...args); + }; + + for (const key of Object.keys(origFn)) { + wrappedFn[key] = origFn[key]; + } + + return wrappedFn; +} + +function tryToRequireSourceMapSupport() { + try { + return require("@cspotcode/source-map-support"); + } catch {} // eslint-disable-line no-empty +} + module.exports = { readFiles, }; diff --git a/src/test-reader/mocha-reader/tree-builder-decorator.js b/src/test-reader/mocha-reader/tree-builder-decorator.js index 27cad49de..1cdd14b67 100644 --- a/src/test-reader/mocha-reader/tree-builder-decorator.js +++ b/src/test-reader/mocha-reader/tree-builder-decorator.js @@ -62,8 +62,8 @@ class TreeBuilderDecorator { } #mkTestObject(Constructor, mochaObject, customOpts) { - const { title, file } = mochaObject; - return Constructor.create({ title, file, ...customOpts }); + const { title, file, location } = mochaObject; + return Constructor.create({ title, file, location, ...customOpts }); } #applyConfig(testObject, mochaObject) { diff --git a/src/test-reader/mocha-reader/utils.js b/src/test-reader/mocha-reader/utils.js index eac1aadce..b9478c474 100644 --- a/src/test-reader/mocha-reader/utils.js +++ b/src/test-reader/mocha-reader/utils.js @@ -59,6 +59,18 @@ const computeFile = mochaSuite => { return null; }; +const getMethodsByInterface = (mochaInterface = "bdd") => { + switch (mochaInterface) { + case "tdd": + case "qunit": + return { suiteMethods: ["suite"], testMethods: ["test"] }; + case "bdd": + default: + return { suiteMethods: ["describe", "context"], testMethods: ["it", "specify"] }; + } +}; + module.exports = { computeFile, + getMethodsByInterface, }; diff --git a/src/test-reader/test-object/configurable-test-object.ts b/src/test-reader/test-object/configurable-test-object.ts index f72f92f24..ca6819996 100644 --- a/src/test-reader/test-object/configurable-test-object.ts +++ b/src/test-reader/test-object/configurable-test-object.ts @@ -2,7 +2,7 @@ import _ from "lodash"; import { TestObject } from "./test-object"; import type { ConfigurableTestObjectData, TestObjectData } from "./types"; -type ConfigurableTestObjectOpts = Pick & TestObjectData; +type ConfigurableTestObjectOpts = Pick & TestObjectData; type SkipData = { reason: string; @@ -11,10 +11,10 @@ type SkipData = { export class ConfigurableTestObject extends TestObject { #data: ConfigurableTestObjectData; - constructor({ title, file, id }: ConfigurableTestObjectOpts) { + constructor({ title, file, id, location }: ConfigurableTestObjectOpts) { super({ title }); - this.#data = { id, file } as ConfigurableTestObjectData; + this.#data = { id, file, location } as ConfigurableTestObjectData; } assign(src: this): this { @@ -109,4 +109,8 @@ export class ConfigurableTestObject extends TestObject { #getInheritedProperty(name: keyof ConfigurableTestObjectData, defaultValue: T): T { return name in this.#data ? (this.#data[name] as T) : (_.get(this.parent, name, defaultValue) as T); } + + get location(): ConfigurableTestObjectData["location"] { + return this.#data.location; + } } diff --git a/src/test-reader/test-object/suite.ts b/src/test-reader/test-object/suite.ts index 90a673c73..78b570bd9 100644 --- a/src/test-reader/test-object/suite.ts +++ b/src/test-reader/test-object/suite.ts @@ -4,7 +4,7 @@ import { Hook } from "./hook"; import { Test } from "./test"; import type { TestObjectData, ConfigurableTestObjectData, TestFunction, TestFunctionCtx } from "./types"; -type SuiteOpts = Pick & TestObjectData; +type SuiteOpts = Pick & TestObjectData; export class Suite extends ConfigurableTestObject { #suites: this[]; @@ -17,8 +17,8 @@ export class Suite extends ConfigurableTestObject { } // used inside test - constructor({ title, file, id }: SuiteOpts = {} as SuiteOpts) { - super({ title, file, id }); + constructor({ title, file, id, location }: SuiteOpts = {} as SuiteOpts) { + super({ title, file, id, location }); this.#suites = []; this.#tests = []; diff --git a/src/test-reader/test-object/test.ts b/src/test-reader/test-object/test.ts index 0e682297a..6641d4194 100644 --- a/src/test-reader/test-object/test.ts +++ b/src/test-reader/test-object/test.ts @@ -2,7 +2,7 @@ import { ConfigurableTestObject } from "./configurable-test-object"; import type { TestObjectData, TestFunction, TestFunctionCtx } from "./types"; type TestOpts = TestObjectData & - Pick & { + Pick & { fn: TestFunction; }; @@ -14,8 +14,8 @@ export class Test extends ConfigurableTestObject { return new this(opts); } - constructor({ title, file, id, fn }: TestOpts) { - super({ title, file, id }); + constructor({ title, file, id, location, fn }: TestOpts) { + super({ title, file, id, location }); this.fn = fn; } @@ -25,6 +25,7 @@ export class Test extends ConfigurableTestObject { title: this.title, file: this.file, id: this.id, + location: this.location, fn: this.fn, }).assign(this); } diff --git a/src/test-reader/test-object/types.ts b/src/test-reader/test-object/types.ts index a26a46863..a5d8b2140 100644 --- a/src/test-reader/test-object/types.ts +++ b/src/test-reader/test-object/types.ts @@ -6,6 +6,11 @@ export type TestObjectData = { title: string; }; +export type Location = { + line: number; + column: number; +}; + export type ConfigurableTestObjectData = { id: string; pending: boolean; @@ -16,6 +21,7 @@ export type ConfigurableTestObjectData = { silentSkip: boolean; browserId: string; browserVersion?: string; + location?: Location; }; export interface TestFunctionCtx { diff --git a/src/test-reader/test-parser.js b/src/test-reader/test-parser.js index 857253dcf..0f52f38a0 100644 --- a/src/test-reader/test-parser.js +++ b/src/test-reader/test-parser.js @@ -37,7 +37,7 @@ class TestParser extends EventEmitter { this.#buildInstructions = new InstructionsList(); } - async loadFiles(files, config) { + async loadFiles(files, { config, runnableOpts }) { const eventBus = new EventEmitter(); const { system: { ctx, mochaOpts }, @@ -71,7 +71,7 @@ class TestParser extends EventEmitter { const rand = Math.random(); const esmDecorator = f => f + `?rand=${rand}`; - await readFiles(files, { esmDecorator, config: mochaOpts, eventBus }); + await readFiles(files, { esmDecorator, config: mochaOpts, eventBus, runnableOpts }); if (config.lastFailed.only) { try { diff --git a/src/testplane.ts b/src/testplane.ts index 7fc00cf5a..afd359e61 100644 --- a/src/testplane.ts +++ b/src/testplane.ts @@ -44,10 +44,15 @@ export type FailedListItem = { fullTitle: string; }; +interface RunnableOpts { + saveLocations?: boolean; +} + interface ReadTestsOpts extends Pick { silent: boolean; ignore: string | string[]; failed: FailedListItem[]; + runnableOpts?: RunnableOpts; } export interface Testplane { @@ -152,7 +157,7 @@ export class Testplane extends BaseTestplane { async readTests( testPaths: string[], - { browsers, sets, grep, silent, ignore, replMode }: Partial = {}, + { browsers, sets, grep, silent, ignore, replMode, runnableOpts }: Partial = {}, ): Promise { const testReader = TestReader.create(this._config); @@ -165,7 +170,7 @@ export class Testplane extends BaseTestplane { ]); } - const specs = await testReader.read({ paths: testPaths, browsers, ignore, sets, grep, replMode }); + const specs = await testReader.read({ paths: testPaths, browsers, ignore, sets, grep, replMode, runnableOpts }); const collection = TestCollection.create(specs); collection.getBrowsers().forEach(bro => { diff --git a/src/utils/cli.ts b/src/utils/cli.ts new file mode 100644 index 000000000..e01ad7a69 --- /dev/null +++ b/src/utils/cli.ts @@ -0,0 +1,51 @@ +import type { Command } from "@gemini-testing/commander"; +import logger from "./logger"; +import { requireModule } from "./module"; + +// used from https://github.com/sindresorhus/escape-string-regexp/blob/main/index.js +const escapeRe = (str: string): string => { + // Escape characters with special meaning either inside or outside character sets. + // Use a simple backslash escape when it’s always valid, and a `\xnn` escape when the simpler form would be disallowed by Unicode patterns’ stricter grammar. + return str.replace(/[|\\{}()[\]^$+*?.]/g, "\\$&").replace(/-/g, "\\x2d"); +}; + +export const collectCliValues = (newValue: unknown, array = [] as unknown[]): unknown[] => { + return array.concat(newValue); +}; + +export const compileGrep = (grep: string): RegExp => { + try { + return new RegExp(`(${grep})|(${escapeRe(grep)})`); + } catch (error) { + logger.warn(`Invalid regexp provided to grep, searching by its string representation. ${error}`); + return new RegExp(escapeRe(grep)); + } +}; + +export const handleRequires = async (requires: string[] = []): Promise => { + for (const modulePath of requires) { + await requireModule(modulePath); + } +}; + +export type CommonCmdOpts = { + config?: string; + browser?: Array; + set?: Array; + require?: Array; + grep?: RegExp; +}; + +export const withCommonCliOptions = ({ cmd, actionName = "run" }: { cmd: Command; actionName: string }): Command => { + const isMainCmd = ["testplane", "hermione"].includes(cmd.name()); + + if (!isMainCmd) { + cmd.option("-c, --config ", "path to configuration file"); + } + + return cmd + .option("-b, --browser ", `${actionName} tests only in specified browser`, collectCliValues) + .option("-s, --set ", `${actionName} tests only in the specified set`, collectCliValues) + .option("-r, --require ", "require module", collectCliValues) + .option("--grep ", `${actionName} only tests matching the pattern`, compileGrep); +}; diff --git a/src/utils/typescript.ts b/src/utils/typescript.ts index 78f49d34f..fb04b020b 100644 --- a/src/utils/typescript.ts +++ b/src/utils/typescript.ts @@ -28,7 +28,6 @@ export const tryToRegisterTsNode = (): void => { transpileOnly: JSON.parse(transpileOnlyRaw), swc: JSON.parse(swcRaw), compilerOptions: { - allowJs: true, module: "nodenext", moduleResolution: "nodenext", }, diff --git a/test/src/cli/commands/list-tests/formatters/list.ts b/test/src/cli/commands/list-tests/formatters/list.ts new file mode 100644 index 000000000..ddc1a6d25 --- /dev/null +++ b/test/src/cli/commands/list-tests/formatters/list.ts @@ -0,0 +1,71 @@ +import path from "node:path"; +import _ from "lodash"; + +import { format } from "../../../../../../src/cli/commands/list-tests/formatters/list"; +import { Test, Suite } from "../../../../../../src/test-reader/test-object"; +import { TestCollection } from "../../../../../../src/test-collection"; + +type TestOpts = { + id: string; + title: string; + file: string; + parent: Suite; + disabled: boolean; +}; + +describe("cli/commands/list-tests/formatters/list", () => { + const mkTest_ = (opts: Partial = { title: "default-title" }): Test => { + const paramNames = ["id", "title", "file"]; + + const test = new Test(_.pick(opts, paramNames) as any); + for (const [key, value] of _.entries(_.omit(opts, paramNames))) { + _.set(test, key, value); + } + + return test; + }; + + it("should return empty array if all tests are disabled", () => { + const collection = TestCollection.create({ + bro1: [mkTest_({ disabled: true })], + bro2: [mkTest_({ disabled: true })], + }); + + const result = format(collection); + + assert.deepEqual(result, []); + }); + + it("should return tests with correct fields", () => { + const root1 = new Suite({ title: "root1" } as any); + const root2 = new Suite({ title: "root2" } as any); + + const file1 = path.resolve(process.cwd(), "./folder/file1.ts"); + const file2 = path.resolve(process.cwd(), "./folder/file2.ts"); + + const test1 = mkTest_({ id: "0", title: "test1", file: file1, parent: root1 }); + const test2 = mkTest_({ id: "1", title: "test2", file: file2, parent: root2 }); + + const collection = TestCollection.create({ + bro1: [test1], + bro2: [test1, test2], + }); + + const result = format(collection); + + assert.deepEqual(result, [ + { + id: "0", + titlePath: ["root1", "test1"], + browserIds: ["bro1", "bro2"], + file: "folder/file1.ts", + }, + { + id: "1", + titlePath: ["root2", "test2"], + browserIds: ["bro2"], + file: "folder/file2.ts", + }, + ]); + }); +}); diff --git a/test/src/cli/commands/list-tests/formatters/tree.ts b/test/src/cli/commands/list-tests/formatters/tree.ts new file mode 100644 index 000000000..02a168df8 --- /dev/null +++ b/test/src/cli/commands/list-tests/formatters/tree.ts @@ -0,0 +1,296 @@ +import path from "node:path"; +import _ from "lodash"; + +import { format } from "../../../../../../src/cli/commands/list-tests/formatters/tree"; +import { Test, Suite } from "../../../../../../src/test-reader/test-object"; +import { TestCollection } from "../../../../../../src/test-collection"; + +type SuiteOpts = { + id: string; + title: string; + file: string; + parent: Suite; + root: boolean; + location: { + line: number; + column: number; + }; +}; + +type TestOpts = Omit & { + disabled: boolean; +}; + +describe("cli/commands/list-tests/formatters/tree", () => { + const mkSuite_ = (opts: Partial = { title: "default-suite-title" }): Suite => { + const paramNames = ["id", "title", "file", "location"]; + + const suite = new Suite(_.pick(opts, paramNames) as any); + for (const [key, value] of _.entries(_.omit(opts, paramNames))) { + _.set(suite, key, value); + } + + return suite; + }; + + const mkTest_ = (opts: Partial = { title: "default-test-title" }): Test => { + const paramNames = ["id", "title", "file", "location"]; + + const test = new Test(_.pick(opts, paramNames) as any); + for (const [key, value] of _.entries(_.omit(opts, paramNames))) { + _.set(test, key, value); + } + + return test; + }; + + it("should return empty array if all tests are disabled", () => { + const collection = TestCollection.create({ + bro1: [mkTest_({ disabled: true })], + bro2: [mkTest_({ disabled: true })], + }); + + const result = format(collection); + + assert.deepEqual(result, []); + }); + + describe("should return main test", () => { + it("in one browser", () => { + const rootSuite = mkSuite_({ id: "0", root: true }); + const testOpts: Partial = { + id: "1", + title: "test", + file: path.resolve(process.cwd(), "./folder/file.ts"), + location: { + line: 1, + column: 1, + }, + parent: rootSuite, + }; + + const collection = TestCollection.create({ + bro1: [mkTest_(testOpts)], + }); + + const result = format(collection); + + assert.deepEqual(result, [ + { + id: "1", + title: "test", + file: "folder/file.ts", + line: 1, + column: 1, + browserIds: ["bro1"], + }, + ]); + }); + + it("in few browsers", () => { + const rootSuite = mkSuite_({ id: "0", root: true }); + const testOpts: Partial = { + id: "1", + title: "test", + file: path.resolve(process.cwd(), "./folder/file.ts"), + location: { + line: 1, + column: 1, + }, + parent: rootSuite, + }; + + const collection = TestCollection.create({ + bro1: [mkTest_(testOpts)], + bro2: [mkTest_(testOpts)], + }); + + const result = format(collection); + + assert.deepEqual(result, [ + { + id: "1", + title: "test", + file: "folder/file.ts", + line: 1, + column: 1, + browserIds: ["bro1", "bro2"], + }, + ]); + }); + }); + + it("should return main tests with equal titles", () => { + const rootSuite = mkSuite_({ id: "0", root: true }); + const commonTestOpts: Partial = { + id: "1", + title: "test", + location: { + line: 1, + column: 1, + }, + parent: rootSuite, + }; + const testOpts1: Partial = { + ...commonTestOpts, + id: "1", + file: path.resolve(process.cwd(), "./folder/file1.ts"), + }; + const testOpts2: Partial = { + ...commonTestOpts, + id: "2", + file: path.resolve(process.cwd(), "./folder/file2.ts"), + }; + + const collection = TestCollection.create({ + bro1: [mkTest_(testOpts1)], + bro2: [mkTest_(testOpts2)], + }); + + const result = format(collection); + + assert.deepEqual(result, [ + { + id: "1", + title: "test", + file: "folder/file1.ts", + line: 1, + column: 1, + browserIds: ["bro1"], + }, + { + id: "2", + title: "test", + file: "folder/file2.ts", + line: 1, + column: 1, + browserIds: ["bro2"], + }, + ]); + }); + + it("should return main suite with one test", () => { + const rootSuite = mkSuite_({ id: "0", root: true }); + const suiteOpts: Partial = { + id: "1", + title: "suite", + file: path.resolve(process.cwd(), "./folder/file.ts"), + location: { + line: 1, + column: 1, + }, + parent: rootSuite, + }; + const suite = mkSuite_(suiteOpts); + + const testOpts: Partial = { + id: "2", + title: "test", + file: path.resolve(process.cwd(), "./folder/file.ts"), + location: { + line: 2, + column: 5, + }, + parent: suite, + }; + + const collection = TestCollection.create({ + bro1: [mkTest_(testOpts)], + }); + + const result = format(collection); + + assert.deepEqual(result, [ + { + id: "1", + title: "suite", + file: "folder/file.ts", + line: 1, + column: 1, + suites: [], + tests: [ + { + id: "2", + title: "test", + line: 2, + column: 5, + browserIds: ["bro1"], + }, + ], + }, + ]); + }); + + it("should return main suite with child suite", () => { + const rootSuite = mkSuite_({ id: "0", root: true }); + const mainSuiteOpts: Partial = { + id: "1", + title: "suite1", + file: path.resolve(process.cwd(), "./folder/file.ts"), + location: { + line: 1, + column: 1, + }, + parent: rootSuite, + }; + const mainSuite = mkSuite_(mainSuiteOpts); + + const childSuiteOpts: Partial = { + id: "2", + title: "suite2", + file: path.resolve(process.cwd(), "./folder/file.ts"), + location: { + line: 2, + column: 5, + }, + parent: mainSuite, + }; + const childSuite = mkSuite_(childSuiteOpts); + + const testOpts: Partial = { + id: "3", + title: "test", + file: path.resolve(process.cwd(), "./folder/file.ts"), + location: { + line: 3, + column: 9, + }, + parent: childSuite, + }; + + const collection = TestCollection.create({ + bro1: [mkTest_(testOpts)], + }); + + const result = format(collection); + + assert.deepEqual(result, [ + { + id: "1", + title: "suite1", + file: "folder/file.ts", + line: 1, + column: 1, + suites: [ + { + id: "2", + title: "suite2", + line: 2, + column: 5, + suites: [], + tests: [ + { + id: "3", + title: "test", + line: 3, + column: 9, + browserIds: ["bro1"], + }, + ], + }, + ], + tests: [], + }, + ]); + }); +}); diff --git a/test/src/cli/commands/list-tests/index.ts b/test/src/cli/commands/list-tests/index.ts new file mode 100644 index 000000000..59493fcdf --- /dev/null +++ b/test/src/cli/commands/list-tests/index.ts @@ -0,0 +1,192 @@ +import path from "node:path"; +import { Command } from "@gemini-testing/commander"; +import fs from "fs-extra"; +import sinon, { SinonStub } from "sinon"; +import proxyquire from "proxyquire"; + +import { AVAILABLE_FORMATTERS, Formatters } from "../../../../../src/cli/commands/list-tests/constants"; +import logger from "../../../../../src/utils/logger"; +import { Testplane } from "../../../../../src/testplane"; +import testplaneCli from "../../../../../src/cli"; +import { TestCollection } from "../../../../../src/test-collection"; + +describe("cli/commands/list-tests", () => { + const sandbox = sinon.createSandbox(); + + const listTests_ = async (argv: string = "", cli: { run: VoidFunction } = testplaneCli): Promise => { + process.argv = ["foo/bar/node", "foo/bar/script", "list-tests", ...argv.split(" ")].filter(Boolean); + cli.run(); + + await (Command.prototype.action as SinonStub).lastCall.returnValue; + }; + + beforeEach(() => { + sandbox.stub(Testplane, "create").returns(Object.create(Testplane.prototype)); + sandbox.stub(Testplane.prototype, "readTests").resolves(TestCollection.create({})); + + sandbox.stub(fs, "ensureDir").resolves(); + sandbox.stub(fs, "writeJson").resolves(); + + sandbox.stub(logger, "error"); + sandbox.stub(console, "info"); + sandbox.stub(process, "exit"); + + sandbox.spy(Command.prototype, "action"); + }); + + afterEach(() => sandbox.restore()); + + it("should throw error if passed formatter is not supported", async () => { + try { + await listTests_("--formatter foo"); + } catch (e) { + assert.calledWithMatch( + logger.error as SinonStub, + `"formatter" option must be one of: ${AVAILABLE_FORMATTERS.join(", ")}`, + ); + } + }); + + it("should exit with code 0", async () => { + await listTests_(); + + assert.calledWith(process.exit as unknown as SinonStub, 0); + }); + + it("should exit with code 1 if read tests failed", async () => { + (Testplane.prototype.readTests as SinonStub).rejects(new Error("o.O")); + + await listTests_(); + + assert.calledWith(process.exit as unknown as SinonStub, 1); + }); + + describe("read tests", () => { + it("should use paths from cli", async () => { + await listTests_("first.testplane.js second.testplane.js"); + + assert.calledWith(Testplane.prototype.readTests as SinonStub, [ + "first.testplane.js", + "second.testplane.js", + ]); + }); + + it("should use browsers from cli", async () => { + await listTests_("--browser first --browser second"); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { + browsers: ["first", "second"], + }); + }); + + it("should use sets from cli", async () => { + await listTests_("--set first --set second"); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { + sets: ["first", "second"], + }); + }); + + it("should use grep from cli", async () => { + await listTests_("--grep some"); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { + grep: sinon.match.instanceOf(RegExp), + }); + }); + + it("should use ignore paths from cli", async () => { + await listTests_("--ignore first --ignore second"); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { + ignore: ["first", "second"], + }); + }); + + describe("silent", () => { + it("should be disabled by default", async () => { + await listTests_(); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { silent: false }); + }); + + it("should use from cli", async () => { + await listTests_("--silent"); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { silent: true }); + }); + }); + + describe("runnableOpts", () => { + it("should not save runnale locations by default", async () => { + await listTests_(); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { + runnableOpts: { + saveLocations: false, + }, + }); + }); + + it(`should save runnale locations if "${Formatters.TREE}" formatter is used`, async () => { + await listTests_(`--formatter ${Formatters.TREE}`); + + assert.calledWithMatch(Testplane.prototype.readTests as SinonStub, sinon.match.any, { + runnableOpts: { + saveLocations: true, + }, + }); + }); + }); + }); + + [Formatters.LIST, Formatters.TREE].forEach(formatterName => { + let formatterStub: SinonStub; + let cli: { run: VoidFunction }; + + beforeEach(() => { + formatterStub = sandbox.stub().returns({}); + + cli = proxyquire("../../../../../src/cli", { + [path.resolve(process.cwd(), "src/cli/commands/list-tests")]: proxyquire( + "../../../../../src/cli/commands/list-tests", + { + [path.resolve(process.cwd(), `src/cli/commands/list-tests/formatters/${formatterName}`)]: { + format: formatterStub, + }, + }, + ), + }); + }); + + describe(`${formatterName} formatter`, () => { + it("should call with test collection", async () => { + const testCollection = TestCollection.create({}); + (Testplane.prototype.readTests as SinonStub).resolves(testCollection); + + await listTests_(`--formatter ${formatterName}`, cli); + + assert.calledOnceWith(formatterStub, testCollection); + }); + + it("should send result to stdout", async () => { + const formatterResult = {}; + formatterStub.returns(formatterResult); + + await listTests_(`--formatter ${formatterName}`, cli); + + assert.calledOnceWith(console.info, JSON.stringify(formatterResult)); + }); + + it("should save result to output file", async () => { + const formatterResult = {}; + formatterStub.returns(formatterResult); + + await listTests_(`--formatter ${formatterName} --output-file ./folder/file.json`, cli); + + (fs.ensureDir as SinonStub).calledOnceWith("./folder"); + (fs.writeJson as SinonStub).calledOnceWith("./folder/file.json", formatterResult); + }); + }); + }); +}); diff --git a/test/src/cli/index.js b/test/src/cli/index.js index 97cdec943..76d874d40 100644 --- a/test/src/cli/index.js +++ b/test/src/cli/index.js @@ -7,6 +7,7 @@ const info = require("src/cli/info"); const defaults = require("src/config/defaults"); const { Testplane } = require("src/testplane"); const logger = require("src/utils/logger"); +const { collectCliValues, withCommonCliOptions } = require("src/utils/cli"); const any = sinon.match.any; @@ -40,10 +41,12 @@ describe("cli", () => { describe("config overriding", () => { it('should show information about config overriding on "--help"', async () => { + sandbox.stub(console, "log"); + await run_("--help"); - assert.calledOnce(logger.log); - assert.calledWith(logger.log, info.configOverriding()); + assert.calledOnce(console.log); + assert.calledWith(console.log, `\n${info.configOverriding()}`); }); it("should show information about testplane by default", async () => { @@ -68,14 +71,14 @@ describe("cli", () => { }); it('should require modules specified in "require" option', async () => { - const requireModule = sandbox.stub(); + const handleRequires = sandbox.stub(); const stubTestplaneCli = proxyquire("src/cli", { - "../utils/module": { requireModule }, + "../utils/cli": { handleRequires, withCommonCliOptions }, }); await run_("--require foo", stubTestplaneCli); - assert.calledOnceWith(requireModule, "foo"); + assert.calledOnceWith(handleRequires, ["foo"]); }); it("should create Testplane without config by default", async () => { @@ -172,7 +175,7 @@ describe("cli", () => { it("should use require modules from cli", async () => { const stubTestplaneCli = proxyquire("src/cli", { - "../utils/module": { requireModule: sandbox.stub() }, + "../utils/cli": { handleRequires: sandbox.stub(), collectCliValues, withCommonCliOptions }, }); await run_("--require foo", stubTestplaneCli); diff --git a/test/src/test-reader/index.js b/test/src/test-reader/index.js index 53415487b..720cf54c8 100644 --- a/test/src/test-reader/index.js +++ b/test/src/test-reader/index.js @@ -155,10 +155,11 @@ describe("test-reader", () => { const config = makeConfigStub(); const files = ["file1.js", "file2.js"]; SetCollection.prototype.getAllFiles.returns(files); + const runnableOpts = { saveLocations: true }; - await readTests_({ config }); + await readTests_({ config, opts: { runnableOpts } }); - assert.calledOnceWith(TestParser.prototype.loadFiles, files, config); + assert.calledOnceWith(TestParser.prototype.loadFiles, files, { config, runnableOpts }); }); it("should load files before parsing", async () => { diff --git a/test/src/test-reader/mocha-reader/index.js b/test/src/test-reader/mocha-reader/index.js index b3ab8a83a..6267f257a 100644 --- a/test/src/test-reader/mocha-reader/index.js +++ b/test/src/test-reader/mocha-reader/index.js @@ -1,5 +1,6 @@ "use strict"; +const _ = require("lodash"); const { MochaEventBus } = require("src/test-reader/mocha-reader/mocha-event-bus"); const { TreeBuilderDecorator } = require("src/test-reader/mocha-reader/tree-builder-decorator"); const { TreeBuilder } = require("src/test-reader/tree-builder"); @@ -14,6 +15,8 @@ describe("test-reader/mocha-reader", () => { const sandbox = sinon.createSandbox(); let MochaConstructorStub; + let SourceMapSupportStub; + let getMethodsByInterfaceStub; let readFiles; const mkMochaSuiteStub_ = () => { @@ -36,8 +39,18 @@ describe("test-reader/mocha-reader", () => { MochaConstructorStub = sinon.stub().returns(mkMochaStub_()); MochaConstructorStub.Suite = Mocha.Suite; + SourceMapSupportStub = { + wrapCallSite: sinon.stub().returns({ + getLineNumber: () => 1, + getColumnNumber: () => 1, + }), + }; + getMethodsByInterfaceStub = sinon.stub().returns({ suiteMethods: [], testMethods: [] }); + readFiles = proxyquire("src/test-reader/mocha-reader", { mocha: MochaConstructorStub, + "@cspotcode/source-map-support": SourceMapSupportStub, + "./utils": { getMethodsByInterface: getMethodsByInterfaceStub }, }).readFiles; sandbox.stub(MochaEventBus, "create").returns(Object.create(MochaEventBus.prototype)); @@ -206,6 +219,135 @@ describe("test-reader/mocha-reader", () => { }); }); + describe("add locations to runnables", () => { + const emitAddRunnable_ = (runnable, event) => { + MochaEventBus.create.lastCall.returnValue.emit(MochaEventBus.events[event], runnable); + }; + + it("should do nothing if 'saveLocations' is not enabled", async () => { + const globalCtx = { + describe: () => {}, + }; + + Mocha.prototype.loadFilesAsync.callsFake(() => { + MochaEventBus.create.lastCall.returnValue.emit( + MochaEventBus.events.EVENT_FILE_PRE_REQUIRE, + globalCtx, + ); + }); + + await readFiles_({ runnableOpts: { saveLocations: false } }); + globalCtx.describe(); + + assert.notCalled(SourceMapSupportStub.wrapCallSite); + }); + + it("should not throw if source-map-support is not installed", async () => { + readFiles = proxyquire("src/test-reader/mocha-reader", { + "@cspotcode/source-map-support": null, + }).readFiles; + + const globalCtx = { describe: _.noop }; + + Mocha.prototype.loadFilesAsync.callsFake(() => { + MochaEventBus.create.lastCall.returnValue.emit( + MochaEventBus.events.EVENT_FILE_PRE_REQUIRE, + globalCtx, + ); + }); + + await readFiles_({ runnableOpts: { saveLocations: true } }); + + assert.doesNotThrow(() => globalCtx.describe()); + }); + + ["describe", "describe.only", "describe.skip", "xdescribe"].forEach(methodName => { + it(`should add location to suite using "${methodName}"`, async () => { + getMethodsByInterfaceStub.withArgs("bdd").returns({ suiteMethods: ["describe"], testMethods: [] }); + const suite = {}; + const globalCtx = _.set({}, methodName, () => emitAddRunnable_(suite, "EVENT_SUITE_ADD_SUITE")); + + Mocha.prototype.loadFilesAsync.callsFake(() => { + MochaEventBus.create.lastCall.returnValue.emit( + MochaEventBus.events.EVENT_FILE_PRE_REQUIRE, + globalCtx, + ); + }); + + SourceMapSupportStub.wrapCallSite.returns({ + getLineNumber: () => 100, + getColumnNumber: () => 500, + }); + + await readFiles_({ config: { ui: "bdd" }, runnableOpts: { saveLocations: true } }); + _.get(globalCtx, methodName)(); + + assert.deepEqual(suite, { location: { line: 100, column: 500 } }); + }); + }); + + ["it", "it.only", "it.skip", "xit"].forEach(methodName => { + it(`should add location to test using "${methodName}"`, async () => { + getMethodsByInterfaceStub.withArgs("bdd").returns({ suiteMethods: [], testMethods: ["it"] }); + const test = {}; + const globalCtx = _.set({}, methodName, () => emitAddRunnable_(test, "EVENT_SUITE_ADD_TEST")); + + Mocha.prototype.loadFilesAsync.callsFake(() => { + MochaEventBus.create.lastCall.returnValue.emit( + MochaEventBus.events.EVENT_FILE_PRE_REQUIRE, + globalCtx, + ); + }); + + SourceMapSupportStub.wrapCallSite.returns({ + getLineNumber: () => 500, + getColumnNumber: () => 100, + }); + + await readFiles_({ config: { ui: "bdd" }, runnableOpts: { saveLocations: true } }); + _.get(globalCtx, methodName)(); + + assert.deepEqual(test, { location: { line: 500, column: 100 } }); + }); + }); + + it(`should add location to each runnable`, async () => { + getMethodsByInterfaceStub.withArgs("bdd").returns({ suiteMethods: ["describe"], testMethods: ["it"] }); + const suite = {}; + const test = {}; + const globalCtx = { + describe: () => emitAddRunnable_(suite, "EVENT_SUITE_ADD_SUITE"), + it: () => emitAddRunnable_(test, "EVENT_SUITE_ADD_TEST"), + }; + + Mocha.prototype.loadFilesAsync.callsFake(() => { + MochaEventBus.create.lastCall.returnValue.emit( + MochaEventBus.events.EVENT_FILE_PRE_REQUIRE, + globalCtx, + ); + }); + + SourceMapSupportStub.wrapCallSite + .onFirstCall() + .returns({ + getLineNumber: () => 111, + getColumnNumber: () => 222, + }) + .onSecondCall() + .returns({ + getLineNumber: () => 333, + getColumnNumber: () => 444, + }); + + await readFiles_({ config: { ui: "bdd" }, runnableOpts: { saveLocations: true } }); + globalCtx.describe(); + globalCtx.it(); + + assert.deepEqual(suite, { location: { line: 111, column: 222 } }); + assert.deepEqual(test, { location: { line: 333, column: 444 } }); + }); + }); + describe("test objects", () => { [ ["EVENT_SUITE_ADD_SUITE", "addSuite"], diff --git a/test/src/test-reader/test-parser.js b/test/src/test-reader/test-parser.js index f901190e2..72be40843 100644 --- a/test/src/test-reader/test-parser.js +++ b/test/src/test-reader/test-parser.js @@ -53,11 +53,11 @@ describe("test-reader/test-parser", () => { }); describe("loadFiles", () => { - const loadFiles_ = async ({ parser, files, config } = {}) => { + const loadFiles_ = async ({ parser, files, config, runnableOpts } = {}) => { parser = parser || new TestParser(); config = config || makeConfigStub(); - return parser.loadFiles(files || [], config); + return parser.loadFiles(files || [], { config, runnableOpts }); }; describe("globals", () => { @@ -413,6 +413,14 @@ describe("test-reader/test-parser", () => { assert.calledWithMatch(readFiles, sinon.match.any, { eventBus: sinon.match.instanceOf(EventEmitter) }); }); + it("should pass runnable options to reader", async () => { + const runnableOpts = { saveLocations: true }; + + await loadFiles_({ runnableOpts }); + + assert.calledWithMatch(readFiles, sinon.match.any, { runnableOpts }); + }); + describe("esm decorator", () => { it("should be passed to mocha reader", async () => { await loadFiles_(); @@ -546,7 +554,7 @@ describe("test-reader/test-parser", () => { }); const parser = new TestParser(); - await parser.loadFiles([], loadFilesConfig); + await parser.loadFiles([], { config: loadFilesConfig }); return parser.parse(files || [], { browserId, config, grep }); }; diff --git a/test/src/testplane.js b/test/src/testplane.js index 8c4f826fd..3b6989e5a 100644 --- a/test/src/testplane.js +++ b/test/src/testplane.js @@ -639,6 +639,9 @@ describe("testplane", () => { sets: ["s1", "s2"], grep: "grep", replMode: { enabled: false }, + runnableOpts: { + saveLocations: true, + }, }); assert.calledOnceWith(TestReader.prototype.read, { @@ -648,6 +651,9 @@ describe("testplane", () => { sets: ["s1", "s2"], grep: "grep", replMode: { enabled: false }, + runnableOpts: { + saveLocations: true, + }, }); }); diff --git a/test/src/utils/typescript.ts b/test/src/utils/typescript.ts index fc78e4e1e..282505ef8 100644 --- a/test/src/utils/typescript.ts +++ b/test/src/utils/typescript.ts @@ -33,19 +33,6 @@ describe("utils/typescript", () => { assert.notCalled(registerStub); }); - it("should pass 'allowJs' option", () => { - ts.tryToRegisterTsNode(); - - assert.calledOnceWith( - registerStub, - sinon.match({ - compilerOptions: { - allowJs: true, - }, - }), - ); - }); - it("should respect env vars", () => { process.env.TS_NODE_SKIP_PROJECT = "false"; process.env.TS_NODE_TRANSPILE_ONLY = "false"; diff --git a/test/src/worker/browser-env/runner/test-runner/index.ts b/test/src/worker/browser-env/runner/test-runner/index.ts index bb35aaef4..492d3b0de 100644 --- a/test/src/worker/browser-env/runner/test-runner/index.ts +++ b/test/src/worker/browser-env/runner/test-runner/index.ts @@ -66,6 +66,7 @@ describe("worker/browser-env/runner/test-runner", () => { file: "/default/file/path", id: "12345", fn: sinon.stub(), + location: undefined, }) as TestType; test.parent = Suite.create({ id: "67890", title: "", file: test.file });