proposal: introduce "jest.runMode" to consolidate and simplify settings

[connectdotz]

Why

1. There seems to be a common misconception that by setting "jest.autoRun": "off" the extension should be disabled, so users are often surprised and confused when the errors reveal the output terminals.
2. The autoRevealOutput setting default is on-run, while make sense for projects that already setup to run, it might not be the right default for projects who are not ready to run (they will see it pop-up due to env related error for example). It seems we might need different defaults for different types of use-case?
3. In v6, we introduced a new setting "jest.enable" (adding jest.enable setting and adjust workspace operations accordingly #1009) to disable the extension, but then discovered there might be another in-between state also desirable (see Performance problems on large, multi-package monorepos #1030).
4. It seems we have more and more settings; while it helps tailor the tools to users' preferences, it has inevitably become complex and hard to discover.
5. Address Feature request: Introduce a power save mode #1025

What

This discussion addresses these issues by consolidating related settings into one "jest.runMode", indicating the type of runtime preference so we could coherently tailor the related settings with appropriate defaults instead of discovering and adjusting them separately.

The new setting jest.runMode will supersede jest.autoRun, jest.autoRevealOutput, and jest.showCoverageOnLoad settings, which will be marked deprecated and be phased out in future releases.

RunMode Type

There are 3 types of run modes (similar to autoRun), with some common options:

interface JestRunModeOptions {
  runAllTestsOnStartup?: boolean;
  coverage?: boolean;
  revealOutput?: 'on-run' | 'on-exec-error' | 'on-demand';
  deferred?: boolean;
}
export type JestRunMode = (
  | { type: 'watch' }
  | { type: 'on-demand' }
  | { type: 'on-save'; testFileOnly?: boolean }
) &
  JestRunModeOptions;

Pre-Defined RunMode ShortCut

There will be some pre-defined run-modes for convenience that most users would probably use, while internally, they will be mapped to the actual RunMode, allowing savvy users to customize further.

Predefined runMode	description	actual value as RunMode
"watch"	run tests by watchman	{type: "watch", revealOutput: "on-run"}
"on-save"	run tests when test or source files are saved	{type: "on-save", revealOutput: "on-run", testFileOnly: false}
"on-demand"	run tests on-demand through UI	{type: "on-demand", revealOutput: "on-run"}
"deferred"	defer test run and discovery until the first on-demand run	{type: "on-demand", revealOutput: "on-run", deferred: true }

• Note there is no "disabled" runMode anymore since it doesn't map to any runMode, and we might just as well continue using jest.enable for it.

Example

with predefined run mode:
"jest.runMode": "deferred", "jest.runMode": "on-demand".

with actual run mode:

"jest.runMode": {
  "type": "on-save",
  "revealOutput": "on-run",
  "runAllTestsOnStartup": true,
  "coverage": true,
  "testFileOnly": true
}

or

"jest.runMode": {
  "type": "on-demand"
  "revealOutput": "on-run",
 "deferred": true
}

Runtime support

We will support runMode switch at run time, similar to the current autoRun/coverage toggling in TestExplorer. We could add a new feature to allow users to update the permanent setting based on the current runtime config, which can also be used for migration purposes.

Implementation Notes

The following are used for implementation discussion. If we set the run mode correctly, hopefully, we don't need to explain any of these to the end-users...

RunMode Features and Operations

The main feature and operation of each mode correspond to:

Type	Activate extension env1 (show TestExplorer roots)	Parse and find TestFiles2 (show test blocks)	AutoRun Tests	Trigger Test via UI
watch	Y	Y	by watchman	Y
on-save	Y	Y	when files are saved	Y
on-demand	Y	Y	N	Y
deferred	Y	N	N	Y
disabled	N	N	N	N

---

1: In this stage, the extension will examine if the environment is ready by finding and validating the settings, jest commands, workspace folders, etc. The TestExplorer will only show qualified workspace items, no test block items yet.
2: In this stage, the extension will invoke the jest command to find test files and parse the test files in order to show test block in the TestExplorer tree view as well as the editor gutter menu.

[albertogasparin]

As someone who is also struggling with a very large codebase (and I’m looking forward to being able to use your extension daily), I wonder if trying to give a name to a group of behaviours is problematic to begin with and might confuse people even more.
What I’m hinting at is that, if I look at the table, I see 3 capabilities (for now, as manual is kind of a constant) and I need to map it to 5 config names. And those five do not even cover the scenario that I’d love to have supported: having auto-run on save without test finder.

So my recommendation is to instead have a capability based setting that allows devs with special requirements to turn off/tweak the “default”, which is not much different from the current status:

type config = {
  enabled: boolean,
  testDiscovery: boolean,
  autoRun: boolean | on-save
}

[connectdotz]

@albertogasparin, thanks for the feedback. I realized the original post might be too heavy on the internal implementation than the end-user experience. The post is updated, and feel free to skim it again.

The feature table mapping is for internal implementation actually, not for end-users. The goal here is to provide a simple use-case centric setting to spare users from needing to know the extension's inner workings. The separate settings approach you suggested, as well as the current settings, could be difficult for inexperienced users to discover what is possible and needed, it also did not prevent the impossible combinations: for example testDiscovery: false and autoRun: true are incompatible.

having auto-run on save without test finder.

This is a good point. We are indeed missing an important key element in the deferred mode! I added the deferredRunMode for users to specify how they want to run the tests once "activated".

Let me know if it makes more sense now.

[albertogasparin]

I like the customisation way, but I wonder if you might be better being more explicit on who is winning with those settings.
For instance, if I look at { "type": "deferred", "deferredRunMode": "on-save" } I might be confused on what is overriding what.

Usually for those sort of combinations of presets/custom, it might be better using 2 distinct keys. For instance:

"jest.runMode": "on-save",
"jest.runModeOverrides": { "runAllTestsOnStartup": true }

So, most users likely would still only have the base setting, and advanced users have clear idea of what they override.

Alternatively, you could just force one or the other, saying that advanced users need to enable the things they want (as by default everything is "off").

"jest.runMode": "on-save",
// or
"jest.runMode": { "runAllTestsOnStartup": true, ... }

[connectdotz]

@albertogasparin, thanks for the question. After thinking more about it, I realized that each basic mode should be deferrable. We don't need a separate "deferred" mode, but a deferred option for each mode. We can still have a pre-defined "deferred" shortcut, but not necessarily an actual run mode type... please see the updated proposal above and let me know if it is clear.

[drew-walker]

I still can't really comprehend from the proposal how deferring test discovery will solve #1030. Even with the "deferred" option, if test discovery happens for the all packages in the workspace, we'll still be exhausting memory, just at a different point in time. Or will the test discovery behavior change such that discovery will be scoped to the package being tested? (Rather than for all packages)

[connectdotz]

@drew-walker, test discovery is performed and controlled per (virtual) workspace folder; runMode is under the same scope.

If you have multiple packages in your project, and you use multi-root workspace, i.e. multiple workspace folders; or virtual workspace folders, i.e. multiple virtual folders, then everything will work on the package level as expected. (you exit defer mode by package: either by running tests through UI or by setting the runMode.deferred option to false)

But if you have just 1 workspace folder that contains multiple packages, then yes the discovery will be performed for the whole folder, including every package under it.

This is the fundamental structure of the extension, the new setting won't change that. I assumed you used a multi-root workspace, is that correct?

[connectdotz]

@drew-walker, @albertogasparin and whoever interested in this, here is a prototype: vscode-jest-6.0.2.vsix.zip, feel free to give it a try when you get a chance and let me know if it addresses your use case.

[connectdotz]

PR #1078 merged. Close the discussion.