Does zora run async tests in parallel?

[mindplay-dk]

I have a bunch of async tests, and all of a sudden, it seems they've all started running on top of each other.

All of the code is async/await, including some async generators with for await in the test functions.

Does zora run tests in parallel?? I didn't think so.

It's more than possible I'm doing something seriously wrong on my end 😅

[lorenzofox3]

They run concurrently (not technically in parallel although I assume that's what you mean in your question).

However for some reason, if you want to run a bunch of tests sequencially, you can simply await one to finish as defined in the documentation.

the long answer

[mindplay-dk]

Right, I know I can await sub-tests inside a test - but these aren't sub-tests.

I can't have top-level await in the target node environment for this project, so the only way to run tests in serial would be to run all my tests from a single file?

Hmm, I think, for this project, I'm going to replace the test function with a function of my own, and simply collect all the tests in a list - I'm already using a custom harness, so the harness can (conditionally, in --debug mode only) run a single test and wrap all the other test calls that way.

This makes me think again about something I think I may have brought up early on, when I first started using zora: I kind of wish that's what the test function would do - collect and queue the tests, rather than starting them right away. In my own PHP-based test-framework that I used and maintained for many years, I was initially running tests right away, but eventually switched to the approach of merely collecting tests to be run by the harness - I think running the tests is typically one of the responsibilities of a test-harness in most test-frameworks?

One reason I switched to this approach in my own framework is because it enabled me to filter the list of tests before running it - for example, this lets me add something like [SLOW] to my test-description and --skip-slow to my test-script, where I would simply .filter the list of tests before launching the test-harness.

I know you're big on simplicity - and as you know, I am too, but I think our definition of simplicity is not always quite the same. 🙂

In this particular case, I would find it simpler if a test-harness was something you provided with a list of tests and reporter - a pure function with certain inputs and outputs. Instead, we have this test function that runs tests and stores results somewhere, for the harness to pick up later - I view this as unnecessary complexity. While, at the same time, I realize a change like that might make it slightly more difficult for a newcomer to figure out how to wire up the list of tests and the reporter with a harness, it's also not hiding any dependency; if you can't call a test-harness without providing it with a list of tests, that makes it completely obvious how things are coupled - it's much less obvious when these dependencies aren't visible at the surface level of an API.

Have you ever watched this? Rich Hickey is my simplicity Guru - my views on this stuff completely changed when I first discovered this video. I push it to every developer I know, in the hopes that this world view will catch on... where we draw a hard line between "simple" and "easy" - and with the awareness that favoring "easy" over "simple" usually means taking on extra complexity.

Also why I pushed for this and the "frameworks" vs "libraries" distinction, which to my understanding is about control - where "libraries" leave your program in control, "frameworks" generally take control and provide various hooks where it cedes control to you; "frameworks" inherently have more complexity in favor of making something "easy", whereas "libraries" generally have less complexity in favor of keeping things "simple". Zora, as I see it, is somewhere between a library and a framework, whereas pta is clearly a framework - I'd like to see zora itself become more of a pure library, making the roles and responsibilities (division of labor) between zora and pta very clear and explicit.

Just something to think about. I love zora either way and can find my way around these problems. I just wish the distinction was a little more clear cut, with zora never just "doing things" without your direct say so, but instead being specifically a tool you can "do things with" - leaving pta to the role of "doing things for you".

[mindplay-dk]

Eventually, if you want to go lower level on how files get loaded and things get reporter: use a custom test harness. (that is what pta does)

You're right, I think this is starting to sound like the best option for me - the test harness is simple enough to just replace.

The main reason this idea never even occurred to me is because it's there, you know? I always kind of figured, if it's there, I should use it - but now that you mention it, this can be viewed as just a default and not something you have to use.

For the record (and to others looking) I did actually solve it, by just adding my own test function:

import { createDiffReporter } from "zora-reporters";
import { createHarness, ISpecFunction } from "zora";

const tests: Array<{ description: string, spec: ISpecFunction }> = [];

export function test(description: string, spec: ISpecFunction) {
  tests.push({ description, spec });
}

export async function runTests() {
  const harness = createHarness({});

  harness.test("Test", async ({ test }) => {
    for (const { description, spec } of tests) {
      await test(description, spec);
    }  
  });

  await harness.report({
    reporter: createDiffReporter(),
  });
}

This just groups all the tests and makes them run within a parent test.

I'm not sure this will be good for reporting - and this is fast on the way to becoming a custom test harness anyway, so I'll probably try that.

Thanks again for taking the time to discuss! 🙂

[lorenzofox3]

oops, I realised I edited your comment instead of replying to it.

for the record

Right, I know I can await sub-tests inside a test - but these aren't sub-tests.

that's just a matter of wrapping within a function, or having an entry point. If you have some tests files overlapping with others, I would say you have an isolation issue in the frist place, even more as there is no such a thing in zora like "global mock", "beforeHook" etc. Most of the time this kind of issues does not belong to the testing code but rather to the tested code. To enforce isolation, many frameworks (ava, jest, etc) have decided to run each file in a child process with impact the performances.

I have managed to run successfully integration tests against a real DB with no issue whatsoever, test on CLI tools, on web components, etc so I beleive you should not have much problem, or you are probably doing something wrong.

I can't have top-level await in the target node environment for this project, so the only way to run tests in serial would be to run all my tests from a single file?

That could be a solution.

Hmm, I think, for this project, I'm going to replace the test function with a function of my own, and simply collect all the tests in a list - I'm already using a custom harness, so the harness can (conditionally, in --debug mode only) run a single test and wrap all the other test calls that way.

not sure hacking the lib is a great solution. It remains easily extensible anyway.

This makes me think again about something I think I may have brought up early on, when I first started using zora: I kind of wish that's what the test function would do - collect and queue the tests, rather than starting them right away. In my own PHP-based test-framework that I used and maintained for many years, I was initially running tests right away, but eventually switched to the approach of merely collecting tests to be run by the harness - I think running the tests is typically one of the responsibilities of a test-harness in most test-frameworks?

I have discussed that matter in length. But I don't want the user to build its own test runner on top of each of its project. One of the design goal is: "you don't need a specific test runner" (or a call to.run as it was the case in the first version), you can simply call node or drop the file in a browser, etc. you can concat/bundle files together with the exact same effect with no explicit dependency between them other than import {test} from 'zora'.

It is possible to build your own test runner thanks to the custom test-harness (like pta for example), and I am pretty happy with this kind of extension point.

There is also a technical limitation in the collect and test phases: for some subtests you need to run the parent test in order to collect them which

1. Is harder to maintain
2. impact the performances

One reason I switched to this approach in my own framework is because it enabled me to filter the list of tests before running it - for example, this lets me add something like [SLOW] to my test-description and --skip-slow to my test-script, where I would simply .filter the list of tests before launching the test-harness.

Yes every design has its tradeoff. In zora you'd need to use .skip or grep the outpout stream. adding a "skip" rule based on some tag in the description would not be too difficult though.

I know you're big on simplicity - and as you know, I am too, but I think our definition of simplicity is not always quite the same. 🙂

In this particular case, I would find it simpler if a test-harness was something you provided with a list of tests and reporter - a pure function with certain inputs and outputs. Instead, we have this test function that runs tests and stores results somewhere, for the harness to pick up later - I view this as unnecessary complexity. While, at the same time, I realize a change like that might make it slightly more difficult for a newcomer to figure out how to wire up the list of tests and the reporter with a harness, it's also not hiding any dependency; if you can't call a test-harness without providing it with a list of tests, that makes it completely obvious how things are coupled - it's much less obvious when these dependencies aren't visible at the surface level of an API.

I differ in opinion here. The majority of things you are referring here are implementation details. The high level contract is "just"

1. create a test with the ... test function
2. add some assertion in your spec function with the injected api
3. Control the concurrency of your tests with regular javascript idioms async/await

You can also easily enhance the assertion API, decorate the test function, handle the reporting on your own; so I beleive the lib does not lack extension points !

Eventually, if you want to go lower level on how files get loaded and things get reporter: use a custom test harness. (that is what pta does)

Have you ever watched this? Rich Hickey is my simplicity Guru - my views on this stuff completely changed when I first discovered this video. I push it to every developer I know, in the hopes that this world view will catch on... where we draw a hard line between "simple" and "easy" - and with the awareness that favoring "easy" over "simple" usually means taking on extra complexity.

Not I have not, thanks for the recommendation.

Also why I pushed for this and the "frameworks" vs "libraries" distinction, which to my understanding is about control - where "libraries" leave your program in control, "frameworks" generally take control and provide various hooks where it cedes control to you; "frameworks" inherently have more complexity in favor of making something "easy", whereas "libraries" generally have less complexity in favor of keeping things "simple". Zora, as I see it, is somewhere between a library and a framework, whereas pta is clearly a framework - I'd like to see zora itself become more of a pure library, making the roles and responsibilities (division of labor) between zora and pta very clear and explicit.

I understand, and I believe that is pretty easy to do easy with packaging and I have already pointed to you an example (removing all the crap that comes with CJS, TS, etc).

However I want also zora to be able to run test in the first place, and in any javascript env. That's why this auto launch thing exists

Just something to think about. I love zora either way and can find my way around these problems. I just wish the distinction was a little more clear cut, with zora never just "doing things" without your direct say so, but instead being specifically a tool you can "do things with" - leaving pta to the role of "doing things for you".

That distinction will not happen beyond packaging (which should be more than enough). Moreover you can already call createTestHarness to hide the side effects if that is what you wish

[mindplay-dk]

So I tried to create my own harness, and I got something that works, but not really the way I wanted or expected.

import { createDiffReporter } from "zora-reporters";
import { createHarness, ISpecFunction } from "zora";
import { options } from "./options"; // this module reads various command-line arguments

const tests: Array<{ description: string, spec: ISpecFunction }> = [];

export function test(description: string, spec: ISpecFunction) {
  tests.push({ description, spec });
}

export async function runTests() {
  const harness = createHarness({});

  for (const { description, spec } of tests) {
    if (options.debug) {
      console.log();
      console.log(`-----[ ${description} ]-----`);
    }

    const result = harness.test(description, spec);

    if (options.debug) {
      await result; // blocks tests from running in parallel
    }
  }

  await harness.report({
    reporter: createDiffReporter(),
  });
}

What I was expecting here, was I would just copy/paste the built-in harness and work from there - but it relies on internals that aren't available to other modules. So I looked at pta to see what it's doing, and that wouldn't have worked for me as a starting point either, as it relies on the built-in harness and the test hook to actually run the tests - that approach doesn't give me control of running the tests.

I was left with no other option besides harnessing the built-in harness. 😄

It works just fine - I now have a function where I can optionally await all tests and run them in serial.

However, this only works for my top-level test replacement function - sub-tests would still run in parallel. I don't see any obvious way to change that.

Writing a true custom harness, rather than wrapping the existing one, would have definitely been my preference - I think this would require exposing things like Assert and createAssert, and the real createHarness from inside the harness.js, without the "test runner" additions from index.js.

Any chance you would still consider the zora/core idea?