Nested tests block the stream?

[mindplay-dk]

I was under the impression that zora would actually stream results to a reporter as soon as possible.

That appears not to be the case?

import { test } from 'zora';

const wait = (time) =>
  new Promise((resolve) => {
    setTimeout(() => resolve(), time);
  });

test('test 1', async (t) => {
  t.ok(true, 'hello 1');

  t.test('test 2', async (t) => {
    t.ok(true, 'hello 2');
    await wait(2000);
  });

  t.ok(true, 'hello 3');
});

In this example, starting test 2 will block stream for 2 seconds - the hello 2 and hello 3 messages will be blocked by from reaching the reporter until test 2 has finished.

Looking at the source-code, sure enough, every test will await testRoutine, effectively waiting for the entire test to complete, before yield* assertion - running results get pushed to a list of assertions by onResult, and are stuck there until the test finishes.

Maybe it's by design. Maybe it doesn't matter. I'm not raising this as a matter of criticism, I'm just trying to understand more intricately how this works. I had assumed the use of asyncIterator was a means of streaming the results all the way from the front-end (tests) to the back-end (reporter).

Is there any particular benefit to using asyncIterator in this way? I mean, is it any better or worse than simply returning a Promise with assertions? All of the results are essentially "stuck" in assertions until the top-level test collects everything anyway - the only thing that gets through right away is the test description.

But maybe that's all it was intended to do? 🙂

[lorenzofox3]

I was under the impression that zora would actually stream results to a reporter as soon as possible.

That appears not to be the case?

In this example, starting test 2 will block stream for 2 seconds - the hello 2 and hello 3 messages will be blocked by from reaching the reporter until test 2 has finished.

Looking at the source-code, sure enough, every test will await testRoutine, effectively waiting for the entire test to complete, before yield* assertion - running results get pushed to a list of assertions by onResult, and are stuck there until the test finishes.

Maybe it's by design. Maybe it doesn't matter. I'm not raising this as a matter of criticism, I'm just trying to understand more intricately how this works. I had assumed the use of asyncIterator was a means of streaming the results all the way from the front-end (tests) to the back-end (reporter).

There are two disctincts things:

1. running the tests
2. reporting the results

By default all the tests run concurrently and eagerly (thanks to this IIFE - immediately invoked function expression) as soon as possible. So if you write:

sample program with logs

import { test } from 'zora';

const wait = (time) =>
  new Promise((resolve) => {
    setTimeout(() => resolve(), time);
  });

test('test 1', async (t) => {
  t.ok(true, 'hello 1');
  console.log('1 done');

  t.test('test 2', async (t) => {
    t.ok(true, 'hello 2');
    await wait(2000);
    console.log('2 done');
  });

  t.ok(true, 'hello 3');
  console.log('3 done');
});

you will see the log for the third test before the one for the second test which has not finished yet.

For the reporting part, the message stream keeps the order as defined by the program: from a reporter perspective, that does not make much sense to know that the fifth test has completed whereas you are still processing the the third one; that would be quite a burden on the reporter to keep track of the state of all the tests for little benefit. It is much easier to read and process a linear sequence of messages (that is a simple for of loop).

It means that the reporting stream can wait for a test to complete before sending messages it might already be aware of, but that does not mean the tests are blocked.

If you write the following program for example:

concurrent tests example

import { test } from 'zora';

const wait = (time) =>
  new Promise((resolve) => {
    setTimeout(() => resolve(), time);
  });

test('parent', async (t) => {
  t.test('test 1', t => {
    t.ok(true,'hello1');
  })

  t.test('test 2', async (t) => {
    t.ok(true, 'hello 2');
    await wait(2000);
  });
  
  t.test('test 3', async (t) => {
    t.ok(true, 'hello 3');
    await wait(3000);
  });
  
  t.test('test 4', async (t) => {
    t.ok(true, 'hello 4');
    await wait(2000);
  });
  
});

You'll see the reporting stream prints the result fort test 1, then waits for 2 seconds before the test 2 complete but it will wait only 1 second (and not three) to report the third test afterward. The fourth test will then follow immediately

Is there any particular benefit to using asyncIterator in this way? I mean, is it any better or worse than simply returning a Promise with assertions? All of the results are essentially "stuck" in assertions until the top-level test collects everything anyway - the only thing that gets through right away is the test description.

But maybe that's all it was intended to do? 🙂

Not all the results are stuck. In your example, you'll still see the test report for the first assertion whereas the reporting stream is still waiting for the second one to complete. And If you write more assertions/sub tests which are faster than 2 seconds before that second test, you will see them as well

The main advantage of async Iterable here is to write in a single imperative function all the lifecycles (started, ended, etc) and handle the recursivity for sub tests: yield *.

Also, Async iterables play nicely with web and node streams while providing a much user friendly API (a whole transform stream is a simple map function):

custom reporter example

// stream-utils.js 
export const map = (mapFn) => async function* (stream) {
    for await (const streamElement of stream) {
        yield mapFn(streamElement);
    }
};

export const filter = (predicate) => async function* (stream) {
    for await (const streamElement of stream) {
        if (predicate(streamElement)) {
            yield streamElement;
        }
    }
};

// reporter.js
import {diffString} from 'json-diff';
import {map, filter} from './stream-utils.js'

const keepFailingOnly = filter((message) => message?.data?.pass === false);
const diff = map(({data: {expected, actual, at}}) => ({
    at,
    diff: diffString(expected, actual)
}));

export const diffReporter = (opts = {}) => async (messageStream) => {
    const sourceStream = diff(keepFailingOnly(messageStream));
    for await (const {at, diff} of sourceStream) {
        console.log(diff);
        console.log(at);
    }
};

// test.js
import {createHarness} from 'zora';
import {diffReporter} from './reporter.js'

const harness = createHarness();

harness.test('test 1', (t) => {
    t.eq({foo: 'bar', woot: 'woot'}, {foo: 'baz', woot: 'woot'});
});

harness.report({reporter: diffReporter()});

And that's it you have a custom reporter !

[mindplay-dk]

It means that the reporting stream can wait for a test to complete before sending messages it might already be aware of, but that does not mean the tests are blocked.

Maybe I should have clarified this point: I understand that the tests are not blocked - what my example illustrates, is that the messages are blocked from reaching the reporter, essentially until all the tests have fully resolved.

To illustrate:

That is, the hello 2 assertion is actually ready to report two seconds before test 2 completes.

I understand the test itself isn't blocked from running, but the reporter could have continued it's work while we're waiting for promises in the test to resolve - we could have been feeding the reporter messages about whatever happens in test 2, while it's happening.

As said though, maybe fixing this is a micro optimization for an edge-case of sorts. In this particular example, it so happens we could have continued reporting, but there would be other examples where that's not the case... for example:

import { test } from 'zora';

const wait = (time) =>
  new Promise((resolve) => {
    setTimeout(() => resolve(), time);
  });

  test('test 1', async (t) => {
    t.ok(true, 'hello 1');
  
    t.test('test 2', async (t) => {
      t.ok(true, 'hello 2');
      await wait(2000);
    });

    t.test('test 3', async (t) => {
      t.ok(true, 'hello 3');
    });
  });

Even if we were able to continue reporting the hello 2 assertion here, we couldn't report anything from test 3 before test 2 has finished, because (assuming we want to support a streaming report output at all) we do still want to report the events in "source order" rather than "execution order".

Part of me wonders though... 🤔

Since (in an example like the last one here) we aren't able to continue with the reporting stream anyhow, and (in practice) all (or most) of the test results can end up buffered in arrays in call-stacks that can't resolve until the whole test has finished anyhow, is there even (in principle) anything won by streaming results to reporters in the first place?

Likely the only reporter than can actually stream it's output while it's building it, is the console reporter anyhow? Something like an XML or HTML reporter, where the output isn't sequential like it is in TAP/JSON/console output, would need to collect all the results and sort through them before producing their output anyhow.

I'm just playing devil's advocate and thinking out loud here, but if some types of reporters have to aggregate everything in memory anyhow, wouldn't it be even simpler to just aggregate all the results in an in-memory data-structure, or a single JSON report, and use that as the input to reporters?

On the other hand, your last example with the custom reporter and stream-utils clearly does potentially save memory in any middle layers of the testing pipeline - even if running the test ends up holding all results in-memory, and a reporter ends up doing the same, at least any transformations in the middle of that pipeline aren't copying the entire stream, right?

---

Having gone through all these motions through the week, I actually arrived at a place where I started to wonder, what is even so great about nested tests in the first place? Is it even a good idea? I started reading up and came across this article by Kent C. Dodds:

https://kentcdodds.com/blog/avoid-nesting-when-youre-testing

He makes some really good arguments. Nested tests aren't actually very easy to reason about. The reader does have to wonder about execution order among the individual nested tests, especially when there is async involved - it can get pretty confusing.

Maybe this feature exists mostly for the sake of logical grouping? But closures do have access to variables in their parent scope, so even if the intention is to use it for logical grouping only, arguably, it does represent a potential footgun, enabling you to write very confusing tests.

(In my usual style, this thread is now on two different topics, hehe - this is just a discussion though, I'm not raising an issue, just trying to get a deeper understanding of this stuff. Don't feel obligated to respond - part of this is probably just my personal process of putting things in writing to solidify what I'm learning. 😊)

[lorenzofox3]

It means that the reporting stream can wait for a test to complete before sending messages it might already be aware of, but that does not mean the tests are blocked.

Maybe I should have clarified this point: I understand that the tests are not blocked - what my example illustrates, is that the messages are blocked from reaching the reporter, essentially until all the tests have fully resolved.

To illustrate:

That is, the hello 2 assertion is actually ready to report two seconds before test 2 completes.

I understand the test itself isn't blocked from running, but the reporter could have continued it's work while we're waiting for promises in the test to resolve - we could have been feeding the reporter messages about whatever happens in test 2, while it's happening.

Ah I see. Yes you are correct. It used to be the case in the earlier versions of zora but I found it made the code more complicated for little benefit.

As said though, maybe fixing this is a micro optimization for an edge-case of sorts. In this particular example, it so happens we could have continued reporting, but there would be other examples where that's not the case... for example:

import { test } from 'zora';

const wait = (time) =>
  new Promise((resolve) => {
    setTimeout(() => resolve(), time);
  });

  test('test 1', async (t) => {
    t.ok(true, 'hello 1');
  
    t.test('test 2', async (t) => {
      t.ok(true, 'hello 2');
      await wait(2000);
    });

    t.test('test 3', async (t) => {
      t.ok(true, 'hello 3');
    });
  });

Even if we were able to continue reporting the hello 2 assertion here, we couldn't report anything from test 3 before test 2 has finished, because (assuming we want to support a streaming report output at all) we do still want to report the events in "source order" rather than "execution order".

Yes, I am not sure it worths the effort. The most important thing to me is to make sure the user understands its tests are running.

Part of me wonders though... 🤔

Since (in an example like the last one here) we aren't able to continue with the reporting stream anyhow, and (in practice) all (or most) of the test results can end up buffered in arrays in call-stacks that can't resolve until the whole test has finished anyhow, is there even (in principle) anything won by streaming results to reporters in the first place?

Not necessary streaming on its own but you still avoid the reporter to flat map the output on its side: some assertions sometimes are sub tests, but sometimes they are not. Again, like this a reporter is a simple for of loop or a map function.

Also, from my experience, the blocking you are facing is not really common or noticeable, so streaming the report gives the impression to the user that the execution is progressing in most of the cases.

Likely the only reporter than can actually stream it's output while it's building it, is the console reporter anyhow? Something like an XML or HTML reporter, where the output isn't sequential like it is in TAP/JSON/console output, would need to collect all the results and sort through them before producing their output anyhow.

I would say the opposite: most of the reporters will be sequential it the way they process the data (because it is much easier to build) even if they buffer the sequence to output the result only once it is finished.

Moreover it is much easier to build a buffer from a stream than the other way round. I mean if you wish to create a specific data structure which output all the result at once, that can be a simple stream reducer:

simple stream reducer which keeps assertions data only

const assertions = filter(({type}) => type === 'ASSERTION');

const createReporter = () => async function (stream) {
    const passing = [];
    const failing = [];
    
    for await (const message of assertions(stream)){
        (message.data.pass === true ? passing : failing).push(message.data)
    }
   
    // returns a custom data structure to be eventually processed down stream
    return {
        passing,
        failing
    }
};

I'm just playing devil's advocate and thinking out loud here, but if some types of reporters have to aggregate everything in memory anyhow, wouldn't it be even simpler to just aggregate all the results in an in-memory data-structure, or a single JSON report, and use that as the input to reporters?

That can be the case, but as in the example above, that will be a simple extra layer on the reporter side. It is not straight forward to come up with a data structure which accommodates all the expectations of all the reporters. A list of messages is much easier to extend and to convert into any data structure. I see very little benefit to do that buffering/etc on the zora's side

On the other hand, your last example with the custom reporter and stream-utils clearly does potentially save memory in any middle layers of the testing pipeline - even if running the test ends up holding all results in-memory, and a reporter ends up doing the same, at least any transformations in the middle of that pipeline aren't copying the entire stream, right?

Correct. I would not be concerned about memory consumption though, unless you have millions of tests.. and in this case your testing architecture will be likely very different.

Having gone through all these motions through the week, I actually arrived at a place where I started to wonder, what is even so great about nested tests in the first place? Is it even a good idea? I started reading up and came across this article by Kent C. Dodds:

https://kentcdodds.com/blog/avoid-nesting-when-youre-testing

He makes some really good arguments. Nested tests aren't actually very easy to reason about. The reader does have to wonder about execution order among the individual nested tests, especially when there is async involved - it can get pretty confusing.

Maybe this feature exists mostly for the sake of logical grouping? But closures do have access to variables in their parent scope, so even if the intention is to use it for logical grouping only, arguably, it does represent a potential footgun, enabling you to write very confusing tests.

Yes that's a good question, the main reason why it does exist in zora is to give the ability to the dev to control the concurrency model with a simple well known control flow: await expression . If you want to build sequential tests for example (integration with a db, e2e scenario, etc ? ):

sequential tests

test('scenario 1`, async t => {
 
  await t.test('step 1`, t => {
      // will mutate a global state
  })
  
  //will only run when step 1 is over
  await t.test('step 2', t => {
    //etc
  })

});

(In my usual style, this thread is now on two different topics, hehe - this is just a discussion though, I'm not raising an issue, just trying to get a deeper understanding of this stuff. Don't feel obligated to respond - part of this is probably just my personal process of putting things in writing to solidify what I'm learning. 😊)

No worries, I appreciate your contributions which definitely make zora better. Even if sometimes my written communication lets you feel I am annoyed or bothered, that is not the case. If I disagree with your opinion I can simply discard your suggestions

[mindplay-dk]

Yeah, after some experimentation, I arrived at the same conclusion: the reporter blockage is not like to be a problem in practice - fixing it did add considerable complexity, and not least a new promise that needs to be resolved for every individual assertion. That alone will add too much overhead - there's no way this would be worth while. And it still wouldn't solve for the case where one test blocks the next, so this was a bad idea overall. 😅

I appreciate you taking the time to discuss! 🙂

[mindplay-dk]

Just for completeness, here's the facility I came up with, which provides a synchronous (push) API and maps it to an async iterable:

https://gist.github.com/mindplay-dk/76d07c2f065c73213556349363c8fb5d

I don't think we have a use for this, just posting in case others or future self finds this and wonders how that was doable. 🙂