feat(tests): Include report section for when tests fail due to missing opportunity data (random mode) #647

lukehesluke · 2024-02-27T17:15:49Z

closes #643

QA

This zip contains some output summary pages from different runs of Test Suite.

Both runs used config/default-dev.json but with useRandomOpportunities: true.

summary_with_no_errors.html is a successful run against just the opportunity-free test. You can see here that there is no "missing data" section
summary_with_lots_of_errors.html is a run against the payment category on a Booking System which had no opportunity data (Reference Implementation using OPPORTUNITY_COUNT=1). You can see here the Missing Opportunity Data section

qa.zip

…om-mode-ran-out-of-data

lukehesluke · 2024-02-28T10:32:37Z

packages/openactive-broker-microservice/app.js

@@ -987,6 +987,7 @@ app.post('/test-interface/datasets/:testDatasetIdentifier/opportunities', functi
    logError(`Random Bookable Opportunity from seller ${sellerId} for ${criteriaName} within ${bookingFlow} (${opportunityType}) call failed: No matching opportunities found`);
    res.status(404).json({
      error: `Opportunity of type '${opportunityType}' that has an \`organizer\`/\`provider\` with \`@id\` '${sellerId}', that also matched '${criteriaName}' within '${bookingFlow}', was not found within the feeds.`,
+      type: 'OpportunityNotFound',


So that this 404 is not confused with other types of 404s when determining if there are missing opportunities

lukehesluke · 2024-02-28T10:34:33Z

packages/openactive-integration-tests/test/helpers/request-helper.js

+    const opportunityNotFound = (
+      respObj.response.statusCode === 404
+      && respObj.body?.type === 'OpportunityNotFound'
+    );
+    if (opportunityNotFound) {
+      this.logger.recordFlowStageEvent(stage, {
+        type: 'OpportunityNotFound',
+        opportunityType,
+        testOpportunityCriteria,
+        bookingFlow,
+        sellerId,
+      });
+    }


This is where the "opportunity missing" event is saved

lukehesluke · 2024-02-28T10:41:06Z

packages/openactive-integration-tests/test/report-templates/summary.md.handlebars

+  {{#each missingOpportunityDataSummary}}
+
+  <li>
+    ❌️ Missing Criteria: <strong><a href="https://openactive.io/test-interface#{{@key}}">{{@key}}</a></strong>


Lots of ❌️s to hopefully make it clear that these are important issues 🤷

civsiv · 2024-02-28T11:56:29Z

packages/openactive-integration-tests/test/helpers/logger.js

@@ -1,3 +1,27 @@
+// TODO fix these issues!


Might be a good issue to raise and do, for maintainability

Good point @civsiv . I've created an issue to do that, and broadly fix all the ESLint sins, here: #648

nickevansuk

Brilliantly elegant solution 👍 - a couple of wording suggestions and something about grouping

nickevansuk · 2024-03-04T14:44:06Z

packages/openactive-integration-tests/test/report-templates/summary.md.handlebars

+Some of the tests failed because there were not enough opportunities matching
+the required [criteria](https://github.com/openactive/openactive-test-suite/blob/master/packages/openactive-integration-tests/test/features/README.md).
+
+This only happens when using [Random Mode](https://developer.openactive.io/open-booking-api/test-suite#random-mode)
+as opposed to [Controlled Mode](https://developer.openactive.io/open-booking-api/test-suite#controlled-mode).
+
+In order to pass these tests, you will need to generate more opportunities in
+your Booking System. Make sure that there are many opportunities available for
+each criteria.
+
+You can use [test-data-generator](https://github.com/openactive/openactive-test-suite/tree/master/packages/openactive-integration-tests/test-data-generator)
+to get a specification of how much of each type of opportunity you need to
+generate.
+
+If any of these criteria are not relevant to your Booking System, you may
+need to remove the relevant feature from your Test Suite configuration
+(See [`implementedFeatures`](https://github.com/openactive/openactive-test-suite/tree/master/packages/openactive-integration-tests#implementedfeatures)).


Suggested change

Some of the tests failed because there were not enough opportunities matching

the required [criteria](https://github.com/openactive/openactive-test-suite/blob/master/packages/openactive-integration-tests/test/features/README.md).

This only happens when using [Random Mode](https://developer.openactive.io/open-booking-api/test-suite#random-mode)

as opposed to [Controlled Mode](https://developer.openactive.io/open-booking-api/test-suite#controlled-mode).

In order to pass these tests, you will need to generate more opportunities in

your Booking System. Make sure that there are many opportunities available for

each criteria.

You can use [test-data-generator](https://github.com/openactive/openactive-test-suite/tree/master/packages/openactive-integration-tests/test-data-generator)

to get a specification of how much of each type of opportunity you need to

generate.

If any of these criteria are not relevant to your Booking System, you may

need to remove the relevant feature from your Test Suite configuration

(See [`implementedFeatures`](https://github.com/openactive/openactive-test-suite/tree/master/packages/openactive-integration-tests#implementedfeatures)).

Some of the tests failed because there were not enough opportunities in the booking system under test that matched the required [criteria](https://github.com/openactive/openactive-test-suite/blob/master/packages/openactive-integration-tests/test/features/README.md).

This only happens when using [Random Mode](https://developer.openactive.io/open-booking-api/test-suite#random-mode)

as opposed to [Controlled Mode](https://developer.openactive.io/open-booking-api/test-suite#controlled-mode).

In order to pass these tests, please increase the number of opportunities in the Booking System. Make sure that there are many opportunities available that match each criteria.

You can use [test-data-generator](https://github.com/openactive/openactive-test-suite/tree/master/packages/openactive-integration-tests/test-data-generator)

to get a specification of how many of each type of opportunity needs to exist in the Booking System.

If any of these criteria are not relevant to your Booking System, you may

need to remove the relevant feature from your Test Suite configuration

(See [`implementedFeatures`](https://github.com/openactive/openactive-test-suite/tree/master/packages/openactive-integration-tests#implementedfeatures)).

I've implemented this

nickevansuk · 2024-03-04T14:50:23Z

packages/openactive-integration-tests/test/report-generator.js

+   * Groups the results from `missingOpportunityDataStats` into something
+   * that can be shown in the summary report.


Suggest that the count of opportunities is displayed for each combination of Sellers, Opportunity Types and Booking Flows.

For example, looking at the error below, it is not clear what opportunities need to be created (how many ScheduledSessions and how many Slots across the 72):

@nickevansuk this is similar to my initial implementation for this, but the issue was that the output would very quickly become huge. Even just for the payment category of tests (let alone the full test run!), the missing data section was several pages long and which is just number counts for each combination, which one can easily get lost in, and we also now have the reverse problem of users not realising that there are comprehensive test results at the end of all this!

I had a chat with @civsiv about this as well and we considered a few possible designs and then agreed that it would be good to get this information across, but difficult at present. And, regardless, it's currently possible to get this information by running Test Data Generator (which is now recommended in the output text if there is any missing data).

Besides that, the main use case which we're aware of from our experience is when a data publisher has simply forgotten to run some script or do some action which generates loads of random test data. So in those sorts of cases, they just need a rough idea that they need to generate more data.

In future, perhaps we could have something like the summarised view as it is but which links to a more complicated file, which perhaps is just generated using test-data-generator (DRY) to shortcut the journey for the user.

Great thinking! Responding to this late via a new issue here: #684

nickevansuk · 2024-03-04T14:55:33Z

packages/openactive-integration-tests/test/report-templates/summary.md.handlebars

+This is a list of the criteria, opportunity type, etc for which no opportunities
+were found, that also includes a list of which tests failed as a result:


Suggested change

This is a list of the criteria, opportunity type, etc for which no opportunities

were found, that also includes a list of which tests failed as a result:

This is a list of the opportunities that the test suite requires which were not found in the Booking System:

(see my next comment)

nickevansuk · 2024-03-04T14:58:03Z

packages/openactive-integration-tests/test/report-templates/summary.md.handlebars

+  {{#each missingOpportunityDataSummary}}
+
+  <li>
+    ❌️ Missing Criteria: <strong><a href="https://openactive.io/test-interface#{{@key}}">{{@key}}</a></strong>


Suggested change

❌️ Missing Criteria: <strong><a href="https://openactive.io/test-interface#{{@key}}">{{@key}}</a></strong>

❌️ Missing opportunities matching criteria: <strong><a href="https://openactive.io/test-interface#{{@key}}">{{@key}}</a></strong>

As it's the opportunities that are missing rather than the criteria?

It's opportunities that are missing but the list is grouped by Criteria. Which was arrived at in the summarization process that I talked about in my earlier comment

…om-mode-ran-out-of-data

lukehesluke added 9 commits February 26, 2024 17:22

log when run out of data (but do not yet report it)

cd9f9ab

wip

d3f26d8

Merge remote-tracking branch 'origin/master' into feature/report-rand…

2233174

…om-mode-ran-out-of-data

kinda works

6c5d2f4

change it up. why not (change the groupings to be a bit more compact)

5138b0d

cleanup

476c3d8

summarize the output

d1f4012

et finit

bade5dc

fix TS errors in report-generator

ae13a92

lukehesluke commented Feb 28, 2024

View reviewed changes

lukehesluke added 3 commits February 28, 2024 10:35

.

10bbb28

.

12e41ff

.

69741df

lukehesluke commented Feb 28, 2024

View reviewed changes

lukehesluke marked this pull request as ready for review February 28, 2024 10:41

lukehesluke requested review from civsiv and nickevansuk and removed request for civsiv February 28, 2024 10:41

civsiv reviewed Feb 28, 2024

View reviewed changes

civsiv approved these changes Feb 28, 2024

View reviewed changes

nickevansuk requested changes Mar 4, 2024

View reviewed changes

Merge remote-tracking branch 'origin/master' into feature/report-rand…

4da1972

…om-mode-ran-out-of-data

lukehesluke force-pushed the feature/report-random-mode-ran-out-of-data branch from 170d0be to 4da1972 Compare March 5, 2024 14:22

lukehesluke added 5 commits March 5, 2024 14:28

wording fixes

bb1511f

Merge branch 'master' into feature/report-random-mode-ran-out-of-data

fb8a33e

Merge branch 'master' into feature/report-random-mode-ran-out-of-data

17d9e86

Merge branch 'master' into feature/report-random-mode-ran-out-of-data

4139a3c

Merge branch 'master' into feature/report-random-mode-ran-out-of-data

149e3fb

lukehesluke merged commit e2f4329 into master Mar 11, 2024
32 checks passed

lukehesluke deleted the feature/report-random-mode-ran-out-of-data branch March 11, 2024 10:24

nickevansuk mentioned this pull request Apr 23, 2024

Improve missing opportunity data errors to include specific numbers #684

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tests): Include report section for when tests fail due to missing opportunity data (random mode) #647

feat(tests): Include report section for when tests fail due to missing opportunity data (random mode) #647

lukehesluke commented Feb 27, 2024 •

edited

Loading

lukehesluke Feb 28, 2024

lukehesluke Feb 28, 2024

lukehesluke Feb 28, 2024 •

edited

Loading

civsiv Feb 28, 2024

lukehesluke Feb 28, 2024

nickevansuk left a comment

nickevansuk Mar 4, 2024

lukehesluke Mar 5, 2024

nickevansuk Mar 4, 2024

lukehesluke Mar 5, 2024 •

edited

Loading

nickevansuk Apr 23, 2024

nickevansuk Mar 4, 2024

lukehesluke Mar 5, 2024

nickevansuk Mar 4, 2024

lukehesluke Mar 5, 2024

		* Groups the results from `missingOpportunityDataStats` into something
		* that can be shown in the summary report.

		This is a list of the criteria, opportunity type, etc for which no opportunities
		were found, that also includes a list of which tests failed as a result:

	This is a list of the criteria, opportunity type, etc for which no opportunities
	were found, that also includes a list of which tests failed as a result:
	This is a list of the opportunities that the test suite requires which were not found in the Booking System:

	❌️ Missing Criteria: <strong><a href="https://openactive.io/test-interface#{{@key}}">{{@key}}</a></strong>
	❌️ Missing opportunities matching criteria: <strong><a href="https://openactive.io/test-interface#{{@key}}">{{@key}}</a></strong>

feat(tests): Include report section for when tests fail due to missing opportunity data (random mode) #647

feat(tests): Include report section for when tests fail due to missing opportunity data (random mode) #647

Conversation

lukehesluke commented Feb 27, 2024 • edited Loading

QA

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukehesluke Feb 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nickevansuk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukehesluke Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukehesluke commented Feb 27, 2024 •

edited

Loading

lukehesluke Feb 28, 2024 •

edited

Loading

lukehesluke Mar 5, 2024 •

edited

Loading