Merge branch 'master' into feature/workflows-readme

CONTRIBUTING.md

@@ -23,33 +23,22 @@ When code is committed, the documentation generator is automatically run. This p
 
 ## Reference Implementation
 
-The OpenActive Test Suite is developed against a reference implementation, [OpenActive.Server.NET](https://github.com/openactive/OpenActive.Server.NET/).
+[**OpenActive.Server.NET**](https://github.com/openactive/OpenActive.Server.NET/) is a .NET SDK which can be used to create Open Booking API implementations.
 
-The CI checks that the Test Suite passes for the reference implementation. Therefore, the approach for adding each new test is to work on both:
+OpenActive.Server.NET contains a **reference implementation** of the Open Booking API, which is called [`BookingSystem.AspNetCore`](https://github.com/openactive/OpenActive.Server.NET/tree/master/Examples/BookingSystem.AspNetCore).
 
-- Reference Implementation: The implementation of the feature that's being tested.
-- Test Suite: The test itself.
+The OpenActive Test Suite is developed against [`BookingSystem.AspNetCore`](https://github.com/openactive/OpenActive.Server.NET/tree/master/Examples/BookingSystem.AspNetCore). Test Suite's CI checks that the Test Suite passes for this reference implementation.
 
-Any new feature that affects test coverage must be developed in a `coverage/*` branch in both reposities, in order for the CI to automatically run both feature branches against each other.
+Therefore, the approach for adding or updating tests is to work on both:
 
-### Locally running the Reference Implementation
+- [OpenActive.Server.NET](https://github.com/openactive/OpenActive.Server.NET/): Ensure that the expected behaviour is implemented in the reference implementation. This may require making changes to the libraries within the SDK, which the reference implementation uses, or the reference implementation ([`BookingSystem.AspNetCore`](https://github.com/openactive/OpenActive.Server.NET/tree/master/Examples/BookingSystem.AspNetCore)) itself.
 
-When you're working on both the Reference Implementation and the Test Suite, you'll want to run the Reference Implementation locally so you can check that the new tests and the new implementations work together.
+    In some cases, no changes are needed here as the updates to tests may be testing behaviours that have already been correctly implemented in the reference implementation.
+- [Test Suite](.): The tests.
 
-Additionally, when running the Reference Implementation locally, there is no limit on the number of tests you can run. The live version has a quota.
+Please be sure to follow the [**Pull Request Process**](#pull-request-process) when making changes to ensure that changes in both projects are tested together in CI.
 
-How to run them both locally:
-
-* Reference Implementation: To run this locally, follow the guidelines in its [project](https://github.com/openactive/OpenActive.Server.NET)'s documentation
-* Test Suite: With Reference Implementation running locally, use the `dev` config by setting env var `NODE_APP_INSTANCE=dev`.
-    e.g., to run broker and a specific test separately:
-    ```sh
-    NODE_APP_INSTANCE=dev npm run start-broker # to start the broker
-    # in another terminal:
-    NODE_APP_INSTANCE=dev npm run start-tests  -- packages/openactive-integration-tests/test/features/core/common-error-conditions/implemented/not-bookable-test.js
-    ```
-
-**N.B.:** When running Reference Implementation locally, the `unknown-endpoint` test in `core/common-error-conditions` will fail as the dev version of Reference Implementation shows a developer error page instead of the regular 404 response.
+When the necessary changes are made to both OpenActive.Server.NET and Test Suite, test them together on your machine by following the steps in [**Locally running Test Suite to test changes**](#locally-running-test-suite-to-test-changes).
 
 ## Pull Request Process
 
@@ -65,3 +54,53 @@ For new features that affect test coverage, use a `coverage/*` branch in this re
 ## Writing a new test
 
 See the [Integration Tests CONTRIBUTING.md](./packages/openactive-integration-tests/CONTRIBUTING.md) for guidance on writing a new test.
+
+## Locally running Test Suite to test changes
+
+When you are making changes to the Test Suite, please run it on your machine to check that the changes work before submitting a [pull request](#pull-request-process). These changes can be checked against the [**reference implementation**](#reference-implementation). You may or may not also have changes to the reference implementation that is required for your updated tests to pass. Either way, you'll need to run both projects on your machine order to test your changes.
+
+How to run them both locally:
+
+* **Reference Implementation**: To run this locally, follow the guidelines in its [project's contribution documentation](https://github.com/openactive/OpenActive.Server.NET/blob/master/CONTRIBUTING.md).
+* **Test Suite**: With the reference implementation running locally, use the [`default-dev.json` config](./config/default-dev.json) by setting env var `NODE_APP_INSTANCE=dev`.
+
+    For example, to run broker and a specific test separately (See [**Tips for quicker test runs**](#tips-for-quicker-test-runs) for a much more efficient process):
+
+    ```sh
+    NODE_APP_INSTANCE=dev npm start
+    ```
+
+**N.B.:** When running the reference implementation locally, the `unknown-endpoint` test in `core/common-error-conditions` will fail as the dev version of reference implementation shows a developer error page instead of the regular 404 response.
+
+### Tips for quicker test runs
+
+Running all of the tests can take a long time. To speed up your development/testing feedback loop, you can:
+
+1. Run a single test at a time, rather than running all of the tests.
+2. Keep [Broker Microservice](./packages/openactive-broker-microservice/) running in the background, rather than restarting it for each test.
+
+Here's an example of how to do this:
+
+```sh
+# In one terminal, start broker
+NODE_APP_INSTANCE=dev npm run start-broker
+
+# Then, in ANOTHER terminal, run only the opportunity-in-past test
+NODE_APP_INSTANCE=dev npm run start-tests -- packages/openactive-integration-tests/test/features/core/common-error-conditions/implemented/opportunity-in-past-test.js
+# OR, run only the tests within the common-error-conditions feature
+NODE_APP_INSTANCE=dev npm run start-tests -- packages/openactive-integration-tests/test/features/core/common-error-conditions/
+```
+
+#### Even quicker test runs - reduce random test data
+
+When using the [`default-dev.json` config](./config/default-dev.json) (i.e. with `NODE_APP_INSTANCE=dev`), Test Suite is set to use [**controlled mode**](https://developer.openactive.io/open-booking-api/key-decisions#controlled-mode). This means that it creates all the data that it needs for testing. Therefore, the [reference implementation](#reference-implementation) does not need to generate its own data, which it does by default.
+
+Turning this off will speed up the bootstrapping of both reference implementation and broker microservice, which no longer needs to harvest lots of data.
+
+To do this, see the [**Optimizing for Controlled Mode**](https://github.com/openactive/OpenActive.Server.NET/blob/master/CONTRIBUTING.md#optimizing-for-controlled-mode) section of OpenActive.Server.NET's contribution documentation.
+
+### NODE_APP_INSTANCE=dev
+
+Setting env var `NODE_APP_INSTANCE=dev` informs Test Suite to use the [`default-dev.json` config file](./config/default-dev.json), which is configured to run against the reference implementation running locally.
+
+`NODE_APP_INSTANCE` is a feature of the [`config`](https://github.com/node-config/node-config/) library.

README.md

@@ -117,6 +117,8 @@ Example:
     },
 ```
 
+In order for Test Suite to be able to log in to your OpenID Connect Provider, you'll need to also configure `broker.loginPagesSelectors` (see [Broker Microservice Configuration](./packages/openactive-broker-microservice/README.md#loginpagesselectors)).
+
 #### Request Headers
 
 Just a set of request HTTP headers which will be used to make booking requests. There are no restrictions on the `requestHeaders` that can be specified.
@@ -263,6 +265,15 @@ Note that running `npm start` in the root `openactive-test-suite` directory will
 
 In order to run the tests in random mode, the target Open Booking API implementation will need to have some Opportunity data pre-loaded. Use [Test Data Generator](./packages/openactive-integration-tests/test-data-generator/) to find out how much data is needed and in what configuration.
 
+## Certification
+
+An OpenActive Conformance Certificate offers a mechanism by which implementing systems can prove their conformance to the OpenActive specifications. Test Suite can be configured to output a Conformance Certificate upon all tests passing.
+
+An example conformance certificate can be found here:
+https://certificates.reference-implementation.openactive.io/examples/all-features/random/
+
+For more information about Certification please see [here](packages/openactive-integration-tests/test/certification/README.md).
+
 # Contributing
 
 - [Contributing to the project](./CONTRIBUTING.md)

packages/openactive-broker-microservice/README.md

@@ -114,7 +114,7 @@ These selectors would work for a login page whose HTML looks like:
 <input type="submit" class="btn-primary" value="Submit" />
 ```
 
-**Context**: For Sellers which use OpenID Connect for authorization, Broker needs to acquire these Sellers' tokens in order to authenticate as each of these Sellers. Broker does this by going through [Authorization Code Flow](https://oauth.net/2/grant-types/authorization-code/), loading the Booking System's login page in a headless browser and entering username/password details therein.
+**Context**: For Sellers which use OpenID Connect for authorization (see [Sellers Authentication Configuration](../../README.md#openid-connect) to see how this is configured), Broker needs to acquire these Sellers' tokens in order to authenticate as each of these Sellers. Broker does this by going through [Authorization Code Flow](https://oauth.net/2/grant-types/authorization-code/), loading the Booking System's login page in a headless browser and entering username/password details therein.
 
 ### `opportunityFeedRequestHeaders`
 

packages/openactive-integration-tests/ARCHITECTURE.md

@@ -59,7 +59,7 @@ These consist of:
 
 - [Chakram](http://dareid.github.io/chakram/): This is a HTTP test framework designed for Mocha (however it works fine on Jest)
 - [Request helper](test/helpers/request-helper.js): This makes HTTP requests (e.g. to the Booking System under test; or to the Broker Microservice), and records the request + response against the **Logger**. There are methods to directly make requests, along with methods for each API endpoint.
-- [Flow Stages](test/helpers/flow-stages/flow-stage.js): A part of the booking flow (e.g. the C1 request). Use it to call the relevant API endpoint. When called, it stores the results, which can then be applied to successive Flow Stages (e.g. C1 output can be used for the C2 request).
+- [Flow Stages](test/helpers/flow-stages/README.md): A part of the booking flow (e.g. the C1 request). Use it to call the relevant API endpoint. When called, it stores the results, which can then be applied to successive Flow Stages (e.g. C1 output can be used for the C2 request).
 
 # Test flow
 

packages/openactive-integration-tests/CONTRIBUTING.md

@@ -54,15 +54,14 @@ For each feature, there also exists a `feature.json` file, which contains metada
 
 `test/features/<testCategory>/<testFeature>/feature.json`
 
-Here is an example `feature.json` file:
+Here is a fairly complete example `feature.json` file:
 
 ```json
 {
   "category": "payment",
   "identifier": "non-free-opportunities",
   "name": "Opportunities with a non-zero price",
   "description": "The most simple form of booking with payment. Does not check for leases.",
-  "explainer": "",
   "specificationReference": "https://www.openactive.io/open-booking-api/EditorsDraft/#step-by-step-process-description",
   "required": false,
   "coverageStatus": "complete",
@@ -76,6 +75,31 @@ Here is an example `feature.json` file:
 }
 ```
 
+Properties in a `feature.json` file:
+
+- `category` (REQUIRED): Identifier of the category that this feature belongs to
+- `identifier` (REQUIRED): This feature's identifier
+- `name` (REQUIRED): Human-readable name of the feature
+- `description` (REQUIRED): Description of the feature
+- `specificationReference` (REQUIRED): A link to the relevant part of the Open Booking API specification.
+- `required` (REQUIRED): Whether or not this feature is required to be implemented. If `true`, then the feature must be implemented for a Booking System to be considered conformant. If `false`, then the feature is optional.
+- `coverageStatus` (REQUIRED): Whether or not this feature is considered to have a complete set of tests written. Options:
+    - `complete`: All tests for this feature have been written.
+    - `partial`: Some tests for this feature have been written.
+    - `none`: No tests for this feature have been written.
+- `links` (OPTIONAL): Links to resources that may be useful for implementing this feature. Each link has a human-readable name, `name`, and a URL, `href`.
+- `requiredCondition` (OPTIONAL): Description of when this feature is required. This is for features which are required only in certain circumstances. For an example, `access-code-update-notifications`'s `feature.json`:
+
+    ```json
+      "requiredCondition": "Required if accessCode can change",
+    ```
+- `requiresOneOfIfImplemented` (OPTIONAL): A list of features which must be implemented if this feature is implemented. The example `feature.json` at the top of this section indicates that, if a Booking System implements `non-free-opportunities`, then they also need to implement at least one of `prepayment-optional`, `prepayment-required`, or `prepayment-unavailable`.
+- `explainer` (OPTIONAL): An explanation of whether or not this feature should be implemented, especially for complicated use cases. For an example, `prepayment-required-unavailable`'s `feature.json`:
+
+    ```json
+      "explainer": "This feature must be implemented if prepayment-required and prepayment-unavailable are both implemented.",
+    ```
+
 ## Anatomy of a test
 
 Most tests are configured using `FeatureHelper.describeFeature(..)`. A simple example usage of this is:

packages/openactive-integration-tests/documentation/featureJson.js

@@ -0,0 +1,43 @@
+const { z } = require('zod');
+
+const FeatureJsonLinkSchema = z.object({
+  name: z.string(),
+  href: z.string().url(),
+});
+
+/**
+ * The shape of data in feature.json files.
+ */
+const FeatureJsonSchema = z.object({
+  category: z.string(),
+  identifier: z.string(),
+  name: z.string(),
+  description: z.string(),
+  explainer: z.string().optional(),
+  /**
+   * URL reference to a section of the Open Booking API
+   */
+  specificationReference: z.string().url(),
+  /**
+   * Is it required for an implementation to implement this feature?
+   */
+  required: z.boolean(),
+  /**
+   * How much test coverage has been written for this feature
+   */
+  coverageStatus: z.enum(['none', 'partial', 'complete']),
+  /**
+   * Description of when this feature is required
+   */
+  requiredCondition: z.string().optional(),
+  links: z.array(FeatureJsonLinkSchema).optional(),
+  requiresOneOfIfImplemented: z.array(z.string()).optional(),
+});
+
+/**
+ * @typedef {z.infer<typeof FeatureJsonSchema>} FeatureJson
+ */
+
+module.exports = {
+  FeatureJsonSchema,
+};

packages/openactive-integration-tests/documentation/generator.js

@@ -11,6 +11,7 @@ const jestConfig = require('../jest.config');
 const defaultConfig = require('../../../config/default.json');
 const { OpportunityCriteriaRequirements, SellerCriteriaRequirements } = require('../test/helpers/criteria-utils');
 const { DefaultMap } = require('../test/helpers/map-utils');
+const { FeatureJsonSchema } = require('./featureJson.js');
 
 const FEATURES_ROOT = path.join(__dirname, '..', 'test', 'features');
 const INDEX_README_FILE = path.join(FEATURES_ROOT, 'README.md');
@@ -21,6 +22,7 @@ const INDEX_TESTS_IMPLEMENTED_JSON_FILE = path.join(FEATURES_ROOT, 'tests-implem
 /**
  * @typedef {import('../test/helpers/feature-helper').TestModuleExports} TestModuleExports
  * @typedef {import('../test/types/OpportunityCriteria').SellerCriteria} SellerCriteria
+ * @typedef {import('./featureJson.js').FeatureJson} FeatureJson
  */
 
 /**
@@ -50,26 +52,6 @@ const INDEX_TESTS_IMPLEMENTED_JSON_FILE = path.join(FEATURES_ROOT, 'tests-implem
  *   They can still easily be interpreted as an array with Object.keys().
  */
 
-/**
- * @typedef {object} FeatureJsonLink
- * @property {string} name
- * @property {string} href
- */
-
-/**
- * @typedef {object} FeatureJson The shape of data in feature.json files.
- * @property {string} category
- * @property {string} identifier
- * @property {string} name
- * @property {string} description
- * @property {string} explainer
- * @property {string} specificationReference URL reference to a section of the Open Booking API
- * @property {boolean} required Is it required for an implementation to implement this feature?
- * @property {'none' | 'partial' | 'complete'} coverageStatus How much test coverage has been written for this feature
- * @property {string} [requiredCondition] Description of when this feature is required
- * @property {FeatureJsonLink[]} [links]
- */
-
 /**
  * @typedef {FeatureJson & {
  *   criteriaRequirement?: Map<string, number>,
@@ -109,8 +91,8 @@ const testMetadata = fg.sync(jestConfig.testMatch, { cwd: rootDirectory }).map(f
 /** @type {FeatureMetadataItem[]} */
 const featureMetadata = fg.sync('**/test/features/**/feature.json', { cwd: rootDirectory }).map(function (file) {
   console.log(`Reading: ${file}`);
-  // TODO: Verify that the data actually conforms to the type.
-  return /** @type {FeatureJson} */(require(`${rootDirectory}${file}`));
+  const rawJson = require(`${rootDirectory}${file}`);
+  return FeatureJsonSchema.parse(rawJson);
 });
 
 // Sort features so that required ones are first